Selecting, dropping and inserting rows in a pandas DataFrame

Most row work in pandas comes down to one idea: build a boolean mask, then keep (or drop) the rows where it is True. This guide covers the cases that come up most often: selecting rows by several conditions, catching missing values, dropping rows by a numeric threshold or a string prefix, and inserting new rows.

Select rows that match several conditions

Sample data: election results where each row is a list, and you want a subset of them.

The goal is a new DataFrame that:

1. Includes rows where “List Number” starts with 23.
2. Includes rows where “Position” is National Deputy.
3. Includes rows with missing or empty “Votes” values.

import pandas as pd

data = {
    'List Number': ['23"A"', '23"B"', '23"C"', '23"D"', '23"F"', 'TOTAL'],
    'Political Group': ['FOR CLUB, CHURCH, AND SCHOOL', 'UNITED FOR THE PEOPLE', 'CITIZENS', 'PRODUCTION AND WORK', 'POPULARS CBA', 'TOTAL UNION POPULAR FEDERAL'],
    'Votes': [1860, 1067, 1130, 2154, 3565, 9776],
    'Percentage': [19.03, 10.91, 11.56, 22.03, 36.47, 0.47],
    'Position': ['Cordoba', 'Cordoba', 'Cordoba', 'Cordoba', 'Cordoba', 'Cordoba']
}

df = pd.DataFrame(data)

Combine conditions with & (and) and | (or). Each condition needs its own parentheses because & and | bind tighter than ==:

df_filtered = df[df['List Number'].str.startswith('23') & (df['Position'] == 'National Deputy')]

With the sample above this returns an empty DataFrame: every row has Position == 'Cordoba'. Swap in a value that exists in your data to see rows come back.

Rows where “Votes” is missing (NaN) or an empty string:

df_missing_votes = df[df["Votes"].isna() | (df["Votes"] == "")]

Append those rows to the filtered set and print the result:

df_combined = pd.concat([df_filtered, df_missing_votes], ignore_index=True)
print(df_combined)

Drop rows based on a column value

Weather data where you want to drop every row with qty < 5:

YEAR,MO,GP,HR,qty,SKYMEAN,SKY100MEAN,SKY100STD,SKY100MEDIAN
1973,1,1,0,1,7,87.5,,87.5
1973,1,1,14,1,8,100,,100
1973,1,1,15,4,5.25,65.625,43.75,87.5
1973,1,1,16,12,4.416666667,55.20833333,48.98351222,87.5
1973,1,1,17,22,5.227272727,65.34090909,43.80214888,87.5
1973,1,1,18,21,5.095238095,63.69047619,39.70688737,87.5
1973,1,1,19,20,5.2,65,39.23612728,87.5
1973,1,1,20,32,5.15625,64.453125,36.63097565,87.5
1973,1,1,21,19,5.526315789,69.07894737,37.5851859,87.5
1973,1,1,22,12,6.083333333,76.04166667,35.93647067,87.5
1973,1,1,24,144,5.263888889,65.79861111,39.13917785,87.5
1973,2,1,0,1,8,100,,100
1973,2,1,15,1,7,87.5,,87.5
1973,2,1,16,10,5.4,67.5,35.93976442,87.5
1973,2,1,17,24,4.458333333,55.72916667,44.83956404,87.5
1973,2,1,18,20,4.4,55,42.2243382,56.25
1973,2,1,19,21,4.714285714,58.92857143,43.50697809,87.5
1973,2,1,20,26,5,62.5,43.30127019,87.5
1973,2,1,21,22,5.636363636,70.45454545,39.8183103,87.5
1973,2,1,22,4,6,75,33.85016002,87.5
1973,2,1,23,2,7.5,93.75,8.838834765,93.75
1973,2,1,24,131,5.007633588,62.59541985,41.32703132,87.5

drop() takes index labels, so pass it the index of the rows that match the condition, then reset the index:

import pandas as pd

sample_data = pd.read_csv('path_to_your_csv_file.csv')

# Drop rows where qty < 5
sample_data.drop(sample_data[sample_data['qty'] < 5].index, inplace=True)

# Reset the index
sample_data = sample_data.reset_index(drop=True)

print(sample_data)

Result: only rows with qty >= 5 remain, re-indexed from 0.

YEAR,MO,GP,HR,qty,SKYMEAN,SKY100MEAN,SKY100STD,SKY100MEDIAN
1973,1,1,16,12,4.416666667,55.20833333,48.98351222,87.5
1973,1,1,17,22,5.227272727,65.34090909,43.80214888,87.5
1973,1,1,18,21,5.095238095,63.69047619,39.70688737,87.5
1973,1,1,19,20,5.2,65,39.23612728,87.5
1973,1,1,20,32,5.15625,64.453125,36.63097565,87.5
1973,1,1,21,19,5.526315789,69.07894737,37.5851859,87.5
1973,1,1,22,12,6.083333333,76.04166667,35.93647067,87.5
1973,1,1,24,144,5.263888889,65.79861111,39.13917785,87.5
1973,2,1,16,10,5.4,67.5,35.93976442,87.5
1973,2,1,17,24,4.458333333,55.72916667,44.83956404,87.5
1973,2,1,18,20,4.4,55,42.2243382,56.25
1973,2,1,19,21,4.714285714,58.92857143,43.50697809,87.5
1973,2,1,20,26,5,62.5,43.30127019,87.5
1973,2,1,21,22,5.636363636,70.45454545,39.8183103,87.5
1973,2,1,24,131,5.007633588,62.59541985,41.32703132,87.5

The equivalent without drop() is to keep the complement: sample_data = sample_data[sample_data['qty'] >= 5].

Drop rows where any column starts with a prefix

A common cleaning step: remove separator rows such as --- that appear in any column.

import pandas as pd

data = {'Column1': ['---Row1', 'Value2', 'Value3', '---Row4', 'Value5'],
        'Column2': ['ValueA', '---RowB', 'ValueC', '---RowD', 'ValueE']}

df = pd.DataFrame(data)

print("Original DataFrame:")
print(df)

# Filter out rows that start with '---' in any column
df = df[~df.apply(lambda row: row.str.startswith('---')).any(axis=1)]

# Reset index
df = df.reset_index(drop=True)

print("\nDataFrame after removing rows that start with '---':")
print(df)

How the mask is built:

• df.apply(lambda row: row.str.startswith('---')) returns a boolean DataFrame with the same shape as df. (apply runs column by column by default, so row here is really each column; the result is the same.)
• .any(axis=1) collapses it to one boolean per row: True if any column matched.
• ~ negates it, so only rows with no match are kept.
• reset_index(drop=True) renumbers the index after the drop.

Output:

Original DataFrame:
   Column1 Column2
0  ---Row1  ValueA
1   Value2  ---RowB
2   Value3  ValueC
3  ---Row4  ---RowD
4   Value5  ValueE

DataFrame after removing rows that start with '---':
   Column1 Column2
0   Value3  ValueC
1   Value5  ValueE

Change the condition inside the lambda to filter on other prefixes or patterns.

Insert new rows

All examples start from the same DataFrame:

import pandas as pd

data = {'Column1': [1, 2, 3],
        'Column2': ['A', 'B', 'C']}
df = pd.DataFrame(data)

One row with loc[]

Assign to the next integer label:

new_row = pd.Series({'Column1': 4, 'Column2': 'D'})

new_index = len(df)
df.loc[new_index] = new_row

print(df)

   Column1 Column2
0        1       A
1        2       B
2        3       C
3        4       D

Several rows with concat()

Build a DataFrame with the new rows and concatenate once:

new_rows = pd.DataFrame({'Column1': [4, 5], 'Column2': ['D', 'E']})

df = pd.concat([df, new_rows], ignore_index=True)

print(df)

   Column1 Column2
0        1       A
1        2       B
2        3       C
3        4       D
4        5       E

append() (pandas < 2.0 only)

Older code uses DataFrame.append():

new_row = {'Column1': 4, 'Column2': 'D'}

df = df.append(new_row, ignore_index=True)

print(df)

   Column1 Column2
0        1       A
1        2       B
2        3       C
3        4       D

append() was deprecated in pandas 1.4 and removed in 2.0. On current pandas it raises AttributeError: 'DataFrame' object has no attribute 'append'. Replace it with pd.concat([df, pd.DataFrame([new_row])], ignore_index=True).

Gotchas

• and/or instead of &/|. Python’s and on two Series raises ValueError: The truth value of a Series is ambiguous. Use &, |, ~ and parenthesise each condition.
• Index gaps after filtering. Dropping rows keeps the original labels (0, 2, 5…). Call reset_index(drop=True) before positional logic or before using df.loc[len(df)] to add a row.
• df.loc[len(df)] on a non-default index. If the index is not 0..n-1, len(df) may be an existing label, and the assignment overwrites that row instead of adding one.
• str.startswith on non-string columns. The .str accessor fails on numeric columns. Restrict apply to string columns, or convert with .astype(str) first.
• Growing a DataFrame in a loop. Adding rows one at a time copies the frame each time. Collect rows in a list and call concat() (or pd.DataFrame(rows)) once.

Related

• The pandas DataFrame apply method explains the apply call used in the prefix filter.
• Creating a pandas DataFrame from a list of lists shows another way to build the input frame.
• Extracting data from fixed-width text files into a pandas DataFrame is a typical source of rows that need cleaning.
• Working with JSON data in Python and pandas covers loading JSON before filtering it.