Open In App

How to Find & Drop duplicate columns in a Pandas DataFrame?

Improve
Improve
Like Article
Like
Save
Share
Report

Let’s discuss How to Find and drop duplicate columns in a Pandas DataFrame. First, Let’s create a simple Dataframe with column names ‘Name’, ‘Age’, ‘Domicile’, and ‘Age’/’Marks’. 

Find Duplicate Columns from a DataFrame

To find duplicate columns we need to iterate through all columns of a DataFrame and for each and every column it will search if any other column exists in DataFrame with the same contents already. If yes then that column name will be stored in the duplicate column set. In the end, the function will return the list of column names of the duplicate column. In this way, we can find duplicate labels in Pandas.

Python3




import pandas as pd
 
def getDuplicateColumns(df):
 
    # Create an empty set
    duplicateColumnNames = set()
 
    # Iterate through all the columns of dataframe
    for x in range(df.shape[1]):
 
        # Take column at xth index.
        col = df.iloc[:, x]
 
        # Iterate through all the columns
        for y in range(x + 1, df.shape[1]):
 
            # Take column at yth index.
            otherCol = df.iloc[:, y]
 
            # Check if two columns at x & y
            if col.equals(otherCol):
                duplicateColumnNames.add(df.columns.values[y])
 
    return list(duplicateColumnNames)
 
 
# Driver code
if __name__ == "__main__":
 
    # List of Tuples
    students = [
        ('Ankit', 34, 'Uttar pradesh', 34),
        ('Riti', 30, 'Delhi', 30),
        ('Aadi', 16, 'Delhi', 16),
        ('Riti', 30, 'Delhi', 30),
        ('Riti', 30, 'Delhi', 30),
        ('Riti', 30, 'Mumbai', 30),
        ('Ankita', 40, 'Bihar', 40),
        ('Sachin', 30, 'Delhi', 30)
    ]
 
    # Create a DataFrame object
    df = pd.DataFrame(students, columns=['Name', 'Age',
                                     'Domicile', 'Marks'])
 
    # Get list of duplicate columns
    duplicateColNames = getDuplicateColumns(df)
 
    for column in duplicateColNames:
        print('Column Name : ', column)


Output:

Column Name:  Marks

Remove Duplicate Columns from a DataFrame

Below are the ways by which we can remove duplicate labels in Pandas in Python:

  • Using drop_duplicates()
  • Using df.loc[]
  • Using df.columns.duplicated()
  • Using df.drop

Drop duplicate columns from a DataFrame using drop_duplicates()

Pandas drop_duplicates() method helps in removing duplicates from the Pandas Dataframe In Python.

Python3




# Drop duplicate columns
df2 = df.T.drop_duplicates().T
print(df2)


Output:

     Name Age       Domicile
0 Ankit 34 Uttar pradesh
1 Riti 30 Delhi
2 Aadi 16 Delhi
3 Riti 30 Mumbai
4 Ankita 40 Bihar
5 Sachin 30 Delhi

Remove duplicate columns from a DataFrame using df.loc[]

Pandas df.loc[] attribute access a group of rows and columns by label(s) or a boolean array in the given DataFrame.

Python3




# Remove duplicate columns pandas DataFrame
df2 = df.loc[:,~df.columns.duplicated()]
print(df2)


Output:

     Name Age       Domicile
0 Ankit 34 Uttar pradesh
1 Riti 30 Delhi
2 Aadi 16 Delhi
3 Riti 30 Mumbai
4 Ankita 40 Bihar
5 Sachin 30 Delhi

Remove duplicate columns from a DataFrame using df.columns.duplicated()

Pandas df.duplicated() method helps in analyzing duplicate values only. It returns a boolean series which is True only for Unique elements.

Python3




# Use DataFrame.columns.duplicated() to drop duplicate columns
duplicate_cols = df.columns[df.columns.duplicated()]
df.drop(columns=duplicate_cols, inplace=True)
print(df)


Output:

     Name       Domicile
0 Ankit Uttar pradesh
1 Riti Delhi
2 Aadi Delhi
3 Riti Mumbai
4 Ankita Bihar
5 Sachin Delhi

Drop duplicate columns in a DataFrame using df.drop

To remove the duplicate columns we can pass the list of duplicate column names returned by our user defines function getDuplicateColumns() to the Dataframe.drop() method. 

Python3




# import pandas library
import pandas as pd
 
def getDuplicateColumns(df):
 
    # Create an empty set
    duplicateColumnNames = set()
     
    # Iterate through all the columns
    # of dataframe
    for x in range(df.shape[1]):
         
        # Take column at xth index.
        col = df.iloc[:, x]
         
        for y in range(x + 1, df.shape[1]):
             
            # Take column at yth index.
            otherCol = df.iloc[:, y]
             
            if col.equals(otherCol):
                duplicateColumnNames.add(df.columns.values[y])
                 
    # Return list of unique column names
    # whose contents are duplicates.
    return list(duplicateColumnNames)
 
# Driver code
if __name__ == "__main__" :
 
    # List of Tuples
    students = [
            ('Ankit', 34, 'Uttar pradesh', 34),
            ('Riti', 30, 'Delhi', 30),
            ('Aadi', 16, 'Delhi', 16),
            ('Riti', 30, 'Delhi', 30),
            ('Riti', 30, 'Delhi', 30),
            ('Riti', 30, 'Mumbai', 30),
            ('Ankita', 40, 'Bihar', 40),
            ('Sachin', 30, 'Delhi', 30)
        ]
 
    # Create a DataFrame object
    df = pd.DataFrame(students,
                        columns =['Name', 'Age', 'Domicile', 'Marks'])
 
    # Dropping duplicate columns
    rslt_df = df.drop(columns = getDuplicateColumns(df))
 
    print("Resultant Dataframe :")
 
    # Show the dataframe
rslt_df


Output:



Last Updated : 22 Dec, 2023
Like Article
Save Article
Previous
Next
Share your thoughts in the comments
Similar Reads