Skip to content
Related Articles

Related Articles

Improve Article
Save Article
Like Article

Concatenate Pandas DataFrames Without Duplicates

  • Difficulty Level : Expert
  • Last Updated : 16 Feb, 2022

In this article, we are going to concatenate two dataframes using pandas module.

In order to perform concatenation of two dataframes, we are going to use the pandas.concat().drop_duplicates() method in pandas module.

Step-by-step Approach: 

  • Import module.
  • Load two sample dataframes as variables.
  • Concatenate the dataframes using pandas.concat().drop_duplicates() method.
  • Display the new dataframe generated.

Below are some examples which depict how to perform concatenation between two dataframes using pandas module without duplicates:

Example 1:

Python3




# Importing pandas library
import pandas as pd
 
# loading dataframes
dataframe1 = pd.DataFrame({'columnA': [20, 30, 40],
                           'columnB': [200, 300, 400]})
 
dataframe2 = pd.DataFrame({'columnA': [50, 20, 60],
                           'columnB': [500, 200, 600]})
 
# Concatenating dataframes without duplicates
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
 
# Display concatenated dataframe
new_dataframe

Output:

Here, we have concatenated two dataframes using pandas.concat() method.

Example 2:

Python3




# Importing pandas library
import pandas as pd
 
# loading dataframes
dataframe1 = pd.DataFrame({'name': ['rahul', 'anjali', 'kajal'],
                           'age': [23, 28, 30]})
 
dataframe2 = pd.DataFrame({'name': ['devesh', 'rashi', 'anjali'],
                           'age': [20, 15, 28]})
 
# Concatenating two dataframes without duplicates
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
 
# Resetting index
new_dataframe = new_dataframe.reset_index(drop=True)
 
# Display dataframe generated
new_dataframe

Output:

As shown in the output image, we get the concatenation of dataframes without removing duplicates.

Example 3:

Python3




# Importing pandas library
import pandas as pd
 
# Loading dataframes
dataframe1 = pd.DataFrame({'empname': ['rohan', 'hina', 'alisa', ],
                           'department': ['IT', 'admin', 'finance', ],
                           'designation': ['Sr.developer', 'administrator', 'executive', ]})
 
dataframe2 = pd.DataFrame({'empname': ['rishi', 'huma', 'alisa', ],
                           'department': ['cyber security', 'HR', 'finance', ],
                           'designation': ['penetration tester', 'HR executive', 'executive', ]})
 
# Concatenating two dataframes without duplicates
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
 
# Resetting index
new_dataframe = new_dataframe.reset_index(drop=True)
 
# Display dataframe generated
new_dataframe

Output:

Here is another example, which depicts how to concatenate two dataframes.

Output dataset for the Example 3


My Personal Notes arrow_drop_up
Recommended Articles
Page :

Start Your Coding Journey Now!