Skip to content
Related Articles

Related Articles

Concatenate Pandas DataFrames Without Duplicates
  • Last Updated : 17 Dec, 2020

In this article, we are going to concatenate two dataframes using pandas module.

In order to perform concatenation of two dataframes, we are going to use the pandas.concat().drop_duplicates() method in pandas module.

Step-by-step Approach: 

  • Import module.
  • Load two sample dataframes as variables.
  • Concatenate the dataframes using pandas.concat().drop_duplicates() method.
  • Display the new dataframe generated.

Below are some examples which depict how to perform concatenation between two dataframes using pandas module without duplicates:

Example 1:



Python3

filter_none

edit
close

play_arrow

link
brightness_4
code

# Importing pandas library
import pandas as pd
  
# loading dataframes
dataframe1 = pd.DataFrame({'columnA': [20, 30, 40],
                           'columnB': [200, 300, 400]})
  
dataframe2 = pd.DataFrame({'columnA': [50, 20, 60],
                           'columnB': [500, 200, 600]})
  
# Concatenating dataframes without duplicates
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
  
# Display concatenated dataframe
new_dataframe

chevron_right


Output:

Here, we have concatenated two dataframes using pandas.concat() method.

Example 2:

Python3

filter_none

edit
close

play_arrow

link
brightness_4
code

# Importing pandas library
import pandas as pd
  
# loading dataframes
dataframe1 = pd.DataFrame({'name': ['rahul', 'anjali', 'kajal'],
                           'age': [23, 28, 30]})
  
dataframe2 = pd.DataFrame({'name': ['devesh', 'rashi', 'anjali'],
                           'age': [20, 15, 28]})
  
# Concatenating two dataframes wtithout duplicates
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
  
# Resetting index
new_dataframe = new_dataframe.reset_index(drop=True)
  
# Display dataframe generated
new_dataframe

chevron_right


Output:



As shown in the output image, we get the concatenation of dataframes without removing duplicates.

Example 3:

Python3

filter_none

edit
close

play_arrow

link
brightness_4
code

# Importing pandas libraray
import pandas as pd
  
# Loading dataframes
dataframe1 = pd.DataFrame({'empname': ['rohan', 'hina', 'alisa', ],
                           'department': ['IT', 'admin', 'finance', ],
                           'designation': ['Sr.developer', 'administrator', 'executive', ]})
  
dataframe2 = pd.DataFrame({'empname': ['rishi', 'huma', 'alisa', ],
                           'department': ['cyber security', 'HR', 'finance', ],
                           'designation': ['penetration tester', 'HR executive', 'executive', ]})
  
# Concatenating two dataframes wtithout duplicates
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
  
# Resetting index
new_dataframe = new_dataframe.reset_index(drop=True)
  
# Display dataframe generated
new_dataframe

chevron_right


Output:

Here is another example, which depicts how to concatenate two dataframes.

Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course.

My Personal Notes arrow_drop_up
Recommended Articles
Page :