Concatenate Pandas DataFrames Without Duplicates
Last Updated :
16 Feb, 2022
In this article, we are going to concatenate two dataframes using pandas module.
In order to perform concatenation of two dataframes, we are going to use the pandas.concat().drop_duplicates() method in pandas module.
Step-by-step Approach:
- Import module.
- Load two sample dataframes as variables.
- Concatenate the dataframes using pandas.concat().drop_duplicates() method.
- Display the new dataframe generated.
Below are some examples which depict how to perform concatenation between two dataframes using pandas module without duplicates:
Example 1:
Python3
import pandas as pd
dataframe1 = pd.DataFrame({ 'columnA' : [ 20 , 30 , 40 ],
'columnB' : [ 200 , 300 , 400 ]})
dataframe2 = pd.DataFrame({ 'columnA' : [ 50 , 20 , 60 ],
'columnB' : [ 500 , 200 , 600 ]})
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
new_dataframe
|
Output:
Here, we have concatenated two dataframes using pandas.concat() method.
Example 2:
Python3
import pandas as pd
dataframe1 = pd.DataFrame({ 'name' : [ 'rahul' , 'anjali' , 'kajal' ],
'age' : [ 23 , 28 , 30 ]})
dataframe2 = pd.DataFrame({ 'name' : [ 'devesh' , 'rashi' , 'anjali' ],
'age' : [ 20 , 15 , 28 ]})
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
new_dataframe = new_dataframe.reset_index(drop = True )
new_dataframe
|
Output:
As shown in the output image, we get the concatenation of dataframes without removing duplicates.
Example 3:
Python3
import pandas as pd
dataframe1 = pd.DataFrame({ 'empname' : [ 'rohan' , 'hina' , 'alisa' , ],
'department' : [ 'IT' , 'admin' , 'finance' , ],
'designation' : [ 'Sr.developer' , 'administrator' , 'executive' , ]})
dataframe2 = pd.DataFrame({ 'empname' : [ 'rishi' , 'huma' , 'alisa' , ],
'department' : [ 'cyber security' , 'HR' , 'finance' , ],
'designation' : [ 'penetration tester' , 'HR executive' , 'executive' , ]})
new_dataframe = pd.concat([dataframe1, dataframe2]).drop_duplicates()
new_dataframe = new_dataframe.reset_index(drop = True )
new_dataframe
|
Output:
Here is another example, which depicts how to concatenate two dataframes.
Output dataset for the Example 3
Like Article
Suggest improvement
Share your thoughts in the comments
Please Login to comment...