Open In App

Python | Removing duplicate dicts in list

Improve
Improve
Improve
Like Article
Like
Save Article
Save
Share
Report issue
Report

Removal of duplicates is essential in many applications. A list of dictionaries is quite common and sometimes we require to duplicate the duplicated. Let’s discuss certain ways in which we can remove duplicate dictionaries in a list in Python.

Using Loop for Removing duplicate dictionaries in a list

The basic method that comes to our mind while performing this operation is the naive method of iterating the list of dictionaries in Python and manually removing the duplicate dictionary and appending to the new list

Python3




# using naive method
 
# initializing list
test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},
             {"Kil" : 2}, {"Akshat" : 3}]
 
# printing original list
print ("Original list : " + str(test_list))
 
# using naive method to
# remove duplicates
res_list = []
for i in range(len(test_list)):
    if test_list[i] not in test_list[i + 1:]:
        res_list.append(test_list[i])
 
# printing resultant list
print ("Resultant list is : " + str(res_list))


Output:

Original list : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}, {'Kil': 2}, {'Akshat': 3}]
Resultant list is : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}]

Using list comprehension for Removing duplicate dictionaries in a list

The use of list comprehension and enumerate can possibly allow to achieve this particular task in a single line and hence is of a good utility. 

Python3




# Python3 code to demonstrate
# remove duplicate dictionary
# using list comprehension
 
# initializing list
test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},
             {"Akshat" : 3}, {"Kil" : 2}]
 
# printing original list
print ("Original list : " + str(test_list))
 
# using list comprehension to
# remove duplicates
res_list = [i for n, i in enumerate(test_list)
            if i not in test_list[:n]]
 
# printing resultant list
print ("Resultant list is : " + str(res_list))


Output:

Original list : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}, {'Kil': 2}, {'Akshat': 3}]
Resultant list is : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}]

Using frozenset for Removing duplicate dictionaries in a list

frozenset is used to assign a value to key in dictionary as a set. The repeated entries of dictionary are hence ignored and hence solving this particular task. 

Python3




# using frozenset
 
# initializing list
test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},
             {"Kil" : 2}, {"Akshat" : 3}]
 
# printing original list
print ("Original list : " + str(test_list))
 
# using frozenset to
# remove duplicates
res_list = {frozenset(item.items()) :
            item for item in test_list}.values()
 
# printing resultant list
print ("Resultant list is : " + str(res_list))


Output:

Original list : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}, {'Kil': 2}, {'Akshat': 3}]
Resultant list is : [{'Kil': 2}, {'Akshat': 3}, {'Akash': 1}]

Using unique everseen() for Removing duplicate dictionaries in a list

everseen() function is used to find all the unique elements present in the iterable and preserving their order of occurrence. Hence it remembers all elements ever seen in the iterable.

Python3




# importing required tools
from iteration_utilities import unique_everseen
 
# initializing list
test_list = [{"Akash" : 1}, {"Kil" : 2}, {"Akshat" : 3},
             {"Kil" : 2}, {"Akshat" : 3}]
 
# remove duplicates
res_list=list(unique_everseen(test_list))
 
# printing original list
print ("Original list : " + str(test_list))
 
# printing resultant list
print ("Resultant list is : " + str(res_list))


Output:

Original list : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}, {'Kil': 2}, {'Akshat': 3}]
Resultant list is : [{'Akash': 1}, {'Kil': 2}, {'Akshat': 3}]


Last Updated : 01 Sep, 2023
Like Article
Save Article
Previous
Next
Share your thoughts in the comments
Similar Reads