Python | Unique dictionary filter in list

While working with Python dictionaries, we can always come across a problem in which we have list of dictionary with possible repetitions and we wish to remove the duplicates. This is common problem one can face in Machine Learning Domain. Let’s discuss certain way in which this problem can be solved.

Method : Using map() + set() + items() + sorted() + tuple() In this method, we just combination of functions to perform this task. The map function performs the task of extending logic to all elements. The set function does the actual filtering on pairs accessed by items(). The sorted function is required by set to remove duplicate. 

Python3

# Python3 code to demonstrate working of
# Unique dictionary filter in list
# Using map() + set() + items() + sorted() + tuple()

# Initialize list
test_list = [{'gfg' : 1, 'is' : 2, 'best' : 3},
            {'gfg' : 1, 'is' : 2, 'best' : 3},
            {'gfg' : 9, 'is' : 8, 'best' : 6}]

# printing original list
print("The original list is : " + str(test_list))

# Unique dictionary filter in list
# Using map() + set() + items() + sorted() + tuple()
res = list(map(dict, set(tuple(sorted(sub.items())) for sub in test_list)))

# printing result
print("The list after removing duplicates : " + str(res))

Output

The original list is : [{'gfg': 1, 'is': 2, 'best': 3}, {'gfg': 1, 'is': 2, 'best': 3}, {'gfg': 9, 'is': 8, 'best': 6}]
The list after removing duplicates : [{'best': 3, 'gfg': 1, 'is': 2}, {'best': 6, 'gfg': 9, 'is': 8}]

Method #2 : Using list comprehension + dict()

This method uses list comprehension to filter unique dictionaries and the dict() function to convert the filtered list of tuples to a list of dictionaries.

Python3

# Python3 code to demonstrate working of
# Unique dictionary filter in list
# Using list comprehension + dict()

# Initialize list
test_list = [{'gfg' : 1, 'is' : 2, 'best' : 3},
             {'gfg' : 1, 'is' : 2, 'best' : 3},
             {'gfg' : 9, 'is' : 8, 'best' : 6}]

# printing original list
print("The original list is : " + str(test_list))

# Unique dictionary filter in list
# Using list comprehension + dict()
res = [dict(t) for t in {tuple(d.items()) for d in test_list}]

# printing result
print("The list after removing duplicates : " + str(res))
#This code is contributed by Edula Vinay Kumar Reddy

Output

The original list is : [{'gfg': 1, 'is': 2, 'best': 3}, {'gfg': 1, 'is': 2, 'best': 3}, {'gfg': 9, 'is': 8, 'best': 6}]
The list after removing duplicates : [{'gfg': 1, 'is': 2, 'best': 3}, {'gfg': 9, 'is': 8, 'best': 6}]

Time complexity: O(n)
Auxiliary Space: O(n)