Python – Remove Duplicate Dictionaries characterized by Key
Last Updated :
14 Apr, 2023
Given a list of dictionaries, remove all the dictionaries which are duplicate with respect to K key.
Input : test_list = [{“Gfg” : 6, “is” : 9, “best” : 10}, {“Gfg” : 8, “is” : 11, “best” : 10}, {“Gfg” : 2, “is” : 16, “best” : 10}], K = “best” Output : [{“Gfg” : 6, “is” : 9, “best” : 10}] Explanation : All keys have 10 value, only 1st record is retained. Input : test_list = [{“Gfg” : 6, “is” : 9, “best” : 10}, {“Gfg” : 8, “is” : 11, “best” : 12}, {“Gfg” : 2, “is” : 16, “best” : 15}], K = “best” Output : [{“Gfg” : 6, “is” : 9, “best” : 10}, {“Gfg” : 8, “is” : 11, “best” : 12}, {“Gfg” : 2, “is” : 16, “best” : 15}] Explanation : All values of “best” are unique, hence no removal of dictionaries.
Method : Using loop
This is brute way in which this task can be performed. In this, we iterate for each dictionary and memoize the Key, if similar key’s same value occur, then that dictionary is avoided in resultant list of dictionaries.
Python3
test_list = [{ "Gfg" : 6 , "is" : 9 , "best" : 10 },
{ "Gfg" : 8 , "is" : 11 , "best" : 19 },
{ "Gfg" : 2 , "is" : 16 , "best" : 10 },
{ "Gfg" : 12 , "is" : 1 , "best" : 8 },
{ "Gfg" : 22 , "is" : 6 , "best" : 8 }]
print ( "The original list : " + str (test_list))
K = "best"
memo = set ()
res = []
for sub in test_list:
if sub[K] not in memo:
res.append(sub)
memo.add(sub[K])
print ( "The filtered list : " + str (res))
|
Output
The original list : [{‘Gfg’: 6, ‘is’: 9, ‘best’: 10}, {‘Gfg’: 8, ‘is’: 11, ‘best’: 19}, {‘Gfg’: 2, ‘is’: 16, ‘best’: 10}, {‘Gfg’: 12, ‘is’: 1, ‘best’: 8}, {‘Gfg’: 22, ‘is’: 6, ‘best’: 8}] The filtered list : [{‘Gfg’: 6, ‘is’: 9, ‘best’: 10}, {‘Gfg’: 8, ‘is’: 11, ‘best’: 19}, {‘Gfg’: 12, ‘is’: 1, ‘best’: 8}]
METHOD 2 : use list comprehension.
Here are the steps to implement it:
- Initialize the input list:
- Specify the key to be used for identifying duplicates
- Use list comprehension to filter out duplicates:
- This line of code creates a set of tuples where each tuple represents a dictionary in the input list. The dict() function is then used to convert each tuple back into a dictionary, and the resulting list contains only unique dictionaries.
- Print the filtered list:
Python3
test_list = [{ "Gfg" : 6 , "is" : 9 , "best" : 10 },
{ "Gfg" : 8 , "is" : 11 , "best" : 19 },
{ "Gfg" : 2 , "is" : 16 , "best" : 10 },
{ "Gfg" : 12 , "is" : 1 , "best" : 8 },
{ "Gfg" : 22 , "is" : 6 , "best" : 8 }]
K = "best"
res = [ dict (t) for t in { tuple (d.items()) for d in test_list}]
print ( "The filtered list : " + str (res))
|
Output
The filtered list : [{'Gfg': 2, 'is': 16, 'best': 10}, {'Gfg': 8, 'is': 11, 'best': 19}, {'Gfg': 6, 'is': 9, 'best': 10}, {'Gfg': 22, 'is': 6, 'best': 8}, {'Gfg': 12, 'is': 1, 'best': 8}]
The time complexity of this approach is O(nlogn) due to the use of sets and tuples.
Auxiliary space complexity is O(n) because we are creating a new list of dictionaries.
Like Article
Suggest improvement
Share your thoughts in the comments
Please Login to comment...