Open In App

Python – Remove duplicate values across Dictionary Values

Improve
Improve
Improve
Like Article
Like
Save Article
Save
Share
Report issue
Report

Sometimes, while working with Python dictionaries, we can have a problem in which we need to remove all the duplicate values across all the dictionary value lists. This problem can have applications in data domains and web development domains. Let’s discuss certain ways in which this task can be performed.

Input: test_dict = {‘Manjeet’: [1], ‘Akash’: [1, 8, 9]} 
Output: {‘Manjeet’: [], ‘Akash’: [8, 9]} 

Input: test_dict = {‘Manjeet’: [1, 1, 1], ‘Akash’: [1, 1, 1]} 
Output: {‘Manjeet’: [], ‘Akash’: []}

Method #1: Using Counter() + list comprehension 

The combination of the above functions can be used to solve this problem. In this, we use Counter() to extract all frequencies and list comprehension to assign the value with a single occurrence in the value list. 

Python3




# Python3 code to demonstrate working of
# Remove duplicate values across Dictionary Values
# Using Counter() + list comprehension
from collections import Counter
 
# initializing dictionary
test_dict = {'Manjeet': [1, 4, 5, 6],
             'Akash': [1, 8, 9],
             'Nikhil': [10, 22, 4],
             'Akshat': [5, 11, 22]}
 
# printing original dictionary
print("The original dictionary : " + str(test_dict))
 
# Remove duplicate values across Dictionary Values
# Using Counter() + list comprehension
cnt = Counter()
for idx in test_dict.values():
    cnt.update(idx)
res = {idx: [key for key in j if cnt[key] == 1]
       for idx, j in test_dict.items()}
 
# printing result
print("Uncommon elements records : " + str(res))


Output

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time complexity: O(n*m), where n is number of keys in dictionary and m is length of the longest list in dictionary.
Auxiliary space: O(n*m), where n is number of keys in dictionary and m is length of the longest list in dictionary.

Method #2 : Using extend(),count(),keys() methods and for loops

Python3




# Python3 code to demonstrate working of
# Remove duplicate values across Dictionary Values
 
# initializing dictionary
test_dict = {'Manjeet': [1, 4, 5, 6],
             'Akash': [1, 8, 9],
             'Nikhil': [10, 22, 4],
             'Akshat': [5, 11, 22]}
 
# printing original dictionary
print("The original dictionary : " + str(test_dict))
 
# Remove duplicate values across Dictionary Values
x = []
for i in test_dict.keys():
    x.extend(test_dict[i])
y = []
for i in x:
    if(x.count(i) == 1):
        y.append(i)
res = dict()
 
for i in test_dict.keys():
    a = []
    for j in test_dict[i]:
        if j in y:
            a.append(j)
        res[i] = a
 
# printing result
print("Uncommon elements records : " + str(res))


Output

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time Complexity: O(n*n)
Auxiliary Space: O(n)

Method #3: Using extend(),operator.countOf(),keys() methods and for loops

Python3




# Python3 code to demonstrate working of
# Remove duplicate values across Dictionary Values
import operator as op
# initializing dictionary
test_dict = {'Manjeet': [1, 4, 5, 6],
             'Akash': [1, 8, 9],
             'Nikhil': [10, 22, 4],
             'Akshat': [5, 11, 22]}
 
# printing original dictionary
print("The original dictionary : " + str(test_dict))
 
# Remove duplicate values across Dictionary Values
x = []
for i in test_dict.keys():
    x.extend(test_dict[i])
y = []
for i in x:
    if(op.countOf(x,i) == 1):
        y.append(i)
res = dict()
 
for i in test_dict.keys():
    a = []
    for j in test_dict[i]:
        if j in y:
            a.append(j)
        res[i] = a
 
# printing result
print("Uncommon elements records : " + str(res))


Output

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time Complexity: O(n*n)
Auxiliary Space: O(n)

Method #4: Using defaultdict() and set()

Step-by-step approach:

  • Import defaultdict from collections
  • Initialize a defaultdict with set as its default_factory
  • Loop through the values of the test_dict
    • Loop through the items of each value
      • Add each item to the corresponding set in the defaultdict
  • Loop through the items of the test_dict
    • Initialize a new list for each item
    • Loop through the items of the value and check if the length of the corresponding set in the defaultdict is 1
      • If the length is 1, append the item to the new list
      • Add the new list to the result dictionary using the item as the key
  • Print the result dictionary

Below is the implementation of the above approach:

Python3




from collections import defaultdict
 
test_dict = {'Manjeet': [1, 4, 5, 6],
             'Akash': [1, 8, 9],
             'Nikhil': [10, 22, 4],
             'Akshat': [5, 11, 22]}
 
print("The original dictionary : " + str(test_dict))
 
d = defaultdict(set)
 
for lst in test_dict.values():
    for item in lst:
        d[item].add(id(lst))
 
res = {}
for k, lst in test_dict.items():
    new_lst = []
    for item in lst:
        if len(d[item]) == 1:
            new_lst.append(item)
    res[k] = new_lst
 
print("Uncommon elements records : " + str(res))


Output

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time complexity: O(n*m), where n is the number of keys in the dictionary and m is the length of the longest list in the dictionary.
Auxiliary space: O(n*m), for the defaultdict and the result dictionary.

Method #5: Using list comprehension and dictionary comprehension

Step-by-step approach:

  • Create a flattened list of all values in the dictionary using list comprehension and the values() method of the dictionary.
  • Create a new list unique_values containing only the values that occur exactly once in the flattened list using list comprehension and the count() method.
  • Create a new dictionary res using dictionary comprehension where each key is a key from the original dictionary and each value is a list of values from the original dictionary that occur in unique_values.

Below is the implementation of the above approach:

Python3




# Python3 code to demonstrate working of
# Remove duplicate values across Dictionary Values
from collections import Counter
 
# initializing dictionary
test_dict = {'Manjeet': [1, 4, 5, 6],
             'Akash': [1, 8, 9],
             'Nikhil': [10, 22, 4],
             'Akshat': [5, 11, 22]}
 
# printing original dictionary
print("The original dictionary : " + str(test_dict))
 
# Remove duplicate values across Dictionary Values
flat_list = [val for sublist in test_dict.values() for val in sublist]
unique_values = [val for val, count in Counter(flat_list).items() if count == 1]
res = {key: [val for val in values if val in unique_values] for key, values in test_dict.items()}
 
# printing result
print("Uncommon elements records : " + str(res))


Output

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time complexity: O(n + m + k), where n is the total number of values in the original dictionary, m is the number of unique values in the original dictionary, and k is the number of keys in the original dictionary.
Auxiliary space: O(n + m + k), where n is the total number of values in the original dictionary, m is the number of unique values in the original dictionary, and k is the number of keys in the original dictionary. 

Method #6:  Using numpy:

Algorithm:

  1. Create an empty dictionary ‘d’ with default values as a set.
  2. Iterate through each value list in the input dictionary and add the id of the list to the set corresponding to each element in the list in the dictionary ‘d’.
  3. Create a new empty dictionary ‘res’.
  4. Iterate through each key-value pair in the input dictionary and for each list in the value, create a new list ‘new_lst’.
  5. For each item in the list, check if its corresponding set in ‘d’ has length 1, indicating it is unique.
  6. If it is unique, append it to the ‘new_lst’.
  7. Add the ‘new_lst’ as the value for the current key in the ‘res’ dictionary.
  8. Print the ‘res’ dictionary as the output.

Python3




import numpy as np
 
test_dict = {'Manjeet': [1, 4, 5, 6],
            'Akash': [1, 8, 9],
            'Nikhil': [10, 22, 4],
            'Akshat': [5, 11, 22]}
 
print("The original dictionary : " + str(test_dict))
 
d = {}
for lst in test_dict.values():
    unique_lst = np.unique(lst)
    for item in unique_lst:
        if item in d:
            d[item] += 1
        else:
            d[item] = 1
 
res = {}
for k, lst in test_dict.items():
    new_lst = []
    for item in lst:
        if d[item] == 1:
            new_lst.append(item)
    res[k] = new_lst
 
print("Uncommon elements records : " + str(res))
#This code is contributed by Jyothi pinjala.


Output:

The original dictionary : {‘Manjeet’: [1, 4, 5, 6], ‘Akash’: [1, 8, 9], ‘Nikhil’: [10, 22, 4], ‘Akshat’: [5, 11, 22]}
Uncommon elements records : {‘Manjeet’: [6], ‘Akash’: [8, 9], ‘Nikhil’: [10], ‘Akshat’: [11]}

Time complexity:
The time complexity of this code is O(n * m), where ‘n’ is the number of keys in the input dictionary and ‘m’ is the maximum number of elements in any list in the dictionary. This is because we iterate through each key-value pair in the input dictionary once and for each element in the lists, we check the corresponding set in the dictionary ‘d’, which has a maximum size of ‘m’.

Auxiliary Space:
The space complexity of this code is O(n * m), where ‘n’ is the number of keys in the input dictionary and ‘m’ is the maximum number of elements in any list in the dictionary. This is because we create a dictionary ‘d’ with ‘n’ keys and each key has a set with maximum size ‘m’. We also create a new dictionary ‘res’ with ‘n’ keys and the maximum length of any list in ‘res’ is also ‘m’.

 



Last Updated : 11 Apr, 2023
Like Article
Save Article
Previous
Next
Share your thoughts in the comments
Similar Reads