Python Remove Duplicates from a List

The job is simple. We need to take a list, with duplicate elements in it and generate another list that only contains the element without the duplicates in them.

Examples: 

Input : [2, 4, 10, 20, 5, 2, 20, 4]
Output : [2, 4, 10, 20, 5]

Input : [28, 42, 28, 16, 90, 42, 42, 28]
Output : [28, 42, 16, 90]

We can use not in on list to find out the duplicate items. We create a result list and insert only those that are not already not in. 

Python3

# Python code to remove duplicate elements
def Remove(duplicate):
    final_list = []
    for num in duplicate:
        if num not in final_list:
            final_list.append(num)
    return final_list

# Driver Code
duplicate = [2, 4, 10, 20, 5, 2, 20, 4]
print(Remove(duplicate))

Output:  

[2, 4, 10, 20, 5]

Easy Implementation:

A quick way to do the above using set data structure from the python standard library (Python 3.x implementation is given below)  

Python3

duplicate = [2, 4, 10, 20, 5, 2, 20, 4]
print(list(set(duplicate)))

Output: 

[2, 4, 10, 20, 5] 

Method 2: Using Dictionary/hashmap

Approach:

1. Create a dictionary and by default keep the count of every element to zero, using the default dict.
2. If the count of elements is ZERO, increment the value in the dictionary and continue.
3. If the count of element is greater than zero, Then remove the element from the given list using the remove() method.

You can read more about default dict here.

Python3

from collections import defaultdict


def default_val():
    return 0


# dict : maintain count of each element. with default value of key is 0
mydict = defaultdict(default_val)

# LIST
l = [1, 2, 3, 2, 6, 3, 5, 3, 7, 8]

for i in l:
    # if the element already present in the array, remove the element.
    if mydict[i] == 1:
        l.remove(i)
    # If the elements appears first time keep it count as 1
    else:
        mydict[i] = 1

# printing the final array
print(l)

Output

[1, 2, 6, 5, 3, 7, 8]

Method: Using dict.fromkeys()

Python3

input_list = [1, 2, 3, 2, 6, 3, 5, 3, 7, 8]
mylist = list(dict.fromkeys(input_list))
print(mylist)

Output

[1, 2, 3, 6, 5, 7, 8]

Time complexity:  Average-case time complexity: O(n); n is the size of the array

Worst-case time complexity: O(); n is the size of the array

Auxiliary Space: O(n); n is the size of the array

Method: Using list comprehension: 

Python

list1 = [1, 2, 3, 2, 6, 3, 5, 3, 7, 8]
mylist = [ list1[i] for i in range(len(list1)) if list1.index(list1[i]) == i]
print(mylist)

Output:

[1, 2, 3, 6, 5, 7, 8]

Method: Using Counter() function

Python3

# Python code to remove duplicate elements
from collections import Counter
# Driver Code
duplicate = [2, 4, 10, 20, 5, 2, 20, 4]
unique = Counter(duplicate)
print(list(unique.keys()))

Output

[2, 4, 10, 20, 5]

Time Complexity: O(N)

Auxiliary Space : O(N)

Method: Using operator.countOf() method

Python3

import operator as op
# Python code to remove duplicate elements


def Remove(duplicate):
    final_list = []
    for num in duplicate:
        if op.countOf(final_list, num) == 0:
            final_list.append(num)
    return final_list


# Driver Code
duplicate = [2, 4, 10, 20, 5, 2, 20, 4]
print(Remove(duplicate))

Output

[2, 4, 10, 20, 5]

Time Complexity: O(N)

Auxiliary Space : O(N)