Python List: Remove Duplicates And Keep The Order

Python lists are versatile data structures that allow the storage of multiple elements in a single variable. While lists provide a convenient way to manage collections of data, duplicates within a list can sometimes pose challenges. In this article, we will explore five different methods to remove duplicates from a Python list while preserving the original order.

Remove Duplicates And Keep The Order in Python List

Below are the examples of Remove Duplicates And Keep The Order in Python.

• Using a For Loop
• Using a Set Method
• Using List Comprehension
• Using OrderedDict.fromkeys()
• Using numpy.unique() Method

Remove Duplicates And Keep The Order Using Loop

In this example, the below code initializes a list named `original_list` with integers and prints it. It then creates an empty list `unique_list` and uses a loop to iterate through `original_list`, appending elements to `unique_list` only if they haven’t been added before. This results in an updated list without duplicates.

Python3

original_list = [1, 2, 3, 4, 2, 1, 5, 6, 4]
print("Original List:", original_list)

unique_list = []
for x in original_list:
    if x not in unique_list:
        unique_list.append(x)
print("Update List:", unique_list)

Output

Original List: [1, 2, 3, 4, 2, 1, 5, 6, 4]
Update List: [1, 2, 3, 4, 5, 6]

Remove Duplicates And Keep The Order Using Set

In this example, the below code initializes a list named `original_list` with various integers. It then prints the original list. Subsequently, it creates a `unique_list` by converting the original list to a set to remove duplicates and prints the updated list.

Python3

original_list = [1, 2, 3, 4, 2, 1, 5, 6, 4]
print("Original List:", original_list)

unique_list = list(set(original_list))
print("Update List:", unique_list)

Output

Original List: [1, 2, 3, 4, 2, 1, 5, 6, 4]
Update List: [1, 2, 3, 4, 5, 6]

Remove Duplicates And Keep The Order Using List Comprehension

In this example, below code initializes a list named `original_list` with various integers and prints it. It then creates an empty list `unique_list` and uses a list comprehension to append elements from the original list to `unique_list` only if they haven’t been added before, effectively removing duplicates.

Python3

original_list = [1, 2, 3, 4, 2, 1, 5, 6, 4]
print("Original List:", original_list)

unique_list = []
[unique_list.append(x) for x in original_list if x not in unique_list]
print("Update List:", unique_list)

Output

Original List: [1, 2, 3, 4, 2, 1, 5, 6, 4]
Update List: [1, 2, 3, 4, 5, 6]

Remove Duplicates And Keep The Order Using OrderedDict.fromkeys()

The code utilizes the `OrderedDict` from the `collections` module to remove duplicates from the list `original_list` while preserving the order. It prints the original list and then creates a `unique_list` by converting the keys of an ordered dictionary created from the original list. The result is an updated list without duplicates.

Python3

from collections import OrderedDict

original_list = [1, 2, 3, 4, 2, 1, 5, 6, 4]
print("Original List:", original_list)

unique_list = list(OrderedDict.fromkeys(original_list))
print("Update List:", unique_list)

Output

Original List: [1, 2, 3, 4, 2, 1, 5, 6, 4]
Update List: [1, 2, 3, 4, 5, 6]

Remove Duplicates And Keep The Order Using numpy.unique() Method

In this example, below code employs NumPy to remove duplicates from the list original_list while preserving the order. It prints the original list and then uses np.unique() to obtain unique elements and their indices. The resulting unique_list represents the updated list without duplicates.

Python3

import numpy as np

original_list = [1, 2, 3, 4, 2, 1, 5, 6, 4]
print("Original List:", original_list)

unique_list, indices = np.unique(original_list, return_index=True)
print("Update List:", unique_list)

Output

Original List: [1, 2, 3, 4, 2, 1, 5, 6, 4]
Update List: [1 2 3 4 5 6]

Conclusion

In conclusion, removing duplicates from a Python list while preserving the original order can be achieved through various methods. Whether using the simplicity of sets, concise list comprehensions, OrderedDict, traditional loops, or the NumPy library, developers have a range of efficient options. The choice of method depends on specific requirements and preferences, allowing for flexibility in managing lists with unique elements.