python

Remove duplicate items from a list in Python

If you are working on a list in Python and have a list of items that contains duplicate elements in it. To remove the duplicate elements we can use the methods explained in this post.

# Define a list
numbers = ['a', 'b', 'a', 'c', 'a', 'b']

# Define result list
result = []

# Use list comprehension to remove duplicates
[result.append(x) for x in alphabets if x not in result]

# print the result
print(result)

Output

['a', 'b', 'c']

Removing duplicates from a list can be helpful in many applications where we want to reduce data and want to get distinct records from the list. Here we are explaining some of the methods that can be useful to get the distinct records from the list.

Remove duplicates from a list using list comprehension

We can use list comprehension to remove duplicates from a list. We can define a new list that will contain distinct items and in each iteration, it will check that item does not exist already in the new list.

The new list with distinct records will also maintain the sequence of the item from the input list.

Code Example

alphabets = [4, 4, 5, 9, 4, 3, 6, 9]

result = []

[result.append(x) for x in alphabets if x not in result]

print(result)

Output

[4, 5, 9, 3, 6]

Remove duplicate items using list() and set() function

Another example of removing duplicate items from the list using list() and set() functions. You just need to pass your list to set() function and then convert it to the list..

Not that this method will not maintain the order of items in the list

Syntax

list(set(List))

Code Example

my_list = ['a', 'b', 'a', 'c', 'd', 'a', 'a']

result = list(set(my_list))

print(result)

Output

['c', 'd', 'a', 'b']

Remove duplicate items using For Loop

If you do not want to use any function or extra module to remove duplicate elements from the list then you can use Python For Loop only to do that.

Code Example

lst = [2, 4, 2, 5, 9, 2, 5, 4, 9]

result = []
for item in lst:
  if item not in result:
    result.append(item)

print(result)

Output

[2, 4, 5, 9]

The above method will also maintain the order of items in the final list.

Use of OrderedDict.fromkeys() to remove duplicate items from a list

from collections import OrderedDict

source = [1, 1, 3, 2, 3, 1, 2, 4]
result = list(OrderedDict.fromkeys(source))

print(result)

# -> [1, 3, 2, 4]

We can also use OrderedDict from the collections module in python to get the distinct records from a python list. We are using OrderedDict.fromkeys(list) function to remove the duplication and then convert it to list using the list() function.

Was this helpful?