python

Pandas - Remove duplicate items from list

If you are using pandas library in your python project and have a list that contains multiple duplicate items in it. To remove duplicate items from the list you can use unique() function of pandas.

import pandas as pd

source_list = [1, 1, 3, 2, 3, 1, 2, 4]

result = pd.unique(source_list).tolist()

print(result)

Output

[1, 3, 2, 4]

Note that the above method of removing duplicate elements from the list will also maintain the order of elements in the list.

Pandas unique() function

Pandas' unique function can be used to get the unique from a python collection. We are using it here to remove the duplicates from a list. It takes the list as a parameter and we convert it to the list using tolist() function.

Syntax

pd.unique(List).tolist()

Code Example

import pandas as pd

my_list = ['a', 'b', 'a', 'd', 'e', 'c', 'd']

my_list = pd.unique(my_list).tolist()

print(my_list)

Output

['a', 'b', 'd', 'e', 'c']

Was this helpful?