How to remove duplicates from a list in Python
list(set(lst)) is fastest but loses order. list(dict.fromkeys(lst)) removes duplicates AND preserves insertion order.
items = ["a", "b", "a", "c", "b", "d"]
unique_ordered = list(dict.fromkeys(items))
print(unique_ordered) # ['a', 'b', 'c', 'd']dict.fromkeys(lst) builds a dict where each key is a list element (values default to None) — since Python 3.7 dicts preserve insertion order, so keys come out in the order they were first seen. Wrapping in list() gives you the deduped-and-ordered result.
list(set(lst)) is slightly faster but the output order is arbitrary (technically insertion order in CPython's set implementation, but that's not a language guarantee). Only use it when order genuinely doesn't matter.
For unhashable elements (e.g. dicts), fall back to a manual loop with an 'already-seen' set of some hashable identity like a tuple of the fields.
Variations
Fastest, order-loss OK
unique = list(set(items))O(n). Order is not guaranteed.
Preserve order, one-liner
unique = list(dict.fromkeys(items))O(n). Uses Python 3.7+ dict ordering guarantee.
Dedup a list of dicts
seen = set()
unique = []
for row in rows:
key = (row["id"],) # or (row["name"], row["email"])
if key not in seen:
seen.add(key)
unique.append(row)Dicts aren't hashable, so pick a tuple of identifying fields.
Get one Python lesson + one career idea every Friday
No spam, no "buy our course now". Three bullets, every Friday. Unsubscribe with one click.