# how to remove duplicate elements in a list

From a list L = ([1,2,3,4],[3,5,6,7],[7,8,9,10],[1,2,3.4]) remove duplicate sets here [1,2,3,4] has occurred twice and put only one making L = ([1,2,3,4],[3,5,6,7],[7,8,9,10])

edit retag close merge delete

L is a tuple. For a list, replace the outer parens with square brackets.

( 2010-09-17 16:45:59 -0500 )edit

( 2010-09-17 16:46:58 -0500 )edit

Sort by » oldest newest most voted

Sage has a uniq() function already. However, it requires that the data is hashable, so you can uniq-ify tuples and frozensets but not lists and sets:

sage: L = [(1, 2, 3, 4), (3, 5, 6, 7), (7, 8, 9, 10), (1, 2, 3, 4)]
sage: uniq(L)
[(1, 2, 3, 4), (3, 5, 6, 7), (7, 8, 9, 10)]

more

The page fastest way to uniquify a list in python addresses this; there, it's claimed that the following function is the "best" (fast, and maintains order):

def f5(seq, idfun=None):
# order preserving
if idfun is None:
def idfun(x): return x
seen = {}
result = []
for item in seq:
marker = idfun(item)
# in old Python versions:
# if seen.has_key(marker)
# but in new ones:
if marker in seen: continue
seen[marker] = 1
result.append(item)
return result

more

If you're lazy and don't care about ordering in L:

sage: list(set(L))


Again, all of this is pure Python, so you might benefit from reading a tutorial on writing Python code.

more

Maybe this might help:

sage: L = [[1,2,3,4], [3,5,6,7], [7,8,9,10], [1,2,3,4]]
sage: seen = []
sage: for e in L:
....:     if e in seen:
....:         continue
....:     seen.append(e)
....:
sage: seen
[[1, 2, 3, 4], [3, 5, 6, 7], [7, 8, 9, 10]]

more