Unique Dictionaries Out Of A List Of Lists?

I have a list called matrix which contains some rows. Each row contains some dictionaries, and each dictionary could be contained in more than one row. I want to generate a list ca

Solution 1:

You could use a list comprehension, but depending on your Python version, using an collections.OrderedDict object with a generator expression to flatten the matrix would actually be more efficient.

When your values are not hashable and thus can't be stored in a set or dictionary, you'll have to use first create an immutable representation, so we can store that representation in a set or dictionary to efficiently track uniqueness.

For dictionaries that are flat structures with all keys and values immutable, just use tuple(sorted(d.items())). This produces a tuple of all (key, value) pairs (also tuples), in sorted order to avoid dictionary order issues.

On Python 3.5 and up, use an OrderedDict() that maps the immutable keys to original dictionaries:

from collections import OrderedDict

key = lambda d: tuple(sorted(d.items()))

dictionaries = list(OrderedDict((key(v), v) for row in matrix for v in row).values())

On Python 3.4 and earlier, OrderedDict is slow and you'd be beter of using a separate set approach for Python 3.4 and below:

key = lambda d: tuple(sorted(d.items()))
seen = set()
seen_add = seen.add
dictionaries = [
    v for row in matrix
    for k, v in ((key(v), v) for v in row)
    if not (k in seen or seen_add(k))]

Quick demo using your input data and an OrderedDict:

>>> from collections import OrderedDict
>>> row1 = [{'NODE':1}, {'NODE':2}, {'NODE':3}]
>>> row2 = [{'NODE':3}, {'NODE':4}, {'NODE':5}]
>>> row3 = [{'NODE':4}, {'NODE':6}, {'NODE':7}]
>>> matrix = [row1, row2, row3]
>>> key = lambda d: tuple(sorted(d.items()))
>>> list(OrderedDict((key(v), v) for row in matrix for v in row).values())
[{'NODE': 1}, {'NODE': 2}, {'NODE': 3}, {'NODE': 4}, {'NODE': 5}, {'NODE': 6}, {'NODE': 7}]

Solution 2:

If you have NumPy:

np.unique(matrix).tolist()

Solution 3:

Flatten the list, then use a set to eliminate dupes.

printset(item for sublist in matrix for item in sublist)