# Using @parallel in creating a list

The question is how to create a list one element at a time using a parallel defined function. I do NOT want to use list comprehension like [f(i) for i in [1..100]] with a parallel defined f. I want to create a list that will still exist in Sage even if I interrupt its calculation before it's done.

It doesn't matter what my function is here. The point is what I'm trying to do here.

sage: @parallel(2)
....: def GMP(n): return (n,GetMinPerimAndSequence(n))
....:
sage: L = []
sage: L.append(GMP([1..100]))


What I intended was that I wanted to run this command with parallel execution, but create a list that would exist even if I interrupt the calculation (such as if it takes a looong time).

sage: L
[<generator object __call__ at 0x4964050>]
sage: L
<generator object __call__ at 0x4964050>


which of course is the same as just making a generator in the first place and using it. So this strategy doesn't work.

edit retag close merge delete

Sort by » oldest newest most voted

The computations are done as you loop over the generator, so if you use a for loop to append to a list (rather than a list comprehension), I think this will do what you want:

sage: @parallel(2)
def GMP(n): sleep(1); return (n,4*n^3 + 2*n)

sage: gen = GMP(range(100))
sage: for t in gen:
data += [(t,t)]
print "{0} --> {1}".format(t,t)
....:
1 --> (1, 6)
0 --> (0, 0)
2 --> (2, 36)
3 --> (3, 114)
4 --> (4, 264)
5 --> (5, 510)
6 --> (6, 876)
7 --> (7, 1386)
8 --> (8, 2064)
9 --> (9, 2934)
^CKilling any remaining workers...
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
...
KeyboardInterrupt:

sage: data.sort()
sage: data
[(0, (0, 0)), (1, (1, 6)), (2, (2, 36)), (3, (3, 114)), (4, (4, 264)), (5, (5, 510)), (6, (6, 876)), (7, (7, 1386)), (8, (8, 2064)), (9, (9, 2934))]

more

Ah, I know how to do that - I use .append instead. But the problem is that I don't know whether this is actually working in parallel! Is it? Would using .append to data also work?

Yes, for both -- if you actually run this, you'll see the messages printed in pairs, as each of the two processes finishes at about the same time. And if you skip the data.sort() step, you'll see that the items are still in the order they were finished. I'm sure .append will work too; I just haven't tried it :)