Using @parallel in creating a list

asked 2011-06-13 17:04:26 +0100

kcrisman
12242 ●42 ●136 ●255

The question is how to create a list one element at a time using a parallel defined function. I do NOT want to use list comprehension like [f(i) for i in [1..100]] with a parallel defined f. I want to create a list that will still exist in Sage even if I interrupt its calculation before it's done.

It doesn't matter what my function is here. The point is what I'm trying to do here.

sage: @parallel(2)
....: def GMP(n): return (n,GetMinPerimAndSequence(n))
....:
sage: L = []
sage: L.append(GMP([1..100]))

What I intended was that I wanted to run this command with parallel execution, but create a list that would exist even if I interrupt the calculation (such as if it takes a looong time).

sage: L
[<generator object __call__ at 0x4964050>]
sage: L[0]
<generator object __call__ at 0x4964050>

which of course is the same as just making a generator in the first place and using it. So this strategy doesn't work.

answered 2011-06-13 17:56:18 +0100

niles
4529 ●23 ●72 ●133 http://nilesjohnson.net/

The computations are done as you loop over the generator, so if you use a for loop to append to a list (rather than a list comprehension), I think this will do what you want:

sage: @parallel(2)
def GMP(n): sleep(1); return (n,4*n^3 + 2*n)

sage: gen = GMP(range(100))
sage: for t in gen:
    data += [(t[0][0][0],t[1])]
    print "{0} --> {1}".format(t[0][0][0],t[1])
....:  
1 --> (1, 6)
0 --> (0, 0)
2 --> (2, 36)
3 --> (3, 114)
4 --> (4, 264)
5 --> (5, 510)
6 --> (6, 876)
7 --> (7, 1386)
8 --> (8, 2064)
9 --> (9, 2934)
^CKilling any remaining workers...
---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
...
KeyboardInterrupt: 

sage: data.sort()
sage: data
[(0, (0, 0)), (1, (1, 6)), (2, (2, 36)), (3, (3, 114)), (4, (4, 264)), (5, (5, 510)), (6, (6, 876)), (7, (7, 1386)), (8, (8, 2064)), (9, (9, 2934))]

edit flag offensive delete link

Comments

Ah, I know how to do that - I use .append instead. But the problem is that I don't know whether this is actually working in parallel! Is it? Would using .append to data also work?

kcrisman ( 2011-06-13 19:23:42 +0100 )edit

Yes, for both -- if you actually run this, you'll see the messages printed in pairs, as each of the two processes finishes at about the same time. And if you skip the data.sort() step, you'll see that the items are still in the order they were finished. I'm sure .append will work too; I just haven't tried it :)

niles ( 2011-06-14 08:26:21 +0100 )edit

add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Using @parallel in creating a list

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

Using @parallel in creating a list edit

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

Using @parallel in creating a list