use of global variables in @parallel

asked 2015-10-28 05:34:47 -0500

stan gravatar image

I would like to speed up some calculations by using the @parallel decorator. Unfortunately, it seems to partly disconnect my function from the global name space, but I don't understand in how far and how to fix this. Here are some examples:

This works, as it only uses local variables:

@parallel(ncpus=3, timeout=10)
def f(a,b): return a*b

list(f([(2,3),(3,5),(5,7)]))

[(((2, 3), {}), 6), (((3, 5), {}), 15), (((5, 7), {}), 35)]

Now I want to pass variables in a dictionary, e.g.:

# Now passing variables in a dictionary
var('a b')
def f(vdict): 
    return (a*b).subs(vdict)
vdict = {a: 1, b: 2}
f(vdict)

However, if I try the same using the @parallel decorator, it complains about the keys in the dictionary:

@parallel(ncpus=2, timeout=10)
def f1(vdict): 
    return (a*b).subs(vdict)
vdict = {a: 1, b: 2}
vdict2 = {a:3, b:4}
list(f1([vdict, vdict2]))

f1() keywords must be strings

f1() keywords must be strings

[(((), {b: 2, a: 1}), 'NO DATA'), (((), {b: 4, a: 3}), 'NO DATA')]

I thought that his has to do with the following comment in the documentation of @parallel: "The parallel subprocesses will not have access to data created in pexpect interfaces." However, the same comment features in the documentation of @fork, yet it does not complain about passing the dictionary. Even the above works, as long as I only pass one dictionary, so the problem seems to come about when converting to the generator object. Any ideas what the exact problem is and how to get around it? Thanks!

edit retag flag offensive close merge delete