I would like to speed up some calculations by using the @parallel decorator. Unfortunately, it seems to partly disconnect my function from the global name space, but I don't understand in how far and how to fix this. Here are some examples:
This works, as it only uses local variables:
@parallel(ncpus=3, timeout=10)
def f(a,b): return a*b
list(f([(2,3),(3,5),(5,7)]))
[(((2, 3), {}), 6), (((3, 5), {}), 15), (((5, 7), {}), 35)]
Now I want to pass variables in a dictionary, e.g.:
# Now passing variables in a dictionary
var('a b')
def f(vdict):
return (a*b).subs(vdict)
vdict = {a: 1, b: 2}
f(vdict)
However, if I try the same using the @parallel decorator, it complains about the keys in the dictionary:
@parallel(ncpus=2, timeout=10)
def f1(vdict):
return (a*b).subs(vdict)
vdict = {a: 1, b: 2}
vdict2 = {a:3, b:4}
list(f1([vdict, vdict2]))
f1() keywords must be strings
f1() keywords must be strings
[(((), {b: 2, a: 1}), 'NO DATA'), (((), {b: 4, a: 3}), 'NO DATA')]
I thought that his has to do with the following comment in the documentation of @parallel: "The parallel subprocesses will not have access to data created in pexpect interfaces." However, the same comment features in the documentation of @fork, yet it does not complain about passing the dictionary. Even the above works, as long as I only pass one dictionary, so the problem seems to come about when converting to the generator object. Any ideas what the exact problem is and how to get around it? Thanks!
