# How to use cached_function and parallel together

I have a function f(a,b,c) which I'd like to save the output values of, so I'm currently using the @cached_function decorator to do so:

@cached_function
def f(a,b,c):
m = g(a+b)
n = h(c)
out = (expensive calculation involving m and m)
return m+n

@cached_function
def g(d):
out = (expensive calculation involving d)
return out

@cached_function
def h(c):
out = (expensive calculation involving c)
return out


I would now like to calculate the values of this function for a number of triples (a,b,c) = (a1,b1,c1), (a2,b2,c2), ..., (aN,bN,cN), and since there are many such triples I'd like to calculate them in parallel. From some Googling it seems that I should use the @parallel decorator, but

1. How do I use the parallel decorator to calculate values of a function which has more than one input parameter?
2. Is it possible for the calculations of these individual values of f to share memory somehow? The calculation of my function f depends on other expensive functions g and h and thus I have also used the @cached_function decorator on them; it would be preferable for g and h to not be run on the same input if multiple processes are used to perform the calculation of f.
edit retag close merge delete

Sort by » oldest newest most voted

To combine caching with parallel computations, you can use cached_function.precompute (documentation).

@cached_function
def f(a,b,c):
print("computing f(%s,%s,%s)" % (a,b,c))
m = g(a+b)
n = h(c)
return m+n

@cached_function
def g(d):
print("computing g(%s)" % d)
out = d * d  # (expensive calculation involving d)
return out

@cached_function
def h(c):
print("computing h(%s)" % c)
out = c * c * c  # (expensive calculation involving c)
return out

triples = IntegerListsLex(length=3, max_part=2, element_constructor=tuple).list()
h.precompute(set([c for a,b,c in triples]), num_processes=8)
g.precompute(set([a+b for a,b,c in triples]), num_processes=8)
f.precompute(triples, num_processes=8)
[f(a,b,c) for a,b,c in triples]

The output shows that every function is evaluated only once for each argument:

computing h(0)
computing h(1)
computing h(2)
computing g(0)
computing g(1)
computing g(2)
computing g(3)
computing g(4)
computing f(2,2,2)
computing f(2,2,1)
computing f(2,2,0)
computing f(2,1,2)
computing f(2,1,1)
computing f(2,1,0)
computing f(2,0,2)
computing f(2,0,1)
computing f(2,0,0)
computing f(1,2,2)
computing f(1,2,1)
computing f(1,2,0)
computing f(1,1,2)
computing f(1,1,1)
computing f(1,1,0)
computing f(1,0,2)
computing f(1,0,1)
computing f(1,0,0)
computing f(0,2,2)
computing f(0,2,1)
computing f(0,2,0)
computing f(0,1,2)
computing f(0,1,1)
computing f(0,1,0)
computing f(0,0,2)
computing f(0,0,1)
computing f(0,0,0)
[24, 17, 16, 17, 10, 9, 12, 5, 4, 17, 10, 9, 12, 5, 4, 9, 2, 1, 12, 5, 4, 9, 2, 1, 8, 1, 0]

For this, one needs to know the arguments to h and g in advance, though, which can be difficult. This is not a problem in your example, but I am not sure how to solve this in general.

Regarding your question 1, wrap the arguments in a tuple to use @parallel with multiargs functions.

more