Ask Your Question

How to use cached_function and parallel together

asked 2021-01-29 16:49:37 +0100

Bakerbakura gravatar image

updated 2021-02-01 10:00:40 +0100

I have a function f(a,b,c) which I'd like to save the output values of, so I'm currently using the @cached_function decorator to do so:

def f(a,b,c):
    m = g(a+b)
    n = h(c)
    out = (expensive calculation involving m and m)
    return m+n

def g(d):
    out = (expensive calculation involving d)
    return out

def h(c):
    out = (expensive calculation involving c)
    return out

I would now like to calculate the values of this function for a number of triples (a,b,c) = (a1,b1,c1), (a2,b2,c2), ..., (aN,bN,cN), and since there are many such triples I'd like to calculate them in parallel. From some Googling it seems that I should use the @parallel decorator, but

  1. How do I use the parallel decorator to calculate values of a function which has more than one input parameter?
  2. Is it possible for the calculations of these individual values of f to share memory somehow? The calculation of my function f depends on other expensive functions g and h and thus I have also used the @cached_function decorator on them; it would be preferable for g and h to not be run on the same input if multiple processes are used to perform the calculation of f.
edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2021-05-22 12:30:39 +0100

mwageringel gravatar image

To combine caching with parallel computations, you can use cached_function.precompute (documentation).

def f(a,b,c):
    print("computing f(%s,%s,%s)" % (a,b,c))
    m = g(a+b)
    n = h(c)
    return m+n

def g(d):
    print("computing g(%s)" % d)
    out = d * d  # (expensive calculation involving d)
    return out

def h(c):
    print("computing h(%s)" % c)
    out = c * c * c  # (expensive calculation involving c)
    return out

triples = IntegerListsLex(length=3, max_part=2, element_constructor=tuple).list()
h.precompute(set([c for a,b,c in triples]), num_processes=8)
g.precompute(set([a+b for a,b,c in triples]), num_processes=8)
f.precompute(triples, num_processes=8)
[f(a,b,c) for a,b,c in triples]

The output shows that every function is evaluated only once for each argument:

computing h(0)
computing h(1)
computing h(2)
computing g(0)
computing g(1)
computing g(2)
computing g(3)
computing g(4)
computing f(2,2,2)
computing f(2,2,1)
computing f(2,2,0)
computing f(2,1,2)
computing f(2,1,1)
computing f(2,1,0)
computing f(2,0,2)
computing f(2,0,1)
computing f(2,0,0)
computing f(1,2,2)
computing f(1,2,1)
computing f(1,2,0)
computing f(1,1,2)
computing f(1,1,1)
computing f(1,1,0)
computing f(1,0,2)
computing f(1,0,1)
computing f(1,0,0)
computing f(0,2,2)
computing f(0,2,1)
computing f(0,2,0)
computing f(0,1,2)
computing f(0,1,1)
computing f(0,1,0)
computing f(0,0,2)
computing f(0,0,1)
computing f(0,0,0)
[24, 17, 16, 17, 10, 9, 12, 5, 4, 17, 10, 9, 12, 5, 4, 9, 2, 1, 12, 5, 4, 9, 2, 1, 8, 1, 0]

For this, one needs to know the arguments to h and g in advance, though, which can be difficult. This is not a problem in your example, but I am not sure how to solve this in general.

Regarding your question 1, wrap the arguments in a tuple to use @parallel with multiargs functions.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2021-01-29 16:49:37 +0100

Seen: 236 times

Last updated: May 22 '21