How to use cached_function and parallel together

asked 2021-01-29 16:49:37 +0100

Bakerbakura
91 ●1 ●3 ●10

updated 2021-02-01 10:00:40 +0100

I have a function f(a,b,c) which I'd like to save the output values of, so I'm currently using the @cached_function decorator to do so:

@cached_function
def f(a,b,c):
    m = g(a+b)
    n = h(c)
    out = (expensive calculation involving m and m)
    return m+n

@cached_function
def g(d):
    out = (expensive calculation involving d)
    return out

@cached_function
def h(c):
    out = (expensive calculation involving c)
    return out

I would now like to calculate the values of this function for a number of triples (a,b,c) = (a1,b1,c1), (a2,b2,c2), ..., (aN,bN,cN), and since there are many such triples I'd like to calculate them in parallel. From some Googling it seems that I should use the @parallel decorator, but

How do I use the parallel decorator to calculate values of a function which has more than one input parameter?
Is it possible for the calculations of these individual values of f to share memory somehow? The calculation of my function f depends on other expensive functions g and h and thus I have also used the @cached_function decorator on them; it would be preferable for g and h to not be run on the same input if multiple processes are used to perform the calculation of f.

edit retag flag offensive close merge delete

add a comment

@cached_function def f(a,b,c): print("computing f(%s,%s,%s)" % (a,b,c)) m = g(a+b) n = h(c) return m+n @cached_function def g(d): print("computing g(%s)" % d) out = d * d # (expensive calculation involving d) return out @cached_function def h(c): print("computing h(%s)" % c) out = c * c * c # (expensive calculation involving c) return out triples = IntegerListsLex(length=3, max_part=2, element_constructor=tuple).list() h.precompute(set([c for a,b,c in triples]), num_processes=8) g.precompute(set([a+b for a,b,c in triples]), num_processes=8) f.precompute(triples, num_processes=8) [f(a,b,c) for a,b,c in triples]

computing h(0) computing h(1) computing h(2) computing g(0) computing g(1) computing g(2) computing g(3) computing g(4) computing f(2,2,2) computing f(2,2,1) computing f(2,2,0) computing f(2,1,2) computing f(2,1,1) computing f(2,1,0) computing f(2,0,2) computing f(2,0,1) computing f(2,0,0) computing f(1,2,2) computing f(1,2,1) computing f(1,2,0) computing f(1,1,2) computing f(1,1,1) computing f(1,1,0) computing f(1,0,2) computing f(1,0,1) computing f(1,0,0) computing f(0,2,2) computing f(0,2,1) computing f(0,2,0) computing f(0,1,2) computing f(0,1,1) computing f(0,1,0) computing f(0,0,2) computing f(0,0,1) computing f(0,0,0) [24, 17, 16, 17, 10, 9, 12, 5, 4, 17, 10, 9, 12, 5, 4, 9, 2, 1, 12, 5, 4, 9, 2, 1, 8, 1, 0]

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

How to use cached_function and parallel together

1 Answer

Your Answer

Question Tools

Stats

Related questions

How to use cached_function and parallel together edit

1 Answer