Ask Your Question
1

How to implement a parallelization?

asked 5 years ago

updated 5 years ago

Consider a code of the kind tree exploration amassing fruits: when the procedure arrives to a new node of the tree, if it is a leaf a possible fruit is collected, else a computation is done to determine the children of the node. I would like to parallelize as follows: the children are allocated to all the available CPUs, each CPU has a queue and a given child is allocated to the CPU with the smallest queue.

It seems to be a generic way to parallelize such a tree exploration.

Question: How to implement such a parallelization?
In addition, how to use GPU (for HPC)?

The code has the following form:

cpdef function(list L1, list L2):
    cdef int i,n #...
    cdef list LL1,LL2 #...
    #...
    # core of the code
    #...
    n= #...
    for i in range(n):
        LL1= #...
        LL2= #...
        function(LL1,LL2)
Preview: (hide)

1 Answer

Sort by » oldest newest most voted
2

answered 5 years ago

slelievre gravatar image

Have you looked at map-reduce?

If I understand correctly, it is exactly designed to parallelize such tree operations, using a "work-stealing" algorithm, whose goal is to minimize the communication overhead when parallelizing operations on highly unbalanced objects such as trees.

See also

Preview: (hide)
link

Comments

It seems to be incompatible with Cython, I got the following error: Error compiling Cython file, AttributeError: 'ClosureScope' object has no attribute 'scope_class'. Do you know how to solve that?

Sébastien Palcoux gravatar imageSébastien Palcoux ( 5 years ago )

And do you know how to use GPU?

Sébastien Palcoux gravatar imageSébastien Palcoux ( 5 years ago )

Regarding Cython or GPU, hopefully someone else can chime in.

slelievre gravatar imageslelievre ( 5 years ago )
slelievre gravatar imageslelievre ( 5 years ago )

Without looking at the code I can't debug the Cython problem. I've used map-reduce and Cython in the past and it worked. Please post another question with some minimal code to reproduce the problem.

Hivert gravatar imageHivert ( 5 years ago )

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 5 years ago

Seen: 663 times

Last updated: May 27 '19