Ask Your Question
0

HowTo implement filter for multiprocessing module?

asked 2010-08-23 09:23:13 -0500

ccanonc gravatar image

updated 2011-04-28 11:38:16 -0500

Kelvin Li gravatar image

from multiprocessing import Pool
p = Pool(2)

I notice that p.map() is implemented, but not p.filter(). What's a good idiom for filter() over multiple processes/cores?

edit retag flag offensive close merge delete

Comments

Bug: When using the Interrupt Command, it *does* stop the pool, but the notebook GUI thinks it hasn't, and keeps sending interrupts.

ccanonc gravatar imageccanonc ( 2010-08-25 06:17:44 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
2

answered 2010-08-23 13:41:51 -0500

Mike Hansen gravatar image

The tricky part about filter is that you typically do something like

filter(lambda x: x % 2 == 0, range(10))

But the multiprocessing module does not support using lambda expressions in things like map, etc. since they are not pickleable by default. That's the first problem. If you're using a function that's not defined in the main session, then you can do something along the following lines:

def pool_filter(pool, func, candidates):
    return [c for c, keep in zip(candidates, p.map(func, candidates)) if keep]

and

sage: from multiprocessing import Pool
sage: p = Pool(2)
sage: pool_filter(p, is_prime, range(1, 20))
[2, 3, 5, 7, 11, 13, 17, 19]

Finally, there is some code at #9501 which makes an @fork decorator which allows you to do parallel stuff with lambda expressions. It would be fairly easy to make a parallel filter function that works fine with lambda functions and other functions defined in the main session.

edit flag offensive delete link more

Comments

Mike: Does that code prevent pre-generation of lists? Is range like xrange? Can I extend pool to have its own filter with a reasonable amount of flexibility?

ccanonc gravatar imageccanonc ( 2010-08-24 12:25:55 -0500 )edit

The python 3.X docs mention a better integration of the itertools and multiprocessing library. Some of itertools is now the default behavior of built-ins.

ccanonc gravatar imageccanonc ( 2010-09-21 19:05:43 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2010-08-23 09:23:13 -0500

Seen: 1,994 times

Last updated: Aug 23 '10