Ask Your Question

HowTo implement filter for multiprocessing module?

asked 2010-08-23 16:23:13 +0200

ccanonc gravatar image

updated 2011-04-28 18:38:16 +0200

Kelvin Li gravatar image

from multiprocessing import Pool
p = Pool(2)

I notice that is implemented, but not p.filter(). What's a good idiom for filter() over multiple processes/cores?

edit retag flag offensive close merge delete


Bug: When using the Interrupt Command, it *does* stop the pool, but the notebook GUI thinks it hasn't, and keeps sending interrupts.

ccanonc gravatar imageccanonc ( 2010-08-25 13:17:44 +0200 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2010-08-23 20:41:51 +0200

Mike Hansen gravatar image

The tricky part about filter is that you typically do something like

filter(lambda x: x % 2 == 0, range(10))

But the multiprocessing module does not support using lambda expressions in things like map, etc. since they are not pickleable by default. That's the first problem. If you're using a function that's not defined in the main session, then you can do something along the following lines:

def pool_filter(pool, func, candidates):
    return [c for c, keep in zip(candidates,, candidates)) if keep]


sage: from multiprocessing import Pool
sage: p = Pool(2)
sage: pool_filter(p, is_prime, range(1, 20))
[2, 3, 5, 7, 11, 13, 17, 19]

Finally, there is some code at #9501 which makes an @fork decorator which allows you to do parallel stuff with lambda expressions. It would be fairly easy to make a parallel filter function that works fine with lambda functions and other functions defined in the main session.

edit flag offensive delete link more


Mike: Does that code prevent pre-generation of lists? Is range like xrange? Can I extend pool to have its own filter with a reasonable amount of flexibility?

ccanonc gravatar imageccanonc ( 2010-08-24 19:25:55 +0200 )edit

The python 3.X docs mention a better integration of the itertools and multiprocessing library. Some of itertools is now the default behavior of built-ins.

ccanonc gravatar imageccanonc ( 2010-09-22 02:05:43 +0200 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2010-08-23 16:23:13 +0200

Seen: 4,552 times

Last updated: Aug 23 '10