Ask Your Question

getting RESetMapReduce to use multiple cores

asked 2020-03-10 04:05:58 +0200

grobber gravatar image

updated 2020-03-10 14:53:08 +0200

I am reading the documentation on how to parallelize recursive-enumeration computations, and I cannot for the life of me get RESetMapReduce to use more than one core of my machine.

I am using the example immediately before the bulleted item titled Generating series on that page, with a minor modification:

sage: from sage.parallel.map_reduce import RESetMapReduce
sage: S = RESetMapReduce(
....:     roots=[[]],
....:     children=lambda l: [l + [0], l + [1]] if len(l) < 32 else [],
....:     map_function=lambda x: 1,
....:     reduce_function=lambda x, y: x + y,
....:     reduce_init=0)

I changed the length to 32 to make the computation heftier. If I run this and watch the process with htop I can see it using only one of my 8 cores to 100% capacity, ignoring the others.

I have even tried passing the argument max_proc=8 to the .run() method, to no effect.

edit retag flag offensive close merge delete

1 Answer

Sort by » oldest newest most voted

answered 2020-03-10 04:16:48 +0200

grobber gravatar image

updated 2020-03-10 12:00:04 +0200

Actually, this worked: I ran the same computation in a terminal after setting the environment variable SAGE_NUM_THREADS to whatever was desired (8 in my case).

All cores are used in that case.

edit flag offensive delete link more



One may also

import os
os.environ["SAGE_NUM_THREADS"] = '16'

as suggested here:

Sébastien gravatar imageSébastien ( 2020-03-10 08:40:14 +0200 )edit

Thanks! I should perhaps have searched more thoroughly; this thread parallels that other one..

grobber gravatar imagegrobber ( 2020-03-10 12:00:48 +0200 )edit

It is a good thing to add to the documentation since this question often comes since ncpus() does not return the number of ncpus anymore. I posted a branch at

Sébastien gravatar imageSébastien ( 2020-03-10 12:23:29 +0200 )edit

Absolutely, that would help! The environment variable is mentioned in passing (twice) on the page I linked, which tipped me off to try to set it, but it's not made clear there that one needs it..

grobber gravatar imagegrobber ( 2020-03-10 14:23:56 +0200 )edit

Oh, and by the way, I tried this just now: if you run this as a sage script (I name mine <blah>.sage, make them executable, and put the shebang line #!/usr/bin/env sage at the top), then you don't actually need to import os: it just works. Maybe sage imports it automatically? I don't know, but I know the explicit import isn't needed.

grobber gravatar imagegrobber ( 2020-03-10 14:49:08 +0200 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2020-03-10 04:05:58 +0200

Seen: 56 times

Last updated: Mar 10 '20