Ask Your Question
3

Is parallel computation with mpi4py still supported?

asked 2021-06-24 14:45:22 +0200

FabianG gravatar image

updated 2021-06-24 20:36:29 +0200

slelievre gravatar image

I am working with Sage code which calculates many independent instances of a problem and is thus fully parallelizable. Now, I would like to port it to a multi-node cluster environment.

My current idea is to use a main process which manages a problem instance queue from which worker processes fetch instances, solve them and save their results to the file system. In the multi-node environment, I understand that some form of message passing is needed to communicate the problem instances to the workers (ruling out task queue management with the @parallel decorator). I have found that the Python package mpi4py provides Python bindings for the Message Passing Interface. It also implements a very convenient MPIPoolExecutor class which manages such a task queue.

The current Sage (9.3) documentation includes a thematic tutorial on mpyi4py mentioning that mpi4py is supported in Sage by means of an optional package. However, I do not find it in the list of optional packages for Sage 9.3 and on my current v9.2 install, the method optional_packages() also does not list it (and neither the other required openmpi package).

Is mpi4py still supported in a current Sage version? Would it be much effort to try and build it for the current version like it was done for openmpi in Sage Trac ticket 8537?

Or are there other recommendations for task distribution with Sage in a multi-node environment?

edit retag flag offensive close merge delete

Comments

The optional mpi4py package was deleted because it didn't build with Python 3 back then. That is already long ago, and mpi4py now works with Python 3. Could you install openmpi using your system's package manager, and run sage -pip install mpi4py to install mpi4py? I don't know if the optional packages did any other magic, so you should check how far you get with that.

rburing gravatar imagerburing ( 2021-06-25 12:07:21 +0200 )edit

Also, to use Sage stuff in your Python script you should do import sage.all first. Then the above instructions seem to work for me.

rburing gravatar imagerburing ( 2021-06-25 12:27:48 +0200 )edit

Thank you! It did work! I wrote this as an answer, but couldn't accept it, and all the credit should of course go to you :)

FabianG gravatar imageFabianG ( 2021-06-25 16:26:51 +0200 )edit

2 Answers

Sort by ยป oldest newest most voted
1

answered 2021-06-25 16:57:42 +0200

rburing gravatar image

You can install openmpi using your system's package manager, and run

sage -pip install mpi4py

to install mpi4py. Then you can run a Python script, say mpi_1.py, by

mpirun -np 4 ./sage -python mpi_1.py

(assuming the sage binary is in the current directory).

Note that in order to use Sage stuff in a Python script, one should import sage.all first.

edit flag offensive delete link more
1

answered 2021-06-25 16:54:02 +0200

FabianG gravatar image

As suggested by rburing, I installed the (Ubuntu packages) openmpi-bin and openmpi-common via the system package manager and mpi4py with sage -pip install mpi4py. The application can be run according to the mpi4py tutorial. On Ubuntu 18.04, an additional problem was an out of date version of libstdc++. It can be upgraded as explained in this StackOverflow discussion.

$ sudo apt-get install openmpi-bin openmpi-common 
$ sage -pip install mpi4py
$ mpiexec -n 1 sage main_mpi.sage
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2021-06-24 14:40:41 +0200

Seen: 368 times

Last updated: Jun 25 '21