I am working with SAGE code which calculates many independent instances of a problem and is thus fully parallelizable. Now, I would like to port it to a multi-node cluster environment.
My current idea is to use a main process which manages a problem instance queue from which worker processes fetch instances, solve them and save their results to the file system. In the multi-node environment, I understand that some form of message passing is needed to communicate the problem instances to the workers (ruling out task queue management with the @parallel decorator). I have found that the python package mpi4py (https:// mpi4py.readthedocs.io/en/stable/) provides Python bindings for the Message Passing Interface. It also implements a very convenient MPIPoolExecutor class (https:// mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#mpipoolexecutor) which manages such a task queue.
The current SAGE (v9.3) documentation (https:// doc.sagemath.org/html/en/thematic_tutorials/numerical_sage/mpi4py.html) mentions that mpi4py is supported in SAGE by means of an optional package. However, I do not find it in the list of optional packages for SAGE v9.3 (https:// doc.sagemath.org/html/en/reference/spkg/index.html) and on my current v9.2 install, the method optional_packages() also does not list it (and neither the also required openmpipackage).
Is mpi4py still supported in a current SAGE version? Would it be much effort to try and build it for the current version like it was done for openmpiin this ticket: https:// trac.sagemath.org/ticket/8537?
Or are there other recommendations for task distribution with SAGE in a multi-node environment?
Note: I had to break the links as a new poster.
