Revision history [back]

Python MultiProcessing module stuck for long outputs

I am using SageMath 9.0 and I tried to do paralle computation in two ways

1) parallel decoration built in SageMath;

2) the MultiProcessing module of Python.

When using paralell decoration, everything works fine. When using MultiProcessing module for the same problem with the same input, everything works fine for short output, but there is a problem when the output is long. SageMath gets stuck after the computation if the output is long. I monitored the CPU usage, it peaks at first and then returns to zero, which means that the computation is complete. However, the output still does not appear.

What puzzles me is that the problem depends on the length of output, not the time of computation. For the same computation, once I add a line to manually set the output to be something short, or extract a small part of the original output, then the computation no longer gets stck in the end, and that small part agrees with the original answer.

I would like to know if there is any hidden parameter to prevent the Python MultiProcessing module from producing long outputs in Sagemath.

Python MultiProcessing module stuck for long outputs

I am using SageMath 9.0 and I tried to do paralle computation in two ways

1) parallel decoration built in SageMath;

2) the MultiProcessing module of Python.

I would like to know if there is any hidden parameter to prevent the Python MultiProcessing module from producing long outputs in Sagemath.

P.S. Following the suggestion of @tmonteil, I attached my code with an example.

def f(n):
    return 2 ^ (n ^ n)


def g(n):
    return factor(2 ^ (n ^ n))


def run(function, parameters):
    from multiprocessing import Process, Queue

    def target_function(x, queue):
        queue.put(function(*x))

    results = list()

    if __name__ == "__main__":
        queue = Queue()
        processes = [Process(target=target_function, args=(i, queue)) for i in parameters]
        for p in processes:
            p.start()
        for p in processes:
            p.join()
        results = [queue.get() for p in processes]

    return results

On my computer, the command

run(f,[(3,),(4,),(5,),(6,)])

works fine but the command

run(f,[(7,)])

gets stuck. On the other hand, the command

run(g,[(3,),(4,),(5,),(6,),(7,),(8,),(9,),(10,)])

works fine. Note that the function g does strictly more jobs than f, but its answers are much shorter.