Performance issues with parallel decoration
Experimenting with @parallel
resulted in unexpected performance issues in Sage 6.4.1. Here is a very simple example:
@parallel(p_iter='multiprocessing', ncpus=6)
def f(n):
return factor(n)
t=walltime()
r = range(1,1000000)
p = sorted(list( f(r)))
print walltime(t)
82.0724880695
t=walltime()
for i in range(1,1000000):
factor(i)
print walltime(t)
12.1648099422
I have 6 physical cores, yet the serial calculation runs more than 6 times faster, even though I can see 6 instances of python running on my computer. Maybe it is pilot error, I have the following questions: 1) Does Sage require a special way of compiling it in order to take full advantage of @parallel? 2) In this case using 'fork' is even worse, it never completes the calculation. 3) How does @parallel distribute the calculations? Since, in general, it takes significantly longer for factor() to process larger numbers, it seems that assigning the case n=1,7,13,... to core_0, n=2,8,14,... to core_1, etc., makes sense. Shuffling the original serial list given to f(n) also seems plausible. However, dividing the whole serial range to 6 intervals and assigning them to the 6 cores, respectively, would be a bad choice and for most of the time only one or two python processes would do anything. Does anyone know what scheme is used in Sage?
Thanks for any suggestions.