Strange output using qsub

output

asked 2013-07-10 17:27:58 +0200

castor
324 ●1 ●4 ●16

updated 2013-07-10 17:35:33 +0200

tmonteil
27323 ●31 ●202 ●514 http://wiki.sagemath.o...

I wrote a Sage code and it works fine on my notebook: nohup < QE.sage > QE.txt & the output file QE.txt looks OK. When I try to use the same code on a cluster using qsub -q test.q QE.sh then the output is

----------------------------------------------------------------------
| Sage Version 5.9, Release Date: 2013-04-30                         |
| Type "notebook()" for the browser-based notebook interface.        |
| Type "help()" for help.                                            |
----------------------------------------------------------------------
.[?1034hsage: ....: ....: ....: sage: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ...
.: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ..
..: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: .
...: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....:
....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....:
 ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....
: sage: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ...
.: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: ....: sage: sage: ....: ....: ....: ....: ....: ....: ....:
Exiting Sage (CPU time 0m1.86s, Wall time 0m1.88s).

here QE.sh is as follows

#!/bin/sh
#$ -N QE.sage
module load sage
sage < QE.sage

Also if I use @parallel() on my notebook, then it works as expected:

@parallel(ncpus=2)
def szamol(a,b):
  w=QEllCur([a,b,1])
  if w==-1:
    return "bad input"
  else:
    return w

W=list(szamol([(a,b) for a in range(-2,3) for b in range(-2,3)]))
for X, Y in sorted(W): print X, Y

The CPU time is about 2 sec on my notebook using 2 cores on the cluster using less than 12 cores it prints the first pair and that is it after running (not waiting in the que) more than 2 hours.

My questions are:

how to get the same output using my code on a (linux) cluster as using it on my (linux) notebook?
how to use @parallel() on a cluster?

edit retag flag offensive close merge delete

add a comment

answered 2013-07-12 10:10:53 +0200

niles
4519 ●23 ●72 ●132 http://nilesjohnson.net/

You might need to use the if __name__ == '__main__' guard around the part of the code that initiates the parallel computation, as in:

if __name__ == '__main__':
    W=list(szamol([(a,b) for a in range(-2,3) for b in range(-2,3)]))
    for X, Y in sorted(W): print X, Y

Without it, you might have each child subprocess trying to start the whole parallel computation over again, spawning its own child processes, which in turn try to start the whole thing again.

This is just a guess though; it seems to be required only for some operating systems, and is typically used when parallelizing with multiprocessing instead of fork. It would be confusing if you need it on the cluster but not your local machine.

To continue testing/debugging, I would suggest:

Try using one of the very simple parallel examples in the parallel decorator documentation to verify that it does or doesn't work as expected.
Tell us the content of test.q
Try running the code without <, as in "sage QE.sage" -- this processes the file with sage, but is somewhat different from an interactive session. Is there some reason you know you need to use <?

Comments

Thank you for the hint to run sage QE.sage instead of sage < QE.sage. Now I obtained the correct output. The examples provided in the documentation work well. I modified the following: @parallel def f(a,b): return a*b for X, Y in sorted(list(f([(2,3),(3,5),(5,7)]))): print X, Y since I also have a function in two variables. I also tried to fix one of the variables, but it turned out that the problem is different. In my function I use Pari procedures as well, I try to figure out if that is the problem. I found a nice package mpi4py, that needs more work than @parallel but seems very useful for future computations. I am at the beginning of the road. There are three queues test.q, serial.q and parallel.q I do not know where the relevant files are ...(more)

castor ( 2013-07-12 16:04:49 +0200 )edit

add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Strange output using qsub

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

Strange output using qsub edit

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

Strange output using qsub