Calculation taking too long, stats from scipy may be the problem?

asked 2012-06-26 23:34:40 -0500

anonymous user


updated 2012-06-27 02:08:08 -0500

DSM gravatar image


I have a question regarding a calculation I'm running. The code includes only 2 loops, one inside another, and some calculus inside. The loop run over many iterations (~100000X100000), but it is not an excuse for the calculation to be taking (so far...!) over 20 hours!

  • The calcultation inside the loops is a fairly simple one: W[k]=W[k]+1/2...+..., but - i also include - sqrt(bla bla bla)stats.norm.rvs . One of the reasons for the ridiculously long calculation time, as I heard, might the calling for library from Scipy, where - as it means calling an outside library, which I don't know how it works, increeses the time of every step in the calculation in such a way that for such many iterations results in the long running time.

Can this be the source of the problem? Can calling an outside library, no matter if - Scipy of whatever else, increase the calculation time by so much?

What can I do?

Thanking you on advance!

edit retag flag offensive close merge delete


- Also, can it be that because Sage and Python are not as basic as programming languages such as C and Fortran etc. - every calculation in them is always bound to take much longer time then in the others??

Erez gravatar imageErez ( 2012-06-26 23:40:04 -0500 )edit

Well, it depends - a lot of stuff interfaces directly with C libraries, and Cython stuff is really more or less as fast as C.

kcrisman gravatar imagekcrisman ( 2012-06-27 03:41:50 -0500 )edit

More to the point, we really need to see the *specific code* in order to profile this. There are also profilers such as %prun on the command line - and see as well. But without the code it's impossible to say.

kcrisman gravatar imagekcrisman ( 2012-06-27 03:42:56 -0500 )edit

O.K, thanks man

Erez gravatar imageErez ( 2012-06-30 21:31:29 -0500 )edit

But you should feel free to edit your original question to show that code! Someone may then be able to help.

kcrisman gravatar imagekcrisman ( 2012-07-02 03:10:28 -0500 )edit