ASKSAGE: Sage Q&A Forum - Individual question feedhttps://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Tue, 18 Aug 2015 23:32:52 -0500Using Sage's save and load methods leads to high memory usagehttps://ask.sagemath.org/question/28850/using-sages-save-and-load-methods-leads-to-high-memory-usage/ I am using Sage to store and load big matrices and I am monitoring the memory usage of the program with htop in a separated terminal window. (is there a better way to look how the sage is using the memory?)
Apparently, the software expends too much memory when saves and loads objects. Anyone knows what is going on and how to workaround it?
----------
Next, an example code to show it. Commented, the memory used by the system on each step:
# 211 Mb
A = Matrix.random(RealField(200)['u','v','t'], 200)
# 291 Mb (+ 80 Mb)
save(A, 'A')
# 458 Mb (+ 168 Mb)
# A.sobj has size 8.55 Mb on disk
Closing and starting a new session of sage...
# 211 Mb
A = load('A')
# 360 Mb (+ 149 Mb)
So, if the matrix A occupies 80Mb, why almost the double of this is used when saving it? And I don't know how to recuperate it without closing the sage session. Also, why when loading A, again almost the double of the memory is used?
----------
I've found a related question on stackoverflow:
http://stackoverflow.com/questions/20294628/using-pythons-pickle-in-sage-results-in-high-memory-usage
But use the open/write method suggested there leads to the same behavior here...Sun, 16 Aug 2015 16:38:42 -0500https://ask.sagemath.org/question/28850/using-sages-save-and-load-methods-leads-to-high-memory-usage/Answer by nbruin for <p>I am using Sage to store and load big matrices and I am monitoring the memory usage of the program with htop in a separated terminal window. (is there a better way to look how the sage is using the memory?)</p>
<p>Apparently, the software expends too much memory when saves and loads objects. Anyone knows what is going on and how to workaround it?</p>
<hr/>
<p>Next, an example code to show it. Commented, the memory used by the system on each step:</p>
<pre><code># 211 Mb
A = Matrix.random(RealField(200)['u','v','t'], 200)
# 291 Mb (+ 80 Mb)
save(A, 'A')
# 458 Mb (+ 168 Mb)
# A.sobj has size 8.55 Mb on disk
</code></pre>
<p>Closing and starting a new session of sage...</p>
<pre><code># 211 Mb
A = load('A')
# 360 Mb (+ 149 Mb)
</code></pre>
<p>So, if the matrix A occupies 80Mb, why almost the double of this is used when saving it? And I don't know how to recuperate it without closing the sage session. Also, why when loading A, again almost the double of the memory is used?</p>
<hr/>
<p>I've found a related question on stackoverflow:</p>
<p><a href="http://stackoverflow.com/questions/20294628/using-pythons-pickle-in-sage-results-in-high-memory-usage">http://stackoverflow.com/questions/20...</a></p>
<p>But use the open/write method suggested there leads to the same behavior here...</p>
https://ask.sagemath.org/question/28850/using-sages-save-and-load-methods-leads-to-high-memory-usage/?answer=28862#post-id-28862 The python pickle protocol is written to store arbitrarily complicated data structures. It writes a little computer program to reconstruct a data structure, together with the ones it depends on. It's not particularly constructed to be ''efficient'' in the process, because of the generality it needs to work at.
If you have data structures that barely fit in memory you should probably look into more special-purpose data storage methods (scipy/numpy probably have something decent for matrices)
I checked your example and it doesn't seem there is anything particular leaking. So after the pickling/unpickling is done, the memory is available again. Whether python gives pages back to the operating system depends on implementation details there. I would expect that CPython wouldn't be in too much of a hurry to give 80 Mb back. See for instance http://stackoverflow.com/questions/15455048/releasing-memory-in-python
Tue, 18 Aug 2015 23:32:52 -0500https://ask.sagemath.org/question/28850/using-sages-save-and-load-methods-leads-to-high-memory-usage/?answer=28862#post-id-28862