ASKSAGE: Sage Q&A Forum - RSS feedhttps://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Mon, 22 Apr 2013 12:16:42 +0200Array of graphs consuming too much memory?https://ask.sagemath.org/question/10048/array-of-graphs-consuming-too-much-memory/I want to work in Sage with a large array of small graphs.
To create this array, I load a textfile that has 100000 lines of the following form
gs = []
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10), (0,1,11),(0,1,12),(0,1,13),(0,1,14)]))
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10),(0,1,11),(0,1,12),(0,1,13),(0,2,14)]))
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10),(0,1,11),(0,1,12),(0,1,13),(2,3,14)]))
The textfile occupies a total of 14Mb of diskspace only, but when imported into Sage, the memory usage of the process jumps by nearly 800Mb, and indeed at intermediate stages of the importing, it jumps to over 5Gb before dropping back down again. (Presumably this is due to some sort of memory allocation strategy that keeps doubling the memory required?)
Clearly I am doing something wrong, because an array of 100000 graphs should be no trouble at all to a modern computer, but I don't know **what** I am doing wrong and how to fix it.Mon, 22 Apr 2013 04:20:57 +0200https://ask.sagemath.org/question/10048/array-of-graphs-consuming-too-much-memory/Comment by fidbc for <p>I want to work in Sage with a large array of small graphs.</p>
<p>To create this array, I load a textfile that has 100000 lines of the following form</p>
<pre><code>gs = []
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10), (0,1,11),(0,1,12),(0,1,13),(0,1,14)]))
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10),(0,1,11),(0,1,12),(0,1,13),(0,2,14)]))
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10),(0,1,11),(0,1,12),(0,1,13),(2,3,14)]))
</code></pre>
<p>The textfile occupies a total of 14Mb of diskspace only, but when imported into Sage, the memory usage of the process jumps by nearly 800Mb, and indeed at intermediate stages of the importing, it jumps to over 5Gb before dropping back down again. (Presumably this is due to some sort of memory allocation strategy that keeps doubling the memory required?)</p>
<p>Clearly I am doing something wrong, because an array of 100000 graphs should be no trouble at all to a modern computer, but I don't know <strong>what</strong> I am doing wrong and how to fix it.</p>
https://ask.sagemath.org/question/10048/array-of-graphs-consuming-too-much-memory/?comment=17829#post-id-17829Loading all graphs at the same time may cause that some times. If not all of them are needed at the same time, perhaps iterating through them, one at a time, would be another option.Mon, 22 Apr 2013 12:16:42 +0200https://ask.sagemath.org/question/10048/array-of-graphs-consuming-too-much-memory/?comment=17829#post-id-17829Answer by Nathann for <p>I want to work in Sage with a large array of small graphs.</p>
<p>To create this array, I load a textfile that has 100000 lines of the following form</p>
<pre><code>gs = []
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10), (0,1,11),(0,1,12),(0,1,13),(0,1,14)]))
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10),(0,1,11),(0,1,12),(0,1,13),(0,2,14)]))
gs.append(Graph([(0,1,0),(0,1,1),(0,1,2),(0,1,3),(2,3,4),(2,3,5),(2,3,6),(4,5,7),(4,5,8),(4,5,9),(0,1,10),(0,1,11),(0,1,12),(0,1,13),(2,3,14)]))
</code></pre>
<p>The textfile occupies a total of 14Mb of diskspace only, but when imported into Sage, the memory usage of the process jumps by nearly 800Mb, and indeed at intermediate stages of the importing, it jumps to over 5Gb before dropping back down again. (Presumably this is due to some sort of memory allocation strategy that keeps doubling the memory required?)</p>
<p>Clearly I am doing something wrong, because an array of 100000 graphs should be no trouble at all to a modern computer, but I don't know <strong>what</strong> I am doing wrong and how to fix it.</p>
https://ask.sagemath.org/question/10048/array-of-graphs-consuming-too-much-memory/?answer=14816#post-id-14816Would it help if you were to put this at the head of your file
import gc
Then if you were to put such a line after every 1000th append statement ?
gc.collect()
It is meant to call Python's garbage collector explicitely.
NathannMon, 22 Apr 2013 04:56:01 +0200https://ask.sagemath.org/question/10048/array-of-graphs-consuming-too-much-memory/?answer=14816#post-id-14816