ASKSAGE: Sage Q&A Forum - RSS feedhttps://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Fri, 01 Mar 2019 12:38:22 +0100problem loading large filehttps://ask.sagemath.org/question/42124/problem-loading-large-file/I have a file which was generated and saved from a Sage notebook worksheet:
`save(G,'G4D200')`
where `G` is a list of vectors of length roughly 20 million, and the file 'G4D200.sobj` is about 370Mb.
When I try to load this file later to do some analysis
`new_G = load('/path_of_file.../G4D200.sobj')`
my computer goes deep into swap memory (with python eating it all up) and basically freezes. I have done the exact same thing for smaller file sizes (<10Mb) with no issues. I've tried waiting over 30 minutes and nothing seems to happen. Is this possibly a memory issue? Something with Sage? Possible workaround? I'm running the Sage notebook in fedora linux with an older laptop with 8Gb of RAM.
Thanks!Sun, 22 Apr 2018 20:45:13 +0200https://ask.sagemath.org/question/42124/problem-loading-large-file/Comment by imnvsh for <p>I have a file which was generated and saved from a Sage notebook worksheet:</p>
<p><code>save(G,'G4D200')</code></p>
<p>where <code>G</code> is a list of vectors of length roughly 20 million, and the file 'G4D200.sobj` is about 370Mb. </p>
<p>When I try to load this file later to do some analysis</p>
<p><code>new_G = load('/path_of_file.../G4D200.sobj')</code> </p>
<p>my computer goes deep into swap memory (with python eating it all up) and basically freezes. I have done the exact same thing for smaller file sizes (<10Mb) with no issues. I've tried waiting over 30 minutes and nothing seems to happen. Is this possibly a memory issue? Something with Sage? Possible workaround? I'm running the Sage notebook in fedora linux with an older laptop with 8Gb of RAM. </p>
<p>Thanks!</p>
https://ask.sagemath.org/question/42124/problem-loading-large-file/?comment=45587#post-id-45587Is there any update on this matter? I am having the same problem here (the generator was suggested by @slelievre to process further)
v = MatrixSpace(GF(2), 3, 6)
rank = 6
im = identity_matrix(3)
g3 = (c for c in Combinations(len(v), 3) if block_matrix(3, 2, [ [ im,v[c[0]] ], [ im,v[c[1]] ], [ im,v[c[2]] ] ]).rank() >= rank)
g3_ = []
for g in g3:
g3_.append(g)
save(g3_, 'g3_.sobj')
After 7 days, I got the 1.84 GB file (renamed and uploaded by me, no virus, only Sagemath format file): https://ufile.io/kakhi
After downloading/having the file, we load the file:
g3 = load('g3_.sobj')
However, it is too large for my RAM. Is there any solution? We might have to split it before "load". Besides it, better solution to save data is appreciated.Fri, 01 Mar 2019 12:38:22 +0100https://ask.sagemath.org/question/42124/problem-loading-large-file/?comment=45587#post-id-45587