ASKSAGE: Sage Q&A Forum - Individual question feedhttp://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Fri, 08 Feb 2019 16:51:52 -0600Sagemath heap size limithttp://ask.sagemath.org/question/45347/sagemath-heap-size-limit/Hi all, I am not new to Python, but new to Sagemath.
My code:
v = MatrixSpace(GF(2), 2, 6)
size = binomial(4096,3)
g3 = []
for c in Combinations(range(4096), 3):
m = block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]])
if m.rank() >= 4:
g3.append(c)
The variable "g3" raises up to about 20 GB in total. Sagemath catches "Memory Error" at some point; therefore, I divide "g3" into 1920 different parts to save. Now I need to process further with "g3", i.e. I need to assign "g3" parts again to some variables to use. The 1st solution that I think of is to create 1920 different variables to store the whole "g3" in my code; however, this way is a bit inconvenient.
Is there any better solution?
For example, increasing the limit of "list" size (python list type []), which might help me to save up to 11.444.858.880 lists in the list "g3" (about 11 billion is the size from binomial(4096,3)). I have a computer of 128 GB RAM, and it is very nice if I can utilize the computer's strength.
There is an old topic on this: trac.sagemath.org/ticket/6772. However, I do not really get the idea there.
I wish to be supported :-) Thank you a lot !!!Wed, 06 Feb 2019 16:43:15 -0600http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/Comment by imnvsh for <p>Hi all, I am not new to Python, but new to Sagemath.</p>
<p>My code:</p>
<pre><code>v = MatrixSpace(GF(2), 2, 6)
size = binomial(4096,3)
g3 = []
for c in Combinations(range(4096), 3):
m = block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]])
if m.rank() >= 4:
g3.append(c)
</code></pre>
<p>The variable "g3" raises up to about 20 GB in total. Sagemath catches "Memory Error" at some point; therefore, I divide "g3" into 1920 different parts to save. Now I need to process further with "g3", i.e. I need to assign "g3" parts again to some variables to use. The 1st solution that I think of is to create 1920 different variables to store the whole "g3" in my code; however, this way is a bit inconvenient. </p>
<p>Is there any better solution? </p>
<p>For example, increasing the limit of "list" size (python list type []), which might help me to save up to 11.444.858.880 lists in the list "g3" (about 11 billion is the size from binomial(4096,3)). I have a computer of 128 GB RAM, and it is very nice if I can utilize the computer's strength. </p>
<p>There is an old topic on this: trac.sagemath.org/ticket/6772. However, I do not really get the idea there.</p>
<p>I wish to be supported :-) Thank you a lot !!!</p>
http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/?comment=45378#post-id-45378I edited my post adding v:
v = MatrixSpace(GF(2), 2, 6)
@dan_fulea, I would like to share with you further process.
Example:
g3 = [(1, 2, 3), (1, 2, 4), (1, 2, 5), (2, 3, 5), (1, 3, 5)]
1) `g3` is a list of all `c` involved in good combinations, i.e. condition `m.rank() >= 4` is satisfied (its result is still a huge list, which we can see by running @slelievre suggestion `sum(1 for g in g3)`, which almost non-stop)
2) I process each list in g3 to learn relationship of any pairs, e.g:
relatives =
{(1, 2): [3, 4, 5],
(1, 3): [2, 5],
(1, 4): [2],
(1, 5): [2, 3],
(2, 3): [1, 5],
(2, 4): [1],
(2, 5): [1, 3],
(3, 5): [2, 1]}
3) Consider (1,2,3), we know (1,2) can go with 4, or 5 besides 3. If all (1,2,5), (1,3,5), (2,3,5) in g3: (1,2,3,5)Fri, 08 Feb 2019 16:51:52 -0600http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/?comment=45378#post-id-45378Comment by slelievre for <p>Hi all, I am not new to Python, but new to Sagemath.</p>
<p>My code:</p>
<pre><code>v = MatrixSpace(GF(2), 2, 6)
size = binomial(4096,3)
g3 = []
for c in Combinations(range(4096), 3):
m = block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]])
if m.rank() >= 4:
g3.append(c)
</code></pre>
<p>The variable "g3" raises up to about 20 GB in total. Sagemath catches "Memory Error" at some point; therefore, I divide "g3" into 1920 different parts to save. Now I need to process further with "g3", i.e. I need to assign "g3" parts again to some variables to use. The 1st solution that I think of is to create 1920 different variables to store the whole "g3" in my code; however, this way is a bit inconvenient. </p>
<p>Is there any better solution? </p>
<p>For example, increasing the limit of "list" size (python list type []), which might help me to save up to 11.444.858.880 lists in the list "g3" (about 11 billion is the size from binomial(4096,3)). I have a computer of 128 GB RAM, and it is very nice if I can utilize the computer's strength. </p>
<p>There is an old topic on this: trac.sagemath.org/ticket/6772. However, I do not really get the idea there.</p>
<p>I wish to be supported :-) Thank you a lot !!!</p>
http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/?comment=45358#post-id-45358Trying to run your code in a fresh Sage session, one gets:
NameError: name 'v' is not definedThu, 07 Feb 2019 14:31:01 -0600http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/?comment=45358#post-id-45358Comment by dan_fulea for <p>Hi all, I am not new to Python, but new to Sagemath.</p>
<p>My code:</p>
<pre><code>v = MatrixSpace(GF(2), 2, 6)
size = binomial(4096,3)
g3 = []
for c in Combinations(range(4096), 3):
m = block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]])
if m.rank() >= 4:
g3.append(c)
</code></pre>
<p>The variable "g3" raises up to about 20 GB in total. Sagemath catches "Memory Error" at some point; therefore, I divide "g3" into 1920 different parts to save. Now I need to process further with "g3", i.e. I need to assign "g3" parts again to some variables to use. The 1st solution that I think of is to create 1920 different variables to store the whole "g3" in my code; however, this way is a bit inconvenient. </p>
<p>Is there any better solution? </p>
<p>For example, increasing the limit of "list" size (python list type []), which might help me to save up to 11.444.858.880 lists in the list "g3" (about 11 billion is the size from binomial(4096,3)). I have a computer of 128 GB RAM, and it is very nice if I can utilize the computer's strength. </p>
<p>There is an old topic on this: trac.sagemath.org/ticket/6772. However, I do not really get the idea there.</p>
<p>I wish to be supported :-) Thank you a lot !!!</p>
http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/?comment=45356#post-id-45356Why do you need to physically save such an object with an easy generator? We do not have the (maybe $10\times 100$ matrix) `v`, but even if the range condition makes a rare selection among the many involved `c`'s, what do you further want to do with this list?Thu, 07 Feb 2019 13:07:21 -0600http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/?comment=45356#post-id-45356Answer by slelievre for <p>Hi all, I am not new to Python, but new to Sagemath.</p>
<p>My code:</p>
<pre><code>v = MatrixSpace(GF(2), 2, 6)
size = binomial(4096,3)
g3 = []
for c in Combinations(range(4096), 3):
m = block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]])
if m.rank() >= 4:
g3.append(c)
</code></pre>
<p>The variable "g3" raises up to about 20 GB in total. Sagemath catches "Memory Error" at some point; therefore, I divide "g3" into 1920 different parts to save. Now I need to process further with "g3", i.e. I need to assign "g3" parts again to some variables to use. The 1st solution that I think of is to create 1920 different variables to store the whole "g3" in my code; however, this way is a bit inconvenient. </p>
<p>Is there any better solution? </p>
<p>For example, increasing the limit of "list" size (python list type []), which might help me to save up to 11.444.858.880 lists in the list "g3" (about 11 billion is the size from binomial(4096,3)). I have a computer of 128 GB RAM, and it is very nice if I can utilize the computer's strength. </p>
<p>There is an old topic on this: trac.sagemath.org/ticket/6772. However, I do not really get the idea there.</p>
<p>I wish to be supported :-) Thank you a lot !!!</p>
http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/?answer=45359#post-id-45359To expand on @dan_fulea's comment, in general, when we have
huge amounts of objects, we don't want to store them in a list!
Using iterators is generally preferred: we can run through an iterator,
dealing with its element one at a time without storing all of them at once.
In the case at hand, having defined
sage: C = Combinations(range(4096), 3)
we see that, as you correctly recalled,
sage: C.cardinality()
11444858880
and we could define
sage: g3 = (c for c in C if block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]]).rank() >= 4)
and count these combinations with
sage: sum(1 for g in g3)
and then if we had another thing to study about these combinations we could re-define
sage: g3 = (c for c in C if block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]]).rank() >= 4)
and run through them once more.Thu, 07 Feb 2019 14:44:50 -0600http://ask.sagemath.org/question/45347/sagemath-heap-size-limit/?answer=45359#post-id-45359