Sagemath heap size limit [closed]

asked 2019-02-06 23:43:15 +0100

imnvsh
35 ●3 ●6 ●10

updated 2019-02-08 23:09:19 +0100

Hi all, I am not new to Python, but new to Sagemath.

My code:

v = MatrixSpace(GF(2), 2, 6)
size = binomial(4096,3)
g3 = []
for c in Combinations(range(4096), 3):
   m = block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]])
   if m.rank() >= 4:
      g3.append(c)

The variable "g3" raises up to about 20 GB in total. Sagemath catches "Memory Error" at some point; therefore, I divide "g3" into 1920 different parts to save. Now I need to process further with "g3", i.e. I need to assign "g3" parts again to some variables to use. The 1st solution that I think of is to create 1920 different variables to store the whole "g3" in my code; however, this way is a bit inconvenient.

Is there any better solution?

For example, increasing the limit of "list" size (python list type []), which might help me to save up to 11.444.858.880 lists in the list "g3" (about 11 billion is the size from binomial(4096,3)). I have a computer of 128 GB RAM, and it is very nice if I can utilize the computer's strength.

There is an old topic on this: trac.sagemath.org/ticket/6772. However, I do not really get the idea there.

I wish to be supported :-) Thank you a lot !!!

edit retag flag offensive reopen merge delete

Closed for the following reason question is not relevant or outdated by imnvsh
close date 2019-02-12 00:14:10.824559

Comments

Why do you need to physically save such an object with an easy generator? We do not have the (maybe $10\times 100$ matrix) v, but even if the range condition makes a rare selection among the many involved c's, what do you further want to do with this list?

dan_fulea ( 2019-02-07 20:07:21 +0100 )edit

Trying to run your code in a fresh Sage session, one gets:

NameError: name 'v' is not defined

slelievre ( 2019-02-07 21:31:01 +0100 )edit

I edited my post adding v:

v = MatrixSpace(GF(2), 2, 6)

@dan_fulea, I would like to share with you further process. Example:

g3 = [(1, 2, 3), (1, 2, 4), (1, 2, 5), (2, 3, 5), (1, 3, 5)]

1) g3 is a list of all c involved in good combinations, i.e. condition m.rank() >= 4 is satisfied (its result is still a huge list, which we can see by running @slelievre suggestion sum(1 for g in g3), which almost non-stop) 2) I process each list in g3 to learn relationship of any pairs, e.g:

relatives =
{(1, 2): [3, 4, 5],
 (1, 3): [2, 5],
 (1, 4): [2],
 (1, 5): [2, 3],
 (2, 3): [1, 5],
 (2, 4): [1],
 (2, 5): [1, 3],
 (3, 5): [2, 1]}

3) Consider (1,2,3), we know (1,2) can go with 4, or 5 besides 3. If all (1 ...(more)

imnvsh ( 2019-02-08 23:51:52 +0100 )edit

add a comment

answered 2019-02-07 21:44:50 +0100

slelievre
17684 ●22 ●160 ●349 http://carva.org/samue...

To expand on @dan_fulea's comment, in general, when we have huge amounts of objects, we don't want to store them in a list!

Using iterators is generally preferred: we can run through an iterator, dealing with its element one at a time without storing all of them at once.

In the case at hand, having defined

sage: C = Combinations(range(4096), 3)

we see that, as you correctly recalled,

sage: C.cardinality()
11444858880

and we could define

sage: g3 = (c for c in C if block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]]).rank() >= 4)

and count these combinations with

sage: sum(1 for g in g3)

and then if we had another thing to study about these combinations we could re-define

sage: g3 = (c for c in C if block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]]).rank() >= 4)

and run through them once more.

edit flag offensive delete link

add a comment

Sagemath heap size limit [closed]

Closed for the following reason question is not relevant or outdated by imnvsh
close date 2019-02-12 00:14:10.824559

Comments

1 Answer

Question Tools

Stats

Related questions

Sagemath heap size limit [closed] edit

Closed for the following reason question is not relevant or outdated by imnvsh close date 2019-02-12 00:14:10.824559

Comments

1 Answer

Question Tools

Stats

Related questions

Sagemath heap size limit [closed]

Closed for the following reason question is not relevant or outdated by imnvsh
close date 2019-02-12 00:14:10.824559