ASKSAGE: Sage Q&A Forum - RSS feedhttps://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Sat, 15 Jan 2022 09:25:42 +0100Matrix constructor runs out of memoryhttps://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/I am trying to construct a matrix with entries in $\mathbb{Q}(q)$, from a list of Sage vectors. I do this using
`A = matrix(QQ['q'].fraction_field(), relations)
`
where the vectors in the list `relations` are Sage vectors over `QQ['q'].fraction_field()`.
The list `relations` is generated by a subroutine involving some randomness, and in many cases the above works fine but sometimes my script uses too much memory and is killed by the operating system. Tracing the memory allocation it is clear that the problem occurs at the given line where I construct the matrix from the list of vectors.
The sorts of matrices for which the script fails are tend to have about 100 columns and 50-100 rows. They are sparse (each row has <= 4 nonzero entries) and the nonzero entries are of the form `-2/q^x + 2` where `x` is in the range `10000 - 100000`.
It seems clear that when some combination of the order of magnitude of the exponents `x` and the matrix size is reached, then the matrix constructor uses too much memory. I wondered if there is a solution anyone can suggest, or even just explain a bit more about why this constructor seems to max out on memory?
(P.S. I plan to re-implement things using sparse matrices, and maybe this will help, though I'd still like to understand better why the dense version fails.)Thu, 13 Jan 2022 19:01:24 +0100https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/Comment by Emmanuel Charpentier for <p>I am trying to construct a matrix with entries in $\mathbb{Q}(q)$, from a list of Sage vectors. I do this using</p>
<p><code>A = matrix(QQ['q'].fraction_field(), relations)
</code> </p>
<p>where the vectors in the list <code>relations</code> are Sage vectors over <code>QQ['q'].fraction_field()</code>.</p>
<p>The list <code>relations</code> is generated by a subroutine involving some randomness, and in many cases the above works fine but sometimes my script uses too much memory and is killed by the operating system. Tracing the memory allocation it is clear that the problem occurs at the given line where I construct the matrix from the list of vectors.</p>
<p>The sorts of matrices for which the script fails are tend to have about 100 columns and 50-100 rows. They are sparse (each row has <= 4 nonzero entries) and the nonzero entries are of the form <code>-2/q^x + 2</code> where <code>x</code> is in the range <code>10000 - 100000</code>.</p>
<p>It seems clear that when some combination of the order of magnitude of the exponents <code>x</code> and the matrix size is reached, then the matrix constructor uses too much memory. I wondered if there is a solution anyone can suggest, or even just explain a bit more about why this constructor seems to max out on memory?</p>
<p>(P.S. I plan to re-implement things using sparse matrices, and maybe this will help, though I'd still like to understand better why the dense version fails.)</p>
https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60685#post-id-60685> I'm not familiar with this parameter or using immutables for memory management: what's the idea behind this?
From `matrix?` :
- "immutable" -- (boolean) make the matrix immutable. By default, the output matrix is mutable.
This saves you one level of indirection. Sometimes comes handy... and may point you to bugs (!).
I recommend reading [this free book](https://www.sagemath.org/sagebook/english.html) to get a better understanding of Sage and its underlying design...Sat, 15 Jan 2022 09:25:42 +0100https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60685#post-id-60685Comment by PatrickKinnear for <p>I am trying to construct a matrix with entries in $\mathbb{Q}(q)$, from a list of Sage vectors. I do this using</p>
<p><code>A = matrix(QQ['q'].fraction_field(), relations)
</code> </p>
<p>where the vectors in the list <code>relations</code> are Sage vectors over <code>QQ['q'].fraction_field()</code>.</p>
<p>The list <code>relations</code> is generated by a subroutine involving some randomness, and in many cases the above works fine but sometimes my script uses too much memory and is killed by the operating system. Tracing the memory allocation it is clear that the problem occurs at the given line where I construct the matrix from the list of vectors.</p>
<p>The sorts of matrices for which the script fails are tend to have about 100 columns and 50-100 rows. They are sparse (each row has <= 4 nonzero entries) and the nonzero entries are of the form <code>-2/q^x + 2</code> where <code>x</code> is in the range <code>10000 - 100000</code>.</p>
<p>It seems clear that when some combination of the order of magnitude of the exponents <code>x</code> and the matrix size is reached, then the matrix constructor uses too much memory. I wondered if there is a solution anyone can suggest, or even just explain a bit more about why this constructor seems to max out on memory?</p>
<p>(P.S. I plan to re-implement things using sparse matrices, and maybe this will help, though I'd still like to understand better why the dense version fails.)</p>
https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60680#post-id-60680Hi Emmanuel, thanks for your suggestion. Adding `sparse=True` to the matrix constructor didn't do much, but adding this to the polynomial ring constructor made a huge improvement. Thanks!
The constructor does eventually run out of memory still but now for much larger matrices, so I think I should be able to get enough data for my application.
I added `immutable=True` to the matrix constructor and didn't see huge improvements on top of those coming from using sparse polynomials. I also couldn't seem to pass this keyword to the `FractionField` or `PolynomialRing` constructors. I'm not familiar with this parameter or using immutables for memory management: what's the idea behind this?Fri, 14 Jan 2022 16:33:24 +0100https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60680#post-id-60680Comment by PatrickKinnear for <p>I am trying to construct a matrix with entries in $\mathbb{Q}(q)$, from a list of Sage vectors. I do this using</p>
<p><code>A = matrix(QQ['q'].fraction_field(), relations)
</code> </p>
<p>where the vectors in the list <code>relations</code> are Sage vectors over <code>QQ['q'].fraction_field()</code>.</p>
<p>The list <code>relations</code> is generated by a subroutine involving some randomness, and in many cases the above works fine but sometimes my script uses too much memory and is killed by the operating system. Tracing the memory allocation it is clear that the problem occurs at the given line where I construct the matrix from the list of vectors.</p>
<p>The sorts of matrices for which the script fails are tend to have about 100 columns and 50-100 rows. They are sparse (each row has <= 4 nonzero entries) and the nonzero entries are of the form <code>-2/q^x + 2</code> where <code>x</code> is in the range <code>10000 - 100000</code>.</p>
<p>It seems clear that when some combination of the order of magnitude of the exponents <code>x</code> and the matrix size is reached, then the matrix constructor uses too much memory. I wondered if there is a solution anyone can suggest, or even just explain a bit more about why this constructor seems to max out on memory?</p>
<p>(P.S. I plan to re-implement things using sparse matrices, and maybe this will help, though I'd still like to understand better why the dense version fails.)</p>
https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60679#post-id-60679Hi John: yes, the computation works when the matrices are smaller. The size of the exponent tends to grow as the matrix size does, and having tried Emmanuel's suggestion I think this was the real issue.Fri, 14 Jan 2022 16:28:42 +0100https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60679#post-id-60679Comment by Emmanuel Charpentier for <p>I am trying to construct a matrix with entries in $\mathbb{Q}(q)$, from a list of Sage vectors. I do this using</p>
<p><code>A = matrix(QQ['q'].fraction_field(), relations)
</code> </p>
<p>where the vectors in the list <code>relations</code> are Sage vectors over <code>QQ['q'].fraction_field()</code>.</p>
<p>The list <code>relations</code> is generated by a subroutine involving some randomness, and in many cases the above works fine but sometimes my script uses too much memory and is killed by the operating system. Tracing the memory allocation it is clear that the problem occurs at the given line where I construct the matrix from the list of vectors.</p>
<p>The sorts of matrices for which the script fails are tend to have about 100 columns and 50-100 rows. They are sparse (each row has <= 4 nonzero entries) and the nonzero entries are of the form <code>-2/q^x + 2</code> where <code>x</code> is in the range <code>10000 - 100000</code>.</p>
<p>It seems clear that when some combination of the order of magnitude of the exponents <code>x</code> and the matrix size is reached, then the matrix constructor uses too much memory. I wondered if there is a solution anyone can suggest, or even just explain a bit more about why this constructor seems to max out on memory?</p>
<p>(P.S. I plan to re-implement things using sparse matrices, and maybe this will help, though I'd still like to understand better why the dense version fails.)</p>
https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60678#post-id-60678Stupid question : did you try to pass `sparse=True` to `matrix` ? And to the construction of `FractionField(PolynomialRing(QQ, "q"))` and the construction of the vectors ? Ditto for `immutable`.
Large degree dense polynomials can be unmanageable beasts... OTOH, sparse polynomials may be way slower.Thu, 13 Jan 2022 23:10:48 +0100https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60678#post-id-60678Comment by John Palmieri for <p>I am trying to construct a matrix with entries in $\mathbb{Q}(q)$, from a list of Sage vectors. I do this using</p>
<p><code>A = matrix(QQ['q'].fraction_field(), relations)
</code> </p>
<p>where the vectors in the list <code>relations</code> are Sage vectors over <code>QQ['q'].fraction_field()</code>.</p>
<p>The list <code>relations</code> is generated by a subroutine involving some randomness, and in many cases the above works fine but sometimes my script uses too much memory and is killed by the operating system. Tracing the memory allocation it is clear that the problem occurs at the given line where I construct the matrix from the list of vectors.</p>
<p>The sorts of matrices for which the script fails are tend to have about 100 columns and 50-100 rows. They are sparse (each row has <= 4 nonzero entries) and the nonzero entries are of the form <code>-2/q^x + 2</code> where <code>x</code> is in the range <code>10000 - 100000</code>.</p>
<p>It seems clear that when some combination of the order of magnitude of the exponents <code>x</code> and the matrix size is reached, then the matrix constructor uses too much memory. I wondered if there is a solution anyone can suggest, or even just explain a bit more about why this constructor seems to max out on memory?</p>
<p>(P.S. I plan to re-implement things using sparse matrices, and maybe this will help, though I'd still like to understand better why the dense version fails.)</p>
https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60677#post-id-60677I'm curious: does the construction succeed with smaller matrices?Thu, 13 Jan 2022 21:55:40 +0100https://ask.sagemath.org/question/60673/matrix-constructor-runs-out-of-memory/?comment=60677#post-id-60677