ASKSAGE: Sage Q&A Forum - Individual question feedhttp://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Tue, 09 Jul 2013 03:10:12 -0500How to prevent memory leak when solving a linear system of equations using left_kernel ?http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/I am having a problem when running the left_kernel function multiple times. Every time I call the function It takes a new part of the memory although I do not create new variables. I tried finding out where does the memory disappear, but without any luck.
here is an example code:
sage: mat
69 x 70 dense matrix over Symbolic Ring (type 'print mat.str()' to see all of the entries)
sage: get_memory_usage() #memory check before call
1170.34765625
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 1st call
1190.5390625
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 2nd call
1194.73828125
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 3rd call
1217.76953125
As you can see every time I call the function, the memory usage increases. Is there a way to release the memory that was used in a previous call ? My program stops after a few iterations because of lack of memory.
Update: (Creating the matrix "mat")
mat=[]
for Coord in range(len(M_col)):
if(M[M_row[Coord],M_col[Coord]]!=0):
s=M[M_row[Coord],M_col[Coord]]
if(s==1):
temp_v=vector(Poly)(x=a^M_col[Coord],y=FIELDinfoBook[M_row[Coord]])
mat.append(vector(W,temp_v))
else:
up=[i for i in range(s)]
down=list(up)
down.reverse()
for Cup in range(s):
for Cdown in range(s):
if(up[Cup]+down[Cdown]<s):
temp_v=list(zero_vector(sum(Len_Poly)))
for j in range(down[Cdown],l+1):
for i in range(up[Cup],Len_Poly[j]):
comb1=len(Combinations(j,down[Cdown]).list())
comb2=len(Combinations(i,up[Cup]).list())
temp=comb1*comb2*x^(i-up[Cup])*y^(j-down[Cdown])
temp=temp(x=a^M_col[Coord],y=FIELDinfoBook[M_row[Coord]])
temp_v[Len_Poly_inc[j]+i]=(temp)
mat.append(vector(temp_v))
The matrix M is a sparse matrix with integers (mostly ones) at certain positions. M_col and M_row are lists with the locations of nonzero elements.
sage: M
64 x 63 dense matrix over Integer Ring (type 'print M.str()' to see all of the entries)
Poly is a list of bivariate polynomials created like this:
Poly=[]
for j in range(l+1):
for i in range(Len_Poly[j]):
Poly.append(x^i*y^j)
And l=1 , Len_poly=[63, 7] and Len_poly_inc=[0, 63, 70]
Mon, 08 Jul 2013 02:00:19 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/Answer by tmonteil for <p>I am having a problem when running the left_kernel function multiple times. Every time I call the function It takes a new part of the memory although I do not create new variables. I tried finding out where does the memory disappear, but without any luck.
here is an example code:</p>
<pre><code> sage: mat
69 x 70 dense matrix over Symbolic Ring (type 'print mat.str()' to see all of the entries)
sage: get_memory_usage() #memory check before call
1170.34765625
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 1st call
1190.5390625
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 2nd call
1194.73828125
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 3rd call
1217.76953125
</code></pre>
<p>As you can see every time I call the function, the memory usage increases. Is there a way to release the memory that was used in a previous call ? My program stops after a few iterations because of lack of memory.</p>
<p>Update: (Creating the matrix "mat")</p>
<pre><code>mat=[]
for Coord in range(len(M_col)):
if(M[M_row[Coord],M_col[Coord]]!=0):
s=M[M_row[Coord],M_col[Coord]]
if(s==1):
temp_v=vector(Poly)(x=a^M_col[Coord],y=FIELDinfoBook[M_row[Coord]])
mat.append(vector(W,temp_v))
else:
up=[i for i in range(s)]
down=list(up)
down.reverse()
for Cup in range(s):
for Cdown in range(s):
if(up[Cup]+down[Cdown]<s):
temp_v=list(zero_vector(sum(Len_Poly)))
for j in range(down[Cdown],l+1):
for i in range(up[Cup],Len_Poly[j]):
comb1=len(Combinations(j,down[Cdown]).list())
comb2=len(Combinations(i,up[Cup]).list())
temp=comb1*comb2*x^(i-up[Cup])*y^(j-down[Cdown])
temp=temp(x=a^M_col[Coord],y=FIELDinfoBook[M_row[Coord]])
temp_v[Len_Poly_inc[j]+i]=(temp)
mat.append(vector(temp_v))
</code></pre>
<p>The matrix M is a sparse matrix with integers (mostly ones) at certain positions. M_col and M_row are lists with the locations of nonzero elements.</p>
<pre><code>sage: M
64 x 63 dense matrix over Integer Ring (type 'print M.str()' to see all of the entries)
</code></pre>
<p>Poly is a list of bivariate polynomials created like this:</p>
<pre><code>Poly=[]
for j in range(l+1):
for i in range(Len_Poly[j]):
Poly.append(x^i*y^j)
</code></pre>
<p>And l=1 , Len_poly=[63, 7] and Len_poly_inc=[0, 63, 70]</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?answer=15205#post-id-15205You can force the garbage collection as follows:
import gc
gc.enable()
mat = random_matrix(SR,69,70)
for i in range(100):
Inter_mat=mat.transpose()
Solution=Inter_mat.left_kernel()
print get_memory_usage()
gc.collect()
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
....
You can also collect less often, set a frequency, and so on. See [this page](http://docs.python.org/2/library/gc.html).
Mon, 08 Jul 2013 03:05:31 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?answer=15205#post-id-15205Comment by M. H. M. for <p>You can force the garbage collection as follows:</p>
<pre><code>import gc
gc.enable()
mat = random_matrix(SR,69,70)
for i in range(100):
Inter_mat=mat.transpose()
Solution=Inter_mat.left_kernel()
print get_memory_usage()
gc.collect()
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
....
</code></pre>
<p>You can also collect less often, set a frequency, and so on. See <a href="http://docs.python.org/2/library/gc.html">this page</a>.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17331#post-id-17331I start with a list of vectors and then transform it to a matrix.
So I write mat=matrix(W,mat)
instead of mat=matrix(mat).
where:
m=6
q=2^m
P=GF(q,'a')
W.<x> = PolynomialRing(P)Mon, 08 Jul 2013 05:17:33 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17331#post-id-17331Comment by M. H. M. for <p>You can force the garbage collection as follows:</p>
<pre><code>import gc
gc.enable()
mat = random_matrix(SR,69,70)
for i in range(100):
Inter_mat=mat.transpose()
Solution=Inter_mat.left_kernel()
print get_memory_usage()
gc.collect()
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
....
</code></pre>
<p>You can also collect less often, set a frequency, and so on. See <a href="http://docs.python.org/2/library/gc.html">this page</a>.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17334#post-id-17334I changed the type of the elements inside the matrix from Symbolic ring to Univariate Polynomial Ring in x over Finite Field in a of size 2^6. Now it works fine without a leak. I do not know why. Thanks for the help. You gave me the idea to change it.
What should I do now that I have my problem solved. Also I would like to point out the problem when it was Symbolic?
Mon, 08 Jul 2013 05:09:49 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17334#post-id-17334Comment by tmonteil for <p>You can force the garbage collection as follows:</p>
<pre><code>import gc
gc.enable()
mat = random_matrix(SR,69,70)
for i in range(100):
Inter_mat=mat.transpose()
Solution=Inter_mat.left_kernel()
print get_memory_usage()
gc.collect()
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
....
</code></pre>
<p>You can also collect less often, set a frequency, and so on. See <a href="http://docs.python.org/2/library/gc.html">this page</a>.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17332#post-id-17332If there is still a problem with symbolic entries, you should give us details on how to reproduce it, so that we can have a look and eventually report it as a bug.Mon, 08 Jul 2013 05:14:14 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17332#post-id-17332Comment by tmonteil for <p>You can force the garbage collection as follows:</p>
<pre><code>import gc
gc.enable()
mat = random_matrix(SR,69,70)
for i in range(100):
Inter_mat=mat.transpose()
Solution=Inter_mat.left_kernel()
print get_memory_usage()
gc.collect()
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
....
</code></pre>
<p>You can also collect less often, set a frequency, and so on. See <a href="http://docs.python.org/2/library/gc.html">this page</a>.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17330#post-id-17330There is a typo somewhere, `mat=matrix(W,mat)` is a cyclic definition. I got:
NameError: name 'mat' is not defined
Since it seems i cannot reproduce your problem with
mat = random_matrix(SR,69,70)
I need a precise procedure to construct `mat`.
Mon, 08 Jul 2013 05:31:11 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17330#post-id-17330Comment by M. H. M. for <p>You can force the garbage collection as follows:</p>
<pre><code>import gc
gc.enable()
mat = random_matrix(SR,69,70)
for i in range(100):
Inter_mat=mat.transpose()
Solution=Inter_mat.left_kernel()
print get_memory_usage()
gc.collect()
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
....
</code></pre>
<p>You can also collect less often, set a frequency, and so on. See <a href="http://docs.python.org/2/library/gc.html">this page</a>.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17336#post-id-17336I tried it before but it didnt help. This is the result I get when I use your code: (using my mat of course)
1133.46875
1153.5234375
1174.0078125
1178.21875
1198.9296875
.....Mon, 08 Jul 2013 04:33:34 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17336#post-id-17336Comment by tmonteil for <p>You can force the garbage collection as follows:</p>
<pre><code>import gc
gc.enable()
mat = random_matrix(SR,69,70)
for i in range(100):
Inter_mat=mat.transpose()
Solution=Inter_mat.left_kernel()
print get_memory_usage()
gc.collect()
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
1109.7109375
....
</code></pre>
<p>You can also collect less often, set a frequency, and so on. See <a href="http://docs.python.org/2/library/gc.html">this page</a>.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17335#post-id-17335Could you provide your matrix so that i can test further ?Mon, 08 Jul 2013 04:55:18 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17335#post-id-17335Answer by M. H. M. for <p>I am having a problem when running the left_kernel function multiple times. Every time I call the function It takes a new part of the memory although I do not create new variables. I tried finding out where does the memory disappear, but without any luck.
here is an example code:</p>
<pre><code> sage: mat
69 x 70 dense matrix over Symbolic Ring (type 'print mat.str()' to see all of the entries)
sage: get_memory_usage() #memory check before call
1170.34765625
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 1st call
1190.5390625
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 2nd call
1194.73828125
sage: Inter_mat=mat.transpose()
sage: Solution=Inter_mat.left_kernel()
sage: get_memory_usage() #memory check after 3rd call
1217.76953125
</code></pre>
<p>As you can see every time I call the function, the memory usage increases. Is there a way to release the memory that was used in a previous call ? My program stops after a few iterations because of lack of memory.</p>
<p>Update: (Creating the matrix "mat")</p>
<pre><code>mat=[]
for Coord in range(len(M_col)):
if(M[M_row[Coord],M_col[Coord]]!=0):
s=M[M_row[Coord],M_col[Coord]]
if(s==1):
temp_v=vector(Poly)(x=a^M_col[Coord],y=FIELDinfoBook[M_row[Coord]])
mat.append(vector(W,temp_v))
else:
up=[i for i in range(s)]
down=list(up)
down.reverse()
for Cup in range(s):
for Cdown in range(s):
if(up[Cup]+down[Cdown]<s):
temp_v=list(zero_vector(sum(Len_Poly)))
for j in range(down[Cdown],l+1):
for i in range(up[Cup],Len_Poly[j]):
comb1=len(Combinations(j,down[Cdown]).list())
comb2=len(Combinations(i,up[Cup]).list())
temp=comb1*comb2*x^(i-up[Cup])*y^(j-down[Cdown])
temp=temp(x=a^M_col[Coord],y=FIELDinfoBook[M_row[Coord]])
temp_v[Len_Poly_inc[j]+i]=(temp)
mat.append(vector(temp_v))
</code></pre>
<p>The matrix M is a sparse matrix with integers (mostly ones) at certain positions. M_col and M_row are lists with the locations of nonzero elements.</p>
<pre><code>sage: M
64 x 63 dense matrix over Integer Ring (type 'print M.str()' to see all of the entries)
</code></pre>
<p>Poly is a list of bivariate polynomials created like this:</p>
<pre><code>Poly=[]
for j in range(l+1):
for i in range(Len_Poly[j]):
Poly.append(x^i*y^j)
</code></pre>
<p>And l=1 , Len_poly=[63, 7] and Len_poly_inc=[0, 63, 70]</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?answer=15208#post-id-15208I solved my problem by changing the type of the elements from Symbolic ring to Univariate Polynomial Ring in x over Finite Field in a of size 2^6.
However, if it remained as a matrix with elements from the Symbolic ring. The leak would still exist.Mon, 08 Jul 2013 05:16:19 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?answer=15208#post-id-15208Comment by tmonteil for <p>I solved my problem by changing the type of the elements from Symbolic ring to Univariate Polynomial Ring in x over Finite Field in a of size 2^6.</p>
<p>However, if it remained as a matrix with elements from the Symbolic ring. The leak would still exist.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17321#post-id-17321You can just paste it as you did for your first question. The comments field is not very convenient since you can not hit <ENTER>. Another possibility is to edit this question and add your code there. To have a nicely printed code, it suffice to add four spaces before each line of code.Tue, 09 Jul 2013 00:31:48 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17321#post-id-17321Comment by M. H. M. for <p>I solved my problem by changing the type of the elements from Symbolic ring to Univariate Polynomial Ring in x over Finite Field in a of size 2^6.</p>
<p>However, if it remained as a matrix with elements from the Symbolic ring. The leak would still exist.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17319#post-id-17319I added my code in the question. I tried to upload the matrix for you but I do not have enough karmaTue, 09 Jul 2013 03:10:12 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17319#post-id-17319Comment by tmonteil for <p>I solved my problem by changing the type of the elements from Symbolic ring to Univariate Polynomial Ring in x over Finite Field in a of size 2^6.</p>
<p>However, if it remained as a matrix with elements from the Symbolic ring. The leak would still exist.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17329#post-id-17329Since it seems i cannot reproduce your problem with
mat = random_matrix(SR,69,70)
I need a precise procedure to construct a bad matrix over the symbolic ring.
Mon, 08 Jul 2013 06:07:45 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17329#post-id-17329Comment by M. H. M. for <p>I solved my problem by changing the type of the elements from Symbolic ring to Univariate Polynomial Ring in x over Finite Field in a of size 2^6.</p>
<p>However, if it remained as a matrix with elements from the Symbolic ring. The leak would still exist.</p>
http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17323#post-id-17323I am new to this system. How can I paste a code in the comment part. In the answer part I can just click on "Insert code" icon. Mon, 08 Jul 2013 23:48:28 -0500http://ask.sagemath.org/question/10330/how-to-prevent-memory-leak-when-solving-a-linear-system-of-equations-using-left_kernel/?comment=17323#post-id-17323