ASKSAGE: Sage Q&A Forum - RSS feedhttps://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Mon, 30 Jul 2012 11:48:22 +0200Runaway memory usage in Sage 5.0?https://ask.sagemath.org/question/9180/runaway-memory-usage-in-sage-50/Hi,
I am running Sage 5.0 on Windows 7 (as it is the latest Windows version available) and my code is crashing after a couple of hours of computation. Downgrading to Sage 4.8 fixes the problem. I'm not sure exactly where the issue is so I will try to say as much about what I'm doing as possible.
I am using the algorithm described in this paper:
http://www.springerlink.com/content/1f6xdjt3a7e7h0qn/
to build a database of the lattices of order $n$ up to isomorphism. I am up to $n=12$ so far, and my goal is to reach $n=15$. The program works by generating the lattices of order $n+1$ from the lattices of order $n$.
As such, I am using lots of Posets and LatticePosets. Sage should not have to store in memory more than a thousand or so Posets on $\leq 15$ nodes at any point during the code's execution, and should not have to hold much else in memory beyond these posets. My code takes as input the lattices of order $n$ and writes the lattices of order $n+1$ as it generates them to a file. I am running Sage 5.0 in VirtualBox with 4 processors and 1500MB RAM allocated.
My code uses the @parallel decorator on one function. With this, the overall memory usage of my system climbs rapidly from what it was before (X) to X+1500MB, and after a few hours one of the return values from the parallelized function will be 'NO DATA' (instead of what I expected, which is a short list of posets), which tells me something went wrong. If I remove the @parallel decorator and just call my function with single inputs instead of lists of inputs, the memory usage of my system rises rapidly to X+1500MB and after a few hours the entire Sage virtual machine just shuts down.
However, if I downgrade to Sage 4.8, dedicate 4 processors and only 1250MB RAM to Virtualbox, I can use the @parallel decorator and my code will run stably for hours and eventually complete, without my system ever going over X+1000MB memory usage.
Does anyone have any idea what's going on here? Is Sage 5.0 caching all of the lattices of order $n+1$ that I'm generating along the way and eventually running out of memory or something?Wed, 25 Jul 2012 14:52:05 +0200https://ask.sagemath.org/question/9180/runaway-memory-usage-in-sage-50/Comment by Jason Grout for <p>Hi,</p>
<p>I am running Sage 5.0 on Windows 7 (as it is the latest Windows version available) and my code is crashing after a couple of hours of computation. Downgrading to Sage 4.8 fixes the problem. I'm not sure exactly where the issue is so I will try to say as much about what I'm doing as possible.</p>
<p>I am using the algorithm described in this paper:</p>
<p><a href="http://www.springerlink.com/content/1f6xdjt3a7e7h0qn/">http://www.springerlink.com/content/1...</a></p>
<p>to build a database of the lattices of order $n$ up to isomorphism. I am up to $n=12$ so far, and my goal is to reach $n=15$. The program works by generating the lattices of order $n+1$ from the lattices of order $n$.</p>
<p>As such, I am using lots of Posets and LatticePosets. Sage should not have to store in memory more than a thousand or so Posets on $\leq 15$ nodes at any point during the code's execution, and should not have to hold much else in memory beyond these posets. My code takes as input the lattices of order $n$ and writes the lattices of order $n+1$ as it generates them to a file. I am running Sage 5.0 in VirtualBox with 4 processors and 1500MB RAM allocated.</p>
<p>My code uses the @parallel decorator on one function. With this, the overall memory usage of my system climbs rapidly from what it was before (X) to X+1500MB, and after a few hours one of the return values from the parallelized function will be 'NO DATA' (instead of what I expected, which is a short list of posets), which tells me something went wrong. If I remove the @parallel decorator and just call my function with single inputs instead of lists of inputs, the memory usage of my system rises rapidly to X+1500MB and after a few hours the entire Sage virtual machine just shuts down.</p>
<p>However, if I downgrade to Sage 4.8, dedicate 4 processors and only 1250MB RAM to Virtualbox, I can use the @parallel decorator and my code will run stably for hours and eventually complete, without my system ever going over X+1000MB memory usage. </p>
<p>Does anyone have any idea what's going on here? Is Sage 5.0 caching all of the lattices of order $n+1$ that I'm generating along the way and eventually running out of memory or something?</p>
https://ask.sagemath.org/question/9180/runaway-memory-usage-in-sage-50/?comment=19322#post-id-19322It might be a good idea to post to the sage-combinat mailing list, where it would receive wider attention from those that presumably know a lot more about this area of the code: https://groups.google.com/forum/?fromgroups#!forum/sage-combinat-develMon, 30 Jul 2012 11:48:22 +0200https://ask.sagemath.org/question/9180/runaway-memory-usage-in-sage-50/?comment=19322#post-id-19322Comment by benjaminfjones for <p>Hi,</p>
<p>I am running Sage 5.0 on Windows 7 (as it is the latest Windows version available) and my code is crashing after a couple of hours of computation. Downgrading to Sage 4.8 fixes the problem. I'm not sure exactly where the issue is so I will try to say as much about what I'm doing as possible.</p>
<p>I am using the algorithm described in this paper:</p>
<p><a href="http://www.springerlink.com/content/1f6xdjt3a7e7h0qn/">http://www.springerlink.com/content/1...</a></p>
<p>to build a database of the lattices of order $n$ up to isomorphism. I am up to $n=12$ so far, and my goal is to reach $n=15$. The program works by generating the lattices of order $n+1$ from the lattices of order $n$.</p>
<p>As such, I am using lots of Posets and LatticePosets. Sage should not have to store in memory more than a thousand or so Posets on $\leq 15$ nodes at any point during the code's execution, and should not have to hold much else in memory beyond these posets. My code takes as input the lattices of order $n$ and writes the lattices of order $n+1$ as it generates them to a file. I am running Sage 5.0 in VirtualBox with 4 processors and 1500MB RAM allocated.</p>
<p>My code uses the @parallel decorator on one function. With this, the overall memory usage of my system climbs rapidly from what it was before (X) to X+1500MB, and after a few hours one of the return values from the parallelized function will be 'NO DATA' (instead of what I expected, which is a short list of posets), which tells me something went wrong. If I remove the @parallel decorator and just call my function with single inputs instead of lists of inputs, the memory usage of my system rises rapidly to X+1500MB and after a few hours the entire Sage virtual machine just shuts down.</p>
<p>However, if I downgrade to Sage 4.8, dedicate 4 processors and only 1250MB RAM to Virtualbox, I can use the @parallel decorator and my code will run stably for hours and eventually complete, without my system ever going over X+1000MB memory usage. </p>
<p>Does anyone have any idea what's going on here? Is Sage 5.0 caching all of the lattices of order $n+1$ that I'm generating along the way and eventually running out of memory or something?</p>
https://ask.sagemath.org/question/9180/runaway-memory-usage-in-sage-50/?comment=19341#post-id-19341If you can post the code (or a relevant portion of it) someone might be able to comment. It depends a lot on how the lattices are being computed, how they are stored, read, cached, etc..Wed, 25 Jul 2012 16:09:01 +0200https://ask.sagemath.org/question/9180/runaway-memory-usage-in-sage-50/?comment=19341#post-id-19341Comment by Martin Malandro for <p>Hi,</p>
<p>I am running Sage 5.0 on Windows 7 (as it is the latest Windows version available) and my code is crashing after a couple of hours of computation. Downgrading to Sage 4.8 fixes the problem. I'm not sure exactly where the issue is so I will try to say as much about what I'm doing as possible.</p>
<p>I am using the algorithm described in this paper:</p>
<p><a href="http://www.springerlink.com/content/1f6xdjt3a7e7h0qn/">http://www.springerlink.com/content/1...</a></p>
<p>to build a database of the lattices of order $n$ up to isomorphism. I am up to $n=12$ so far, and my goal is to reach $n=15$. The program works by generating the lattices of order $n+1$ from the lattices of order $n$.</p>
<p>As such, I am using lots of Posets and LatticePosets. Sage should not have to store in memory more than a thousand or so Posets on $\leq 15$ nodes at any point during the code's execution, and should not have to hold much else in memory beyond these posets. My code takes as input the lattices of order $n$ and writes the lattices of order $n+1$ as it generates them to a file. I am running Sage 5.0 in VirtualBox with 4 processors and 1500MB RAM allocated.</p>
<p>My code uses the @parallel decorator on one function. With this, the overall memory usage of my system climbs rapidly from what it was before (X) to X+1500MB, and after a few hours one of the return values from the parallelized function will be 'NO DATA' (instead of what I expected, which is a short list of posets), which tells me something went wrong. If I remove the @parallel decorator and just call my function with single inputs instead of lists of inputs, the memory usage of my system rises rapidly to X+1500MB and after a few hours the entire Sage virtual machine just shuts down.</p>
<p>However, if I downgrade to Sage 4.8, dedicate 4 processors and only 1250MB RAM to Virtualbox, I can use the @parallel decorator and my code will run stably for hours and eventually complete, without my system ever going over X+1000MB memory usage. </p>
<p>Does anyone have any idea what's going on here? Is Sage 5.0 caching all of the lattices of order $n+1$ that I'm generating along the way and eventually running out of memory or something?</p>
https://ask.sagemath.org/question/9180/runaway-memory-usage-in-sage-50/?comment=19340#post-id-19340I'm not sure I could post a relevant portion because I'm not sure where the issue is. The code itself is kind of long. I would be happy to send the code (and the relevant input files) to anyone willing to take a look at it.Wed, 25 Jul 2012 17:13:14 +0200https://ask.sagemath.org/question/9180/runaway-memory-usage-in-sage-50/?comment=19340#post-id-19340