ASKSAGE: Sage Q&A Forum - Latest question feedhttp://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Mon, 04 Mar 2019 10:08:10 -0600Save/load huge dictionaryhttp://ask.sagemath.org/question/45651/saveload-huge-dictionary/I have a huge dictionary about 50 GB. After generating this dictionary, I do not have any space left on my memory. I still run Sagemath standard `save(mydict, 'myfile')`. However, the save runs almost forever.
What should I do? Storing it in multiple files is also fine to me. I really need to load it again to use in the future.
Maybe another approach is helpful. Besides the above dictionary, I have another huge redundant dictionary `mydict2`, which I tried using `del mydict2` to get some extra memory for the above Sagemath `save`; however, the memory usage still stays the same as before calling `del mydict2`. I guess its keys are still stored in memory. I do not need keys from `mydict2`, but its value is used in `mydict`.imnvshMon, 04 Mar 2019 10:08:10 -0600http://ask.sagemath.org/question/45651/Divide Combinations(n,k) into multiple parts (n > 1110)http://ask.sagemath.org/question/45606/divide-combinationsnk-into-multiple-parts-n-1110/I asked a similar question here: https://ask.sagemath.org/question/45347/sagemath-heap-size-limit/. I closed it, because the answer matched the question there. However, now I have an extra question related to it:
C = Combinations(2110, 3)
C_cardinality = binomial(2110, 3) # 1.563.429.820
N = C_cardinality / 95 # Number of parts = 95 (chunk size)
R = range(0, C_cardinality , N)
Because C_cardinality is too big, I would like to divide C into multiple parts, which are then used on multiple computers to process separately. In order to be sure that these multiple parts are totally different but covering the whole C, I use C's list:
c0 = C.list()[R[0]:R[0]+N]
c1 = C.list()[R[1]:R[1]+N]
c2 = C.list()[R[2]:R[2]+N]
...
However, my computer's RAM (memory 128GB) is not enough to store the `C`'s list. Therefore, I iterate `C` and store in a CSV instead:
def storeCSV(data, fo, wtype='a'):
with open(fo, wtype) as wFile:
write_file = csv.writer(wFile)
write_file.writerow([data])
index = 0
count = 0
for c in C:
storeCSV(c, "part_%s.csv" % index)
count += 1
if count == N:
count = 0
index += 1
print index
Unfortunately, this CSV write is too slow, i.e ~6000 write/min, which means I have to wait about 3800 hours for the whole `C` to finish.
You might wonder why I need to divide `C` into multiple parts. After having these parts, I will run the following code on different computers with a purpose to collect information among set of 3 numbers:
v = MatrixSpace(GF(2), 3,9)
g3 = (c for c in c0 if block_matrix(3, 1, [v[c[0]], v[c[1]], v[c[2]]]).rank() >= 6) # or c1, c2, ..., cN
rel = dict()
for g in list(map(tuple,g3)):
# (1, 2, 3)
rel.setdefault(g[::2], []).append(g[1]) # '[1, 3]': 2
rel.setdefault(g[:2], []).append(g[-1]) # '[1, 2]': 3
rel.setdefault(g[1:], []).append(g[0]) # '[2, 3]': 1
Specifically, I got all Combinations of 3 numbers within `c0, c1,...,cN`. Then, I filter what I need and store in `g3`. For all `g3` of format (A, B, C), I collect (A,B), (B,C), (A,C) as keys and C,A,B as values respectively. The most important reason I must collect this in a Python dictionary is that I need to retrieve any `rel[(X, Y)]` later on.
I wish the problem is described clearly and someone can support. Thank a lot !!!imnvshSat, 02 Mar 2019 18:25:24 -0600http://ask.sagemath.org/question/45606/Loading a 5GB dictionary of matrices uses up all of 64GB RAMhttp://ask.sagemath.org/question/45062/loading-a-5gb-dictionary-of-matrices-uses-up-all-of-64gb-ram/ In a 5GB `file.sage`, I stored a chain complex as a dictionary of sparse matrices (created it in Mathematica with no problems). When I run `load(file.sage)`, the program uses up all 64GB RAM + 64GB swap and crashes. Why does Sage use that much of memory for a small file?
I tried splitting the file into three smaller ones and load one after another, but already with the first 1.3GB file, the system crashes after using all RAM, but without using swap. I get:
---------------------------------------------------------------------------
MemoryError Traceback (most recent call last)
<ipython-input-1-8cecfac681b8> in <module>()
----> 1 load('/home/leon/file.sage');
sage/structure/sage_object.pyx in sage.structure.sage_object.load (build/cythonized/sage/structure/sage_object.c:12879)()
/usr/lib/python2.7/dist-packages/sage/repl/load.pyc in load(filename, globals, attach)
245 if attach:
246 add_attached_file(fpath)
--> 247 exec(preparse_file(open(fpath).read()) + "\n", globals)
248 elif ext == '.spyx' or ext == '.pyx':
249 if attach:
MemoryError: LeonTue, 15 Jan 2019 17:42:46 -0600http://ask.sagemath.org/question/45062/save command in SAGEhttp://ask.sagemath.org/question/10496/save-command-in-sage/I'm trying to save a 'big' rational matrix in SAGE, but I'm running into problems.
After computing my matrix A, which has size 5 x 10,000 and each entry contains rational numbers in fraction form with total number of digits for numerator and denominator more than 10 pages, I run the following command:
save(A, DATA + 'A').
This gives me the following error message:
Traceback(most recent call last):
...
RuntimeError: Segmentation fault.
After this, I can't access A at all any more. In other words, when I try to compute say A[2,3], I get an error. However, before running the save command I was able to compute this entry. Also, I tried the same save command with a 'smaller' matrix and that worked fine. I should also note that I'm using a laptop with 64-bit operating system, x64-based processor, Windows 8, i7 CPU @ 2.40 GHz and 8 GB RAM. Let me know if you need further information. My questions are:
1. Why can't I save my matrix? Why do I get the above error message? What does it mean?
2. How can I save my matrix A? Is there any other way I can save it?Sandi XhumariSun, 01 Sep 2013 11:07:15 -0500http://ask.sagemath.org/question/10496/