Revision history [back]

I have a huge dictionary about 50 GB. After generating this dictionary, I do not have any space left on my memory. I still run Sagemath standard save(mydict, 'myfile'). However, the save runs almost forever.

What should I do? I really need to load it again to use in the future.

P/S: storing it in multiple files is also fine to me.

I have a huge dictionary about 50 GB. After generating this dictionary, I do not have any space left on my memory. I still run Sagemath standard save(mydict, 'myfile'). However, the save runs almost forever.

What should I do? I really need to load it again to use in the future.

P/S: storing Storing it in multiple files is also fine to me.me. I really need to load it again to use in the future.

Maybe another approach is helpful. Besides the above dictionary, I have another huge redundant dictionary mydict2, which I tried using del mydict2 to get some extra memory for the above Sagemath save; however, the memory usage still stays the same as before calling del mydict2. I guess its keys are still stored in memory. For example:

a = {(1,2):[3]}
x = a.keys()[0]
id(a.keys()[0])
id(x)
del a
print x

After deleting, the value of x is still preserved. The values of this mydict2 are important, because I use them in mydict.

I have a huge dictionary about 50 GB. After generating this dictionary, I do not have any space left on my memory. I still run Sagemath standard save(mydict, 'myfile'). However, the save runs almost forever.

What should I do? Storing it in multiple files is also fine to me. I really need to load it again to use in the future.

Maybe another approach is helpful. Besides the above dictionary, I have another huge redundant dictionary mydict2, which I tried using del mydict2 to get some extra memory for the above Sagemath save; however, the memory usage still stays the same as before calling del mydict2. I guess its keys are still stored in memory. For example:

a d = {(1,2):[3]}
x = a.keys()[0]
id(a.keys()[0])
d.keys()[0]
id(d.keys()[0])
id(x)
del a
d
print x

After deleting, the value of x is still preserved. The values of this mydict2 are important, because I use them in mydict.

I have a huge dictionary about 50 GB. After generating this dictionary, I do not have any space left on my memory. I still run Sagemath standard save(mydict, 'myfile'). However, the save runs almost forever.

What should I do? Storing it in multiple files is also fine to me. I really need to load it again to use in the future.

Maybe another approach is helpful. Besides the above dictionary, I have another huge redundant dictionary mydict2, which I tried using del mydict2 to get some extra memory for the above Sagemath save; however, the memory usage still stays the same as before calling del mydict2. I guess its keys are still stored in memory. For example:

d = {(1,2):[3]}
x = d.keys()[0]
id(d.keys()[0])
id(x)
del d
print x

After deleting, the value of x is still preserved. The values of this mydict2 are important, because I use them in mydict.memory.

I have a huge dictionary about 50 GB. After generating this dictionary, I do not have any space left on my memory. I still run Sagemath standard save(mydict, 'myfile'). However, the save runs almost forever.

What should I do? Storing it in multiple files is also fine to me. I really need to load it again to use in the future.

Maybe another approach is helpful. Besides the above dictionary, I have another huge redundant dictionary mydict2, which I tried using del mydict2 to get some extra memory for the above Sagemath save; however, the memory usage still stays the same as before calling del mydict2. I guess its keys are still stored in memory.memory. I do not need keys from mydict2, but its value is used in mydict.