Ask Your Question
2

Using Sage's save and load methods leads to high memory usage

asked 2015-08-16 23:38:42 +0200

Fabricio gravatar image

I am using Sage to store and load big matrices and I am monitoring the memory usage of the program with htop in a separated terminal window. (is there a better way to look how the sage is using the memory?)

Apparently, the software expends too much memory when saves and loads objects. Anyone knows what is going on and how to workaround it?


Next, an example code to show it. Commented, the memory used by the system on each step:

# 211 Mb

A = Matrix.random(RealField(200)['u','v','t'], 200)

# 291 Mb (+ 80 Mb)

save(A, 'A')

# 458 Mb (+ 168 Mb)
# A.sobj has size 8.55 Mb on disk

Closing and starting a new session of sage...

# 211 Mb

A = load('A')

# 360 Mb (+ 149 Mb)

So, if the matrix A occupies 80Mb, why almost the double of this is used when saving it? And I don't know how to recuperate it without closing the sage session. Also, why when loading A, again almost the double of the memory is used?


I've found a related question on stackoverflow:

http://stackoverflow.com/questions/20...

But use the open/write method suggested there leads to the same behavior here...

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
2

answered 2015-08-19 06:32:52 +0200

nbruin gravatar image

The python pickle protocol is written to store arbitrarily complicated data structures. It writes a little computer program to reconstruct a data structure, together with the ones it depends on. It's not particularly constructed to be ''efficient'' in the process, because of the generality it needs to work at.

If you have data structures that barely fit in memory you should probably look into more special-purpose data storage methods (scipy/numpy probably have something decent for matrices)

I checked your example and it doesn't seem there is anything particular leaking. So after the pickling/unpickling is done, the memory is available again. Whether python gives pages back to the operating system depends on implementation details there. I would expect that CPython wouldn't be in too much of a hurry to give 80 Mb back. See for instance http://stackoverflow.com/questions/15...

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2015-08-16 23:38:42 +0200

Seen: 1,003 times

Last updated: Aug 19 '15