Ask Your Question

Is HDF5 or the Python interface h5py supported in Sage?

asked 2015-03-17 12:37:47 -0500

ikol gravatar image

I am saving very large list objects to disk using the save() command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore save() use, there are legacy 32-bit integers that result in a serious limitation in using save() with even moderately large (hundreds of MB) objects. See OverflowError: size does not fit in an int. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to save() in Sage to save objects to disk?

edit retag flag offensive close merge delete


I will state the obvious and say that Python is part of Sage... could you just use h5py directly from Sage's Python? You would have to install it first but unless it requires Python 3 that should be pretty straightforward using sage -sh and then pip install, or even the Sage subcommand sage -pip install foo.

kcrisman gravatar imagekcrisman ( 2015-03-17 12:47:54 -0500 )edit

Looks like Sage Math Cloud has it installed.

calc314 gravatar imagecalc314 ( 2015-03-17 12:52:46 -0500 )edit

3 answers

Sort by ยป oldest newest most voted

answered 2015-03-17 14:50:48 -0500

tmonteil gravatar image

You can very easily install h5py in Sage, just open a terminal and type:

sage -pip install h5py

Indeed, the cython and numpy dependencies are already satisfied.

edit flag offensive delete link more


Got it. Great, thank you!


ikol gravatar imageikol ( 2015-03-18 08:06:25 -0500 )edit

Glad I was right :-)

kcrisman gravatar imagekcrisman ( 2015-03-18 09:39:56 -0500 )edit

Yep, having pip and most of the scipy stack by default eases many things for using Python-related stuff within Sage.

tmonteil gravatar imagetmonteil ( 2015-03-18 12:31:02 -0500 )edit

answered 2015-04-01 01:48:10 -0500

ikol gravatar image

Of course, hdf5 must be installed first and since it is not a Python package, pip will likely need explicit information about hdf5 libraries and include files. The following command worked for me:

$ sage -pip install --global-option=build_ext --global-option="-L/usr/local/hdf5/lib" --global-option="-l/usr/local/hdf5/lib" --global-option="-I/usr/local/hdf5/include" --global-option="-R/usr/local/hdf5/lib" h5py

edit flag offensive delete link more


How do we install hdf5, then? Just for completeness.

kcrisman gravatar imagekcrisman ( 2015-04-01 12:18:53 -0500 )edit

answered 2015-04-02 07:09:14 -0500

ikol gravatar image

updated 2015-04-02 07:11:17 -0500

Download the package from and the installation is straightforward. On Linux:

1) untar the downloaded file
2) cd to the hdf5 directory
3) ./configure --prefix=/where/you/want/hdf5/to/be/installed (in my case it was /usr/local/hdf5)
4) make
5) make check
6) sudo make install (sudo needed if the location is not in your own user area)
7) sudo make check-install
edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2015-03-17 12:37:47 -0500

Seen: 316 times

Last updated: Apr 01 '15