Is HDF5 or the Python interface h5py supported in Sage?

asked 2015-03-17 18:37:47 +0200

ikol
143 ●3 ●10 ●16

I am saving very large list objects to disk using the save() command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore save() use, there are legacy 32-bit integers that result in a serious limitation in using save() with even moderately large (hundreds of MB) objects. See OverflowError: size does not fit in an int. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to save() in Sage to save objects to disk?

edit retag flag offensive close merge delete

Comments

I will state the obvious and say that Python is part of Sage... could you just use h5py directly from Sage's Python? You would have to install it first but unless it requires Python 3 that should be pretty straightforward using sage -sh and then pip install, or even the Sage subcommand sage -pip install foo.

kcrisman ( 2015-03-17 18:47:54 +0200 )edit

1

Looks like Sage Math Cloud has it installed.

calc314 ( 2015-03-17 18:52:46 +0200 )edit

add a comment

2

answered 2015-03-17 20:50:48 +0200

tmonteil
27323 ●31 ●202 ●514 http://wiki.sagemath.o...

You can very easily install h5py in Sage, just open a terminal and type:

sage -pip install h5py

Indeed, the cython and numpy dependencies are already satisfied.

edit flag offensive delete link

Comments

Got it. Great, thank you!

Istvan

ikol ( 2015-03-18 14:06:25 +0200 )edit

Glad I was right :-)

kcrisman ( 2015-03-18 15:39:56 +0200 )edit

Yep, having pip and most of the scipy stack by default eases many things for using Python-related stuff within Sage.

tmonteil ( 2015-03-18 18:31:02 +0200 )edit

add a comment

1

answered 2015-04-01 08:48:10 +0200

ikol
143 ●3 ●10 ●16

Of course, hdf5 must be installed first and since it is not a Python package, pip will likely need explicit information about hdf5 libraries and include files. The following command worked for me:

$ sage -pip install --global-option=build_ext --global-option="-L/usr/local/hdf5/lib" --global-option="-l/usr/local/hdf5/lib" --global-option="-I/usr/local/hdf5/include" --global-option="-R/usr/local/hdf5/lib" h5py

edit flag offensive delete link

Comments

How do we install hdf5, then? Just for completeness.

kcrisman ( 2015-04-01 19:18:53 +0200 )edit

add a comment

0

answered 2015-04-02 14:09:14 +0200

ikol
143 ●3 ●10 ●16

updated 2015-04-02 14:11:17 +0200

Download the package from http://www.hdfgroup.org/downloads/index.html and the installation is straightforward. On Linux:

1) untar the downloaded file
2) cd to the hdf5 directory
3) ./configure --prefix=/where/you/want/hdf5/to/be/installed (in my case it was /usr/local/hdf5)
4) make
5) make check
6) sudo make install (sudo needed if the location is not in your own user area)
7) sudo make check-install

edit flag offensive delete link

add a comment

Is HDF5 or the Python interface h5py supported in Sage?

Comments

3 Answers

Comments

Comments

Your Answer

Question Tools

Stats

Related questions

Is HDF5 or the Python interface h5py supported in Sage? edit

Comments

3 Answers

Comments

Comments

Your Answer

Question Tools

Stats

Related questions

Is HDF5 or the Python interface h5py supported in Sage?