ASKSAGE: Sage Q&A Forum - Individual question feedhttp://ask.sagemath.org/questions/Q&A Forum for SageenCopyright Sage, 2010. Some rights reserved under creative commons license.Thu, 02 Apr 2015 07:09:14 -0500Is HDF5 or the Python interface h5py supported in Sage?http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/I am saving very large list objects to disk using the `save()` command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore `save()` use, there are legacy 32-bit integers that result in a serious limitation in using `save()` with even moderately large (hundreds of MB) objects. See [OverflowError: size does not fit in an int](https://github.com/joblib/joblib/issues/122). The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to `save()` in Sage to save objects to disk?Tue, 17 Mar 2015 12:37:47 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/Comment by calc314 for <p>I am saving very large list objects to disk using the <code>save()</code> command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore <code>save()</code> use, there are legacy 32-bit integers that result in a serious limitation in using <code>save()</code> with even moderately large (hundreds of MB) objects. See <a href="https://github.com/joblib/joblib/issues/122">OverflowError: size does not fit in an int</a>. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to <code>save()</code> in Sage to save objects to disk?</p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26228#post-id-26228Looks like Sage Math Cloud has it installed.Tue, 17 Mar 2015 12:52:46 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26228#post-id-26228Comment by kcrisman for <p>I am saving very large list objects to disk using the <code>save()</code> command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore <code>save()</code> use, there are legacy 32-bit integers that result in a serious limitation in using <code>save()</code> with even moderately large (hundreds of MB) objects. See <a href="https://github.com/joblib/joblib/issues/122">OverflowError: size does not fit in an int</a>. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to <code>save()</code> in Sage to save objects to disk?</p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26227#post-id-26227I will state the obvious and say that Python is part of Sage... could you just use `h5py` directly from Sage's Python? You would have to install it first but unless it requires Python 3 that should be pretty straightforward using `sage -sh` and then `pip install`, or even the Sage subcommand `sage -pip install foo`.Tue, 17 Mar 2015 12:47:54 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26227#post-id-26227Answer by tmonteil for <p>I am saving very large list objects to disk using the <code>save()</code> command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore <code>save()</code> use, there are legacy 32-bit integers that result in a serious limitation in using <code>save()</code> with even moderately large (hundreds of MB) objects. See <a href="https://github.com/joblib/joblib/issues/122">OverflowError: size does not fit in an int</a>. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to <code>save()</code> in Sage to save objects to disk?</p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?answer=26231#post-id-26231You can very easily install `h5py` in Sage, just open a terminal and type:
sage -pip install h5py
Indeed, the `cython` and `numpy` dependencies are already satisfied.
Tue, 17 Mar 2015 14:50:48 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?answer=26231#post-id-26231Comment by tmonteil for <p>You can very easily install <code>h5py</code> in Sage, just open a terminal and type:</p>
<pre><code>sage -pip install h5py
</code></pre>
<p>Indeed, the <code>cython</code> and <code>numpy</code> dependencies are already satisfied.</p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26243#post-id-26243Yep, having `pip` and most of the `scipy` stack by default eases many things for using Python-related stuff within Sage.Wed, 18 Mar 2015 12:31:02 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26243#post-id-26243Comment by kcrisman for <p>You can very easily install <code>h5py</code> in Sage, just open a terminal and type:</p>
<pre><code>sage -pip install h5py
</code></pre>
<p>Indeed, the <code>cython</code> and <code>numpy</code> dependencies are already satisfied.</p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26242#post-id-26242Glad I was right :-)Wed, 18 Mar 2015 09:39:56 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26242#post-id-26242Comment by ikol for <p>You can very easily install <code>h5py</code> in Sage, just open a terminal and type:</p>
<pre><code>sage -pip install h5py
</code></pre>
<p>Indeed, the <code>cython</code> and <code>numpy</code> dependencies are already satisfied.</p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26239#post-id-26239Got it. Great, thank you!
IstvanWed, 18 Mar 2015 08:06:25 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26239#post-id-26239Answer by ikol for <p>I am saving very large list objects to disk using the <code>save()</code> command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore <code>save()</code> use, there are legacy 32-bit integers that result in a serious limitation in using <code>save()</code> with even moderately large (hundreds of MB) objects. See <a href="https://github.com/joblib/joblib/issues/122">OverflowError: size does not fit in an int</a>. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to <code>save()</code> in Sage to save objects to disk?</p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?answer=26405#post-id-26405Of course, `hdf5` must be installed first and since it is not a Python package, `pip` will likely need explicit information about `hdf5` libraries and include files. The following command worked for me:
`$ sage -pip install --global-option=build_ext --global-option="-L/usr/local/hdf5/lib" --global-option="-l/usr/local/hdf5/lib" --global-option="-I/usr/local/hdf5/include" --global-option="-R/usr/local/hdf5/lib" h5py`
Wed, 01 Apr 2015 01:48:10 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?answer=26405#post-id-26405Comment by kcrisman for <p>Of course, <code>hdf5</code> must be installed first and since it is not a Python package, <code>pip</code> will likely need explicit information about <code>hdf5</code> libraries and include files. The following command worked for me:</p>
<p><code>$ sage -pip install --global-option=build_ext --global-option="-L/usr/local/hdf5/lib" --global-option="-l/usr/local/hdf5/lib" --global-option="-I/usr/local/hdf5/include" --global-option="-R/usr/local/hdf5/lib" h5py</code></p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26416#post-id-26416How do we install `hdf5`, then? Just for completeness.Wed, 01 Apr 2015 12:18:53 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?comment=26416#post-id-26416Answer by ikol for <p>I am saving very large list objects to disk using the <code>save()</code> command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore <code>save()</code> use, there are legacy 32-bit integers that result in a serious limitation in using <code>save()</code> with even moderately large (hundreds of MB) objects. See <a href="https://github.com/joblib/joblib/issues/122">OverflowError: size does not fit in an int</a>. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to <code>save()</code> in Sage to save objects to disk?</p>
http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?answer=26427#post-id-26427Download the package from `http://www.hdfgroup.org/downloads/index.html` and the installation is straightforward. On Linux:
1) untar the downloaded file
2) cd to the hdf5 directory
3) ./configure --prefix=/where/you/want/hdf5/to/be/installed (in my case it was /usr/local/hdf5)
4) make
5) make check
6) sudo make install (sudo needed if the location is not in your own user area)
7) sudo make check-installThu, 02 Apr 2015 07:09:14 -0500http://ask.sagemath.org/question/26226/is-hdf5-or-the-python-interface-h5py-supported-in-sage/?answer=26427#post-id-26427