Is HDF5 or the Python interface h5py supported in Sage?
I am saving very large list objects to disk using the save() command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore save() use, there are legacy 32-bit integers that result in a serious limitation in using save() with even moderately large (hundreds of MB) objects. See OverflowError: size does not fit in an int. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to save() in Sage to save objects to disk?
I will state the obvious and say that Python is part of Sage... could you just use
h5pydirectly from Sage's Python? You would have to install it first but unless it requires Python 3 that should be pretty straightforward usingsage -shand thenpip install, or even the Sage subcommandsage -pip install foo.Looks like Sage Math Cloud has it installed.