Is HDF5 or the Python interface h5py supported in Sage?
I am saving very large list objects to disk using the
save() command in Sage, which is utilizing Python's Pickles package. There is a known deficiency/bug that is unlikely to go away, namely that deeply buried in the compression code in the Python standard library that Pickles and therefore
save() use, there are legacy 32-bit integers that result in a serious limitation in using
save() with even moderately large (hundreds of MB) objects. See OverflowError: size does not fit in an int. The h5py package provides a Python interface to the HDF5 library and HDF5 can deal with multiple terabytes easily. Does anyone know if h5py is/will be implemented in Sage? Is there another alternative to
save() in Sage to save objects to disk?