Loading and analyzing data, Datamining

edit

This comes very late, but I thought I would post it anyway for future reference.

Apart from R, there is a pure python-based data mining and visualization package called Orange besides a custom nice frontend, the whole thing works as a python library and can be installed into sage just with

easy_install orange

run from the sage shell.

Many of the operations you listed can be accomplished using pure Python and its standard library. As far as I know, Sage does not provide much more in this area than Python itself.

Custom data structures may be needed for the actual data manipulations. The data structure can be as "dumb" as a two-dimensional nested list (representing records and fields). Or it could be well-packaged using an object-oriented approach. Which one to choose depends on your specific usage case.

Regarding the mundane parsing of CSV (I am assuming "Comma Separated Values") files, there is a Python module called csv. This is more convenient to use, but if for some reason it is not flexible enough, there is also the regular expression module re.

File compression/decompression is provided by the gzip, bz2, zipfile, and tarfile Python modules, for the respective formats.

datetime, time, and calendar provide date/time-related functions.

Although you stated that you wanted to avoid SQL, it might be worth mentioning that Python's standard library also features SQLite in the sqlite3 module.

You might be best off using one of the subcomponents of Sage for this. R in particular is best suited, and has many things going on in the professional data mining community with this.

(That doesn't mean you couldn't use Sage directly, or with Python tools - I'm sure a web search will turn up people using some library!)