Ask Your Question
0

How make Notebook not to write in /tmp

asked 2015-09-04 22:03:39 +0100

ikol gravatar image

I am using Sage Notebook to generate very large files and then process them. The size of the files is tens of Gigabytes but that wouldn't be a problem the way things are set up in my worksheet. However, it seems no matter where the worksheet is stored the associated files are temporarily stored in /tmp whil ethe worksheet is working. For example,

$ ll /raid/istvan/Playground/Sage.sagenb/home/__store__/2/21/212/2123/admin/17/cells/4/
total 8
drwx------ 2 istvan istvan 4096 Sep  4 15:36 ./
drwxrwxr-x 9 istvan istvan 4096 Sep  4 15:34 ../
lrwxrwxrwx 1 istvan istvan   49 Sep  4 15:36 A.mmap -> /tmp/tmpa6CJmh/A.mmap
lrwxrwxrwx 1 istvan istvan   49 Sep  4 15:36 B.mmap -> /tmp/tmpa6CJmh/B.mmap
lrwxrwxrwx 1 istvan istvan   49 Sep  4 15:36 C.mmap -> /tmp/tmpa6CJmh/C.mmap
lrwxrwxrwx 1 istvan istvan   49 Sep  4 15:36 D.mmap -> /tmp/tmpa6CJmh/D.mmap

$ ll /tmp/tmpa6CJmh
total 68860
drwx------  2 istvan istvan        4096 Sep  4 15:35 ./
drwxrwxrwt 14 root   root         12288 Sep  4 15:35 ../
-rw-rw-r--  1 istvan istvan        1688 Sep  4 15:35 ___code___.py
-rw-rw-r--  1 istvan istvan 10000000000 Sep  4 15:37 A.mmap
lrwxrwxrwx  1 istvan istvan          54 Sep  4 15:35 data -> /raid/istvan/Playground/Sage.sagenb/home/admin/17/data/
-rw-rw-r--  1 istvan istvan 10000000000 Sep  4 15:37 B.mmap
-rw-rw-r--  1 istvan istvan 80000000000 Sep  4 15:37 C.mmap
-rw-rw-r--  1 istvan istvan 10000000000 Sep  4 15:37 D.mmap
-rw-rw-r--  1 istvan istvan        2219 Sep  4 15:35 _sage_input_6.py

The total size of the files is about 110 GB and I have plenty of room on the /raid partition where the Sage Notebook resides, but my / including /tmp partition is way too small for that. How can I make the Notebook NOT to write to /tmp? Is there an envar to set that?

Thanks for any suggestion,

Istvan

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2015-09-04 22:38:51 +0100

nbruin gravatar image

If you just use python-level file access with full filenames, I don't think the notebook ever gets a chance to copy the file, because it will never know of its existence. I don't think the notebook goes digging into the python memory process just to save the data.

You might also try if setting the environment variables TMPDIR or SAGENB_TMPDIR before starting the sage notebook has an effect. According to the documentation of "notebook" it should affect the location where data is stored that needs to be transferred between the notebook server process and the worker process.

edit flag offensive delete link more

Comments

I believe that the latter paragraph should be correct, though I've never tried it.

kcrisman gravatar imagekcrisman ( 2015-09-05 03:07:40 +0100 )edit
1

I tried both and SAGENB_TMPDIR doesn't work, but setting TMPDIR does.

Thanks!

ikol gravatar imageikol ( 2015-09-05 03:21:27 +0100 )edit

Awesome! If there is something in the documentation that needs to be changed, by the way, please let us know so we can open a ticket.

kcrisman gravatar imagekcrisman ( 2015-09-06 03:02:27 +0100 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2015-09-04 22:03:39 +0100

Seen: 708 times

Last updated: Sep 04 '15