Ask Your Question
0

How make Notebook not to write in /tmp

asked 9 years ago

ikol gravatar image

I am using Sage Notebook to generate very large files and then process them. The size of the files is tens of Gigabytes but that wouldn't be a problem the way things are set up in my worksheet. However, it seems no matter where the worksheet is stored the associated files are temporarily stored in /tmp whil ethe worksheet is working. For example,

$ ll /raid/istvan/Playground/Sage.sagenb/home/__store__/2/21/212/2123/admin/17/cells/4/
total 8
drwx------ 2 istvan istvan 4096 Sep  4 15:36 ./
drwxrwxr-x 9 istvan istvan 4096 Sep  4 15:34 ../
lrwxrwxrwx 1 istvan istvan   49 Sep  4 15:36 A.mmap -> /tmp/tmpa6CJmh/A.mmap
lrwxrwxrwx 1 istvan istvan   49 Sep  4 15:36 B.mmap -> /tmp/tmpa6CJmh/B.mmap
lrwxrwxrwx 1 istvan istvan   49 Sep  4 15:36 C.mmap -> /tmp/tmpa6CJmh/C.mmap
lrwxrwxrwx 1 istvan istvan   49 Sep  4 15:36 D.mmap -> /tmp/tmpa6CJmh/D.mmap

$ ll /tmp/tmpa6CJmh
total 68860
drwx------  2 istvan istvan        4096 Sep  4 15:35 ./
drwxrwxrwt 14 root   root         12288 Sep  4 15:35 ../
-rw-rw-r--  1 istvan istvan        1688 Sep  4 15:35 ___code___.py
-rw-rw-r--  1 istvan istvan 10000000000 Sep  4 15:37 A.mmap
lrwxrwxrwx  1 istvan istvan          54 Sep  4 15:35 data -> /raid/istvan/Playground/Sage.sagenb/home/admin/17/data/
-rw-rw-r--  1 istvan istvan 10000000000 Sep  4 15:37 B.mmap
-rw-rw-r--  1 istvan istvan 80000000000 Sep  4 15:37 C.mmap
-rw-rw-r--  1 istvan istvan 10000000000 Sep  4 15:37 D.mmap
-rw-rw-r--  1 istvan istvan        2219 Sep  4 15:35 _sage_input_6.py

The total size of the files is about 110 GB and I have plenty of room on the /raid partition where the Sage Notebook resides, but my / including /tmp partition is way too small for that. How can I make the Notebook NOT to write to /tmp? Is there an envar to set that?

Thanks for any suggestion,

Istvan

Preview: (hide)

1 Answer

Sort by » oldest newest most voted
1

answered 9 years ago

nbruin gravatar image

If you just use python-level file access with full filenames, I don't think the notebook ever gets a chance to copy the file, because it will never know of its existence. I don't think the notebook goes digging into the python memory process just to save the data.

You might also try if setting the environment variables TMPDIR or SAGENB_TMPDIR before starting the sage notebook has an effect. According to the documentation of "notebook" it should affect the location where data is stored that needs to be transferred between the notebook server process and the worker process.

Preview: (hide)
link

Comments

I believe that the latter paragraph should be correct, though I've never tried it.

kcrisman gravatar imagekcrisman ( 9 years ago )
1

I tried both and SAGENB_TMPDIR doesn't work, but setting TMPDIR does.

Thanks!

ikol gravatar imageikol ( 9 years ago )

Awesome! If there is something in the documentation that needs to be changed, by the way, please let us know so we can open a ticket.

kcrisman gravatar imagekcrisman ( 9 years ago )

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 9 years ago

Seen: 823 times

Last updated: Sep 04 '15