[feature request] direct pickle to render path

[stas00]

@Da1sypetals, as you have been making this tool great, there is one remaining key feature that can make it the new recommended rendering tool for pytorch mem profiler tracing. Currently Snapviewer requires many steps to get to rendering as compared to download pickle and load it directly in Chrome - immediate rendering there.

The usability and preferring this tool over Chrome would be massively improved if a user could do:

python gui.py  --res 1200 500 -rr 2.0 --pickle foo.pickle

i.e. have the application do all the intermediary steps on its own.

A 2nd issue is that SnapViewer is much slower to prepare the data as compared to an almost instant loading in Chrome (when <1GB pickle).

I'd say pre-processing speed is secondary to getting rid of manual intermediary steps.

But otherwise I'm already using your tool for 1GB+ pickle files since Chrome crashes on those. So super appreciating your work, @Da1sypetals!

[Da1sypetals]

I kind of designed it to be like this on purpose when I started this project months ago, because of the following constraints:

• I wanted to be able to query the call stack by substring, so I used a sqlite database to reuse the ability of SQL, thus avoid reinventing wheels;
• I found that loading the entire database into memory can easily blow up machines with 16GB RAM, making it stuck and not responding (in fact the many memory logging was added when I was dealing with this issue);
• So my design (actually after discussing with many LLMs) was to only load the allocation graphics into memory and put everything else (mainly call stack) on a sqlite db on disk.
  Therefore, to convert the .pickle to a sqlite db, the preprocess step is necessary.

There is definitely some better ways that I am not aware of, so maybe we could discuss here to come up with an approach on how to implement the direct pickle-to-render path. I would appreciate your opinion! ❤️ @stas00

[stas00]

I hear you, @Da1sypetals. I think my request wasn't fully clear then. What I was suggesting is that SnapViewer will take .pickle as an input and do all the preprocessing behind the scenes - so I'm not proposing to change anything but to add a wrapper that can automate all the intermediate steps, does it make sense?

Surely a user could write such a wrapper themselves, but what I'm trying to communicate is that by allowing taking .pickle file directly you'd bring the ease of use to the same level as dropping .pickle into Chrome.

So basically folding the current 3 steps -> convert+unzip+render - with perhaps a caching should the same .pickle be passed again. And I feel the unzip stage can be eliminated altogether by writing out the target dir directly. what's the point of zipping something if it's immediately getting unzip'ed, no? This will save quite a lot of overhead as well.

Please let me know what you think

[Da1sypetals]

I think I get most of your ideas. For the caching part, would it be a better practice to let users specify which directory to store the caches, or store everything under somewhere like ~/.snapviewer_cache ？Or do you have other suggestions 🌹 @stas00

[stas00]

Have a default path like you proposed and let users override if they need to?

To cache correctly you'd need some sort of light checksum on the pickle file since typically at least in my use case it's the same file name, but different contents every the time - as I re-run the workload and overwrite the previous pickle. md5sum would be very slow, there should be some much faster version out there, we don't care for cryptography-level checksum'ing - just something that can tell 2 files with the same name apart.

I appreciate your engagement, @Da1sypetals - this is going to be a super useful application for pytorch users.

[Da1sypetals]

Implemented in branch pickle2renderer
--pickle flag is now supported. it handles the full pipeline automatically:

• computes a blake3 hash of the first 128MB of the file for cache validation to be fast
• caches preprocessed results under ~/.snapviewer_cache/<hash>_dev<id>_v<version>/ , different device has different dir names
• --dir and --pickle are mutually exclusive

Also I now have access to windows machines and I tested this version on windows. It is working and I am considering removing the python-rust interfacing via PyO3 completely, does this seem reasonable? @stas00

[Da1sypetals]

Use main branch instead; I have merged everything to it

[stas00]

I don't think blake3 works on mac (after installing requirements.txt)

$ python gui.py  --res 1200 500 -rr 2.0 --pickle data/small.pickle
Traceback (most recent call last):
  File "/Users/sbekman/github/SnapViewer/gui.py", line 26, in <module>
    from blake3 import blake3 as blake3_hasher
ModuleNotFoundError: No module named 'blake3'

Isolating it

$ uv pip uninstall blake3
$ uv pip install blake3
Using Python 3.11.11 environment at: /Users/sbekman/anaconda3/envs/py311-pt26
Resolved 2 packages in 2ms
Installed 1 package in 8ms
 + blake3==1.0.8

$ python -c "import blake3"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'blake3'

I will try it later on my linux box

[stas00]

on linux it works great, tested a new and a cached run.

one new feedback here is to add some indication the program is working - e.g. with a huge pickle file it sat silently for some 5min after:

2026-02-26 13:36:13,474 - INFO - Database written to temporary file: /tmp/tmpr0rfrqdk.db

which looks like a program hanging. So some "The next operation may take many minutes to complete..." print might be helpful.

[stas00]

I suppose the next stage is to make it pip install snapviewer which will install the application while building it. So may be python gui.py will become snapview.py? which may have snapview.py --convert and by default it'll render directly w/o any args other than the pickle file?

p.s. I don't know anything about Windows.

[Da1sypetals]

@stas00 For the blake3 problem,I pretty much have no idea because this was developed on my MacBook,and it works there.

For context, I manage Python environments with conda, and I have no experience working with uv. 😥

$ pip show blake3
Name: blake3
Version: 1.0.8
Summary: Python bindings for the Rust blake3 crate
Home-page: https://github.com/oconnor663/blake3-py
Author: Jack O'Connor <oconnor663@gmail.com>
Author-email: Jack O'Connor <oconnor663@gmail.com>
License: CC0-1.0 OR Apache-2.0
Location: /Users/daisy/miniconda3/lib/python3.13/site-packages
Requires: 
Required-by:

[Da1sypetals]

which looks like a program hanging

Newest main branch fixes this

[stas00]

wow, you even added a progress bar - amazing! Great work, @Da1sypetals!

[stas00]

uv is just 10-20x faster version of pip as it's written in Rust. Normally it works identically, but clearly it fails. Using pip fixed the problem. I filed an issue with uv.

All works then.

[stas00]

let me know if you want a new issue for each subject matter and I will create one.

now that pickle can be processed directly, the only remaining thing is perhaps speeding up the first rendering?

$ time python gui.py --log info --res 2400 1000 --pickle data/huge-2.pickle 
real    9m21.632s
user    1m22.165s
sys     0m16.406s

That's a long long time to wait to render :) since usually the generate to observe trace cycles need to be relatively fast to be efficient at debugging.

Granted it's a big file.

$ ls -sh data/huge-2.pickle
1.2G data/huge-2.pickle

Do you think there could be some way to continue processing while showing the rendering version and unpacking the traces as they are accessed/clicked on? I'm not familiar with the pickled structure to tell if it's feasible. I wonder how Chrome does it so much faster (but under 1GB pickle).

[Da1sypetals]

uv is just 10-20x faster version of pip as it's written in Rust. Normally it works identically, but clearly it fails. Using pip fixed the problem. I filed an issue with uv.

All works then.

The last time I used uv (around 2025.6) it could not manage global envs, its envs must be project specific. Usually I would prefer Rust alternative but not here because of this.

Did they fix this since then?