Skip to content

Materialize unplugged from global cache#261

Draft
cijiugechu wants to merge 5 commits intoyarnpkg:mainfrom
cijiugechu:perf/global-unplugged-clonefile
Draft

Materialize unplugged from global cache#261
cijiugechu wants to merge 5 commits intoyarnpkg:mainfrom
cijiugechu:perf/global-unplugged-clonefile

Conversation

@cijiugechu
Copy link
Contributor

Adds /unplugged extracted cache and materializes per-project unplugged dirs from it.

macos uses clonefile, Linux uses ioctl reflinks with a copy fallback. Falls back to zip extraction when cloning/copying isn't possible.

On my local macos machine, install-cache-and-lock got :

gatsby:mean 0.3227s -> 0.1561s(-51.6%,~2.07x)
next:mean 0.1379s -> 0.1073s(-22.2%,~1.29x)

Adds <globalFolder>/unplugged extracted cache and materializes per-project unplugged dirs from it.

macOS uses clonefile(2); Linux uses ioctl(FICLONE) reflinks with a copy fallback. Falls back to zip extraction when cloning/copying isn't possible.

Also adds Project::global_unplugged_path and Path::fs_clonefile.
@github-actions
Copy link

github-actions bot commented Feb 28, 2026

⏱️ Benchmark Results

gatsby install-full-cold

Metric Base Head Difference
Mean 2.561s 2.861s +11.73% ⚠️
Median 2.569s 2.867s +11.59% ⚠️
Min 2.360s 2.723s
Max 2.733s 2.955s
Std Dev 0.063s 0.055s
📊 Raw benchmark data (gatsby install-full-cold)

Base times: 2.733s, 2.581s, 2.569s, 2.616s, 2.576s, 2.533s, 2.615s, 2.597s, 2.508s, 2.501s, 2.603s, 2.569s, 2.570s, 2.543s, 2.575s, 2.557s, 2.540s, 2.360s, 2.561s, 2.568s, 2.577s, 2.543s, 2.606s, 2.546s, 2.628s, 2.510s, 2.519s, 2.589s, 2.444s, 2.584s

Head times: 2.908s, 2.820s, 2.920s, 2.839s, 2.810s, 2.854s, 2.875s, 2.955s, 2.847s, 2.861s, 2.821s, 2.941s, 2.904s, 2.925s, 2.836s, 2.803s, 2.791s, 2.878s, 2.881s, 2.895s, 2.723s, 2.779s, 2.855s, 2.873s, 2.872s, 2.807s, 2.954s, 2.906s, 2.885s, 2.812s


gatsby install-cache-and-lock (warm, with lockfile)

Metric Base Head Difference
Mean 0.446s 0.547s +22.55% ⚠️
Median 0.445s 0.546s +22.80% ⚠️
Min 0.436s 0.537s
Max 0.471s 0.560s
Std Dev 0.008s 0.005s
📊 Raw benchmark data (gatsby install-cache-and-lock (warm, with lockfile))

Base times: 0.437s, 0.437s, 0.436s, 0.461s, 0.440s, 0.471s, 0.470s, 0.438s, 0.441s, 0.444s, 0.445s, 0.451s, 0.443s, 0.449s, 0.443s, 0.447s, 0.439s, 0.441s, 0.447s, 0.446s, 0.446s, 0.445s, 0.446s, 0.442s, 0.446s, 0.444s, 0.441s, 0.448s, 0.450s, 0.448s

Head times: 0.540s, 0.537s, 0.547s, 0.542s, 0.543s, 0.541s, 0.543s, 0.540s, 0.544s, 0.544s, 0.538s, 0.560s, 0.545s, 0.543s, 0.548s, 0.546s, 0.546s, 0.547s, 0.548s, 0.551s, 0.553s, 0.546s, 0.551s, 0.557s, 0.554s, 0.549s, 0.551s, 0.550s, 0.548s, 0.546s

Gate global unplugged materialization on an existing lockfile to avoid regressing fully-cold installs (no lockfile yet).
@arcanis
Copy link
Member

arcanis commented Feb 28, 2026

Gate global unplugged materialization on an existing lockfile to avoid regressing fully-cold installs (no lockfile yet).

Lil' bit worried - if it's slower when there isn't a lockfile, wouldn't it also be slower when there's already one?

@cijiugechu
Copy link
Contributor Author

Gate global unplugged materialization on an existing lockfile to avoid regressing fully-cold installs (no lockfile yet).

Lil' bit worried - if it's slower when there isn't a lockfile, wouldn't it also be slower when there's already one?

Based on the current implementation, I don’t see an easy way to avoid touching cold-install performance here—but I might be missing something. If there’s a better approach, I’m very open to it. Otherwise I can follow up by recovering some performance budget elsewhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants