Skip to content

Performance tweaks#254

Merged
j-rafique merged 4 commits intomasterfrom
performance-tweaks
Jan 2, 2026
Merged

Performance tweaks#254
j-rafique merged 4 commits intomasterfrom
performance-tweaks

Conversation

@j-rafique
Copy link
Contributor

@j-rafique j-rafique commented Jan 1, 2026

    1. supernode/cascade/download.go: Caps how many candidate keys get passed into P2PClient.BatchRetrieveStream (keyCap = max(need*50, 5000)) and stops building remainingKeys once that cap is reached; rationale is to avoid DHT per-key preprocessing cost on huge key lists when only a small top-up is needed; impact is noticeably lower CPU/alloc/GC late in the download when need is small (possible extra attempt if many keys are missing/unavailable).
    1. supernode/cascade/download.go: Raises the initial decode target from 17% to 20% (targetRequiredPercent) and skips RQ.DecodeFromPrepared until have >= reqCount; rationale is to avoid expensive decode attempts at borderline/under-target coverage; impact is fewer wasted decode cycles and typically faster time-to-success, with a small increase in symbols fetched up front.
    1. supernode/cascade/download.go: Uses a prepared workspace + streaming retrieval with incremental retries (attempt loop with reqCount increasing by ~5%); rationale is to reduce memory usage and improve resilience vs single-shot decode; impact is higher success rate and often better overall download time, though worst-case can do multiple retrieve+decode rounds (more CPU/network than fail-fast).
    1. supernode/adaptors/p2p.go: Derives symbol keys from codec.Layout (symbolKeysFromLayout) instead of recursively walking the symbols directory; rationale is to avoid expensive filesystem traversal; impact is materially less disk I/O and faster first-pass store (risk only if layout and on-disk symbols diverge).

@roomote
Copy link

roomote bot commented Jan 1, 2026

Rooviewer Clock   See task on Roo Cloud

Review complete. No issues found.

This PR continues the performance improvements with two additional commits:

  • Allocation reduction in closest-node selection: Pre-computes ignored node set once for batch operations, pre-allocates slices based on total node count, and releases locks earlier (before sorting) to reduce contention
  • Synchronous StoreBatch for original data: Ensures original/local batches are durably committed before returning, preventing data loss if callers delete source files immediately after StoreBatch returns
Previous reviews

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

roomote[bot]
roomote bot previously approved these changes Jan 1, 2026
@j-rafique j-rafique merged commit 3a74db5 into master Jan 2, 2026
7 checks passed
@j-rafique j-rafique deleted the performance-tweaks branch February 19, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

Comments