treap, ffldb: improve pool recycling, add batched Delete

[Roasbeef]

Change Description

In this PR, we improve the sync.Pool-based node recycling introduced in #2425 and extend the same optimization to Delete operations.

Faster Recycling Check in Put()

The original Put() recycling loop called newTreap.get(node.key) for each intermediate node to check if it was still live in the latest treap version. This worked correctly but cost O(depth × log n) key comparisons per batch iteration. We replace this with a pointer-set approach: put() now returns both the set of newly created nodes and the set of original source nodes that were cloned (the "replaced" set). The recycling loop checks pointer identity via nodeInSet(), which is just a linear scan of a [staticDepth]*treapNode array. This brings the cost down to O(depth²) pointer comparisons, i.e. ~400 pointer equality checks for a typical depth of 20, versus ~8000 key comparisons before.

We also document the MVCC snapshot safety argument directly in the Put() godoc. The core insight is that intermediate clones are always fresh pool allocations, never shared with any pre-existing snapshot. Only clones that were subsequently re-cloned (and thus replaced in the new treap) are candidates for recycling. Clones still structurally shared with the new treap are left alone.

To gain confidence in this safety argument, we built a Quint formal model (TLA+-inspired) that models the treap as a set of node IDs, non-deterministically applies batched puts, deletes, snapshot captures, and snapshot releases, then checks three invariants across 30,000 randomized traces: no recycled node appears in any snapshot, in the current treap, or anywhere live. All three invariants passed. The model confirmed that fresh clone IDs are always >= nextId at allocation time, so they can never collide with any pre-existing snapshot, which is exactly what allows us to use the simpler pointer-set check instead of a full treap traversal.

Batched Delete() with Pool Recycling

Previously, Delete accepted a single key and allocated cloned nodes from the pool via cloneTreapNode without ever returning them. During IBD, commitTx issued many individual Delete calls inside ForEach loops, creating a one-way pool drain where nodes were pulled from the pool but never recycled back.

We extend Delete to accept variadic keys (Delete(keys ...[]byte)), mirroring Put's batched API. The internal delete() method returns the same (created, replaced) arrays, enabling the same inter-step recycling. The commitTx callsite in dbcache.go now collects deletion keys into slices and issues single batched calls. We also fix the pendingRemove path to pass nil as the value in KVPair rather than carrying the actual value, since the remove treap only tracks keys.

Test Fixes

We fix three test error messages that printed i+1 as the expected value when the actual expected value was (i+1)*keyCount. We also add single-key and small-batch (3 keys) snapshot immutability tests that exercise the recycling logic at the granularity where intermediate node recycling actually occurs. The small-batch test additionally verifies that all 1000 original keys in the snapshot remain accessible with correct values after a Put with recycling.

Benchmarks

goos: darwin
goarch: arm64
cpu: Apple M4 Max

                                   │    old     │              new               │
                                   │   B/op     │    B/op      vs base           │
ImmutableDeleteBatch/batch=10      │  2.932Ki   │  1.251Ki   -57.33% (p=0.002)  │
ImmutableDeleteBatch/batch=100     │  67.66Ki   │  42.03Ki   -37.89% (p=0.002)  │
ImmutableDeleteBatch/batch=1000    │  1323.5Ki  │  925.6Ki   -30.07% (p=0.002)  │
ImmutableMixedPutDelete            │  605.8Ki   │  524.3Ki   -13.46% (p=0.002)  │

                                   │  allocs/op │  allocs/op   vs base           │
ImmutableDeleteBatch/batch=10      │    44.50   │    23.00   -48.31% (p=0.002)  │
ImmutableDeleteBatch/batch=100     │    935.5   │    607.5   -35.06% (p=0.002)  │
ImmutableDeleteBatch/batch=1000    │   17.63k   │   12.53k   -28.90% (p=0.002)  │
ImmutableMixedPutDelete            │   8.274k   │   7.230k   -12.61% (p=0.002)  │

The batched Delete shows 30-57% reduction in bytes allocated and 29-48% fewer allocations. The mixed Put+Delete benchmark (which mirrors the commitTx pattern) shows a 13% reduction in both memory and allocations.

Put performance is roughly neutral, as expected: the pointer-set approach trades O(log n) key comparisons for O(depth) pointer comparisons, which are similar in practice for typical treap sizes.

See each commit message for a detailed description w.r.t the incremental changes.

Steps to Test

go test ./database/internal/treap/ -v -race -count=1
go test ./database/ffldb/ -v -race -short -count=1
go test ./database/internal/treap/ -bench=BenchmarkImmutable -benchmem

Pull Request Checklist

Testing

• Your PR passes all CI checks.
• Tests covering the positive and negative (error paths) are included.

Code Style and Documentation

• The change obeys the Code Documentation and Commenting guidelines, and lines wrap at 80.
• Commits follow the Ideal Git Commit Structure.

[coveralls]

Pull Request Test Coverage Report for Build 23569826387

Details

• 119 of 120 (99.17%) changed or added relevant lines in 2 files are covered.
• 6 unchanged lines in 2 files lost coverage.
• Overall coverage increased (+0.07%) to 56.299%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
database/internal/treap/immutable.go	108	109	99.08%

Files with Coverage Reduction	New Missed Lines	%
peer/peer.go	3	72.41%
rpcclient/infrastructure.go	3	47.89%

Totals
Change from base Build 23569410477:	0.07%
Covered Lines:	32040
Relevant Lines:	56910

---

💛 - Coveralls