Skip to content

[ENG-756] feat(daemon): UTXO performance optimizations for large-scale wallets#24

Open
pycckuu wants to merge 1 commit intomainfrom
ENG-756/pycckuu/utxo-perf-optimizations
Open

[ENG-756] feat(daemon): UTXO performance optimizations for large-scale wallets#24
pycckuu wants to merge 1 commit intomainfrom
ENG-756/pycckuu/utxo-perf-optimizations

Conversation

@pycckuu
Copy link
Contributor

@pycckuu pycckuu commented Mar 3, 2026

Summary

  • Snapshot double-buffering (RwLock<Arc<UtxoState>>) eliminates 23% reader unavailability from exclusive Mutex lock during UTXO sort operations
  • Mempool overlay view (UtxoStateView) replaces full UTXO set cloning with zero-copy merge iterator
  • O(N×M) → O(N+M) transaction filtering via HashSet pre-build
  • Incremental round-robin address scanning replaces O(N) full rescan per sync tick
  • Address derivation caching: direct ChildNumber paths + pre-sorted multisig keys + versioned monitored address cache
  • Fix critical sync interval default mismatch (struct Default was 10ms vs clap 10,000ms)
  • Add benchmarks for address scaling, UTXO scaling, lock contention, and production-scale stress testing

Changes

  • daemon/src/utxo_manager.rs: New UtxoState snapshot with HashMap + sorted Vec<(u64, WalletOutpoint)> index; UtxoStateView overlay for mempool txs; snapshot swap under brief write-lock
  • daemon/src/sync_manager.rs: RPC calls without holding UTXO lock; incremental address scanning with round-robin cursor
  • daemon/src/address_manager.rs: Versioned address cache (Arc<Vec<Address>>), AddressQuerySet eliminating string round-trips, cheaper derivation paths, pre-sorted multisig keys
  • daemon/src/service/get_utxos.rs: Snapshot-once reads, pre-computed dust threshold, HashSet address filter
  • daemon/src/service/get_balance.rs: Snapshot-based balance computation
  • daemon/src/transaction_generator.rs: HashSet filter for select_utxos and more_utxos_for_merge_transaction
  • daemon/src/args.rs: Shared DEFAULT_SYNC_INTERVAL_MILLIS const + validation test
  • daemon/benches/: 4 benchmark files (address scaling, UTXO scaling, contention, filter performance)
  • daemon/src/bin/kaswallet_stress_bench.rs: Extreme-scale synthetic benchmark (1M addresses, 10M UTXOs)

Test plan

  • Run cargo build and cargo test to verify compilation and existing tests pass
  • Run benchmarks: cargo bench -p kaswallet-daemon --features bench
  • Run stress bench: cargo run -p kaswallet-daemon --features bench --release --bin kaswallet-stress-bench -- --i-understand --addresses 100000 --utxos 1000000
  • Verify sync loop works correctly with concurrent read operations
  • Test with real node at scale to confirm lock contention improvement

Closes ENG-756

@pycckuu pycckuu force-pushed the ENG-756/pycckuu/utxo-perf-optimizations branch from e4eda3e to 3c7ebc9 Compare March 3, 2026 10:21
@pycckuu pycckuu changed the title feat(daemon): UTXO performance optimizations for large-scale wallets [ENG-756] feat(daemon): UTXO performance optimizations for large-scale wallets Mar 3, 2026
…e wallets

Addresses critical scaling bottlenecks for wallets with 10M+ UTXOs:

- Snapshot double-buffering (RwLock<Arc<UtxoState>>) eliminates 23% reader
  unavailability caused by exclusive Mutex lock during sort operations
- Mempool overlay view (UtxoStateView) replaces full UTXO set cloning
- O(N×M) to O(N+M) transaction filtering via HashSet pre-build
- Incremental round-robin address scanning instead of O(N) full rescan
- Address derivation caching with direct ChildNumber paths and pre-sorted
  multisig keys
- Versioned monitored address cache avoids rebuild/parse per sync tick
- Fix sync interval default mismatch (struct Default was 10ms vs clap 10s)
- Add benchmarks for address scaling, UTXO scaling, lock contention,
  and stress testing at production scale
@pycckuu pycckuu force-pushed the ENG-756/pycckuu/utxo-perf-optimizations branch 2 times, most recently from 054d80f to 3f450dc Compare March 3, 2026 14:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants