[ENG-756] feat(daemon): UTXO performance optimizations for large-scale wallets#24
Open
[ENG-756] feat(daemon): UTXO performance optimizations for large-scale wallets#24
Conversation
e4eda3e to
3c7ebc9
Compare
…e wallets Addresses critical scaling bottlenecks for wallets with 10M+ UTXOs: - Snapshot double-buffering (RwLock<Arc<UtxoState>>) eliminates 23% reader unavailability caused by exclusive Mutex lock during sort operations - Mempool overlay view (UtxoStateView) replaces full UTXO set cloning - O(N×M) to O(N+M) transaction filtering via HashSet pre-build - Incremental round-robin address scanning instead of O(N) full rescan - Address derivation caching with direct ChildNumber paths and pre-sorted multisig keys - Versioned monitored address cache avoids rebuild/parse per sync tick - Fix sync interval default mismatch (struct Default was 10ms vs clap 10s) - Add benchmarks for address scaling, UTXO scaling, lock contention, and stress testing at production scale
054d80f to
3f450dc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RwLock<Arc<UtxoState>>) eliminates 23% reader unavailability from exclusive Mutex lock during UTXO sort operationsUtxoStateView) replaces full UTXO set cloning with zero-copy merge iteratorChildNumberpaths + pre-sorted multisig keys + versioned monitored address cacheChanges
daemon/src/utxo_manager.rs: NewUtxoStatesnapshot withHashMap+ sortedVec<(u64, WalletOutpoint)>index;UtxoStateViewoverlay for mempool txs; snapshot swap under brief write-lockdaemon/src/sync_manager.rs: RPC calls without holding UTXO lock; incremental address scanning with round-robin cursordaemon/src/address_manager.rs: Versioned address cache (Arc<Vec<Address>>),AddressQuerySeteliminating string round-trips, cheaper derivation paths, pre-sorted multisig keysdaemon/src/service/get_utxos.rs: Snapshot-once reads, pre-computed dust threshold,HashSetaddress filterdaemon/src/service/get_balance.rs: Snapshot-based balance computationdaemon/src/transaction_generator.rs: HashSet filter forselect_utxosandmore_utxos_for_merge_transactiondaemon/src/args.rs: SharedDEFAULT_SYNC_INTERVAL_MILLISconst + validation testdaemon/benches/: 4 benchmark files (address scaling, UTXO scaling, contention, filter performance)daemon/src/bin/kaswallet_stress_bench.rs: Extreme-scale synthetic benchmark (1M addresses, 10M UTXOs)Test plan
cargo buildandcargo testto verify compilation and existing tests passcargo bench -p kaswallet-daemon --features benchcargo run -p kaswallet-daemon --features bench --release --bin kaswallet-stress-bench -- --i-understand --addresses 100000 --utxos 1000000Closes ENG-756