feat(dash-spv): cache the filter-scan query keyed by wallet monitor revision - #951
Conversation
…evision scan_batch re-collected every behind wallet's scan scripts and re-grouped them into FilterQuerys on every batch, even though the query only changes when the wallet's monitored set does. The pruning pass added for #948 made that per-batch assembly O(total addresses) per wallet (~165us at 6k historical CoinJoin addresses), repeated across the ~1,160 batches of a full mainnet scan while nothing changed. Maintain the query as persistent state instead: FiltersManager keeps a per-wallet CachedWalletQuery (scripts, bare elements, pre-grouped FilterQuery) keyed by the new WalletInterface::wallet_monitor_revision — per-wallet account revisions plus account_generation, which move exactly when an address is derived, an account is added, or a UTXO is created or spent. The union query over the behind set is cached the same way, keyed by the sorted (wallet, revision) pairs it was assembled from. During a quiet catch-up the revision never moves, so consecutive batches share one assembled query and the wallet read lock is held only for the revision check. check_compact_filters_for_query lets callers match a pre-built FilterQuery; check_compact_filters_for_elements builds one and delegates. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## claude/rust-dashcore-948-667282 #951 +/- ##
===================================================================
+ Coverage 75.42% 75.47% +0.04%
===================================================================
Files 328 328
Lines 78630 78753 +123
===================================================================
+ Hits 59305 59437 +132
+ Misses 19325 19316 -9
|
…he cached scan query scan_batch_query mirrors dash-spv's scan_batch for one behind wallet: "rebuilt" re-collects the scan scripts and re-groups the queries every batch (pre-cache shape), "cached" does a revision check and matches the pre-assembled query. 512 filters, pruned query (555 scripts): used=500 rebuilt 5.54ms | cached 5.43ms (~2% saved) used=2000 rebuilt 5.42ms | cached 5.28ms (~2.5% saved) used=6000 rebuilt 5.71ms | cached 5.27ms (~7.6% saved) The cached path is flat regardless of wallet history; the rebuilt path grows with total historical addresses because the pruning walk is O(all pool addresses) per batch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Benchmarked the cache savings with a new
The key property: the cached path is flat regardless of wallet history, while the rebuilt path grows with total historical addresses — the pruning walk is O(all pool addresses) per batch, so pre-cache it was the last per-batch cost still scaling with mixing history. At 6,000 used addresses that's ~0.43 ms/batch, ~0.5 s of CPU across a full 2.32M-filter mainnet scan, plus the eliminated per-batch allocations and shorter wallet-lock hold. With larger real batches the relative share shrinks (matching grows, assembly doesn't) — consistent with the "second-order but structural" sizing in the PR description. 🤖 Generated with Claude Code |
|
Benchmark shows this isn't actually important. |
Stacked on #949 (base is its branch; retarget to
devonce #949 merges). Follow-up to #948.Problem
Even with #949's pruning,
scan_batchstill re-collects every behind wallet's scan scripts and re-groups them intoFilterQuerys on every batch — and the pruning pass itself walks all pool addresses, so that per-batch assembly is O(total historical addresses) per wallet (~165 µs at 6,000 historical CoinJoin addresses, measured by the newquery_assemblybench). Over the ~1,160 batches of a full mainnet scan that work repeats identically while nothing in the wallet changed, all under the wallet read lock.Change
Make the scan query persistent state that is modified on change instead of recreated per batch:
WalletInterface::wallet_monitor_revision(wallet_id)— per-wallet account revisions plusaccount_generation, which move exactly when an address is derived, an account is added, or a UTXO is created/spent (the only events that can change the scan set). Default falls back to the globalmonitor_revision()so existing implementations stay correct. Alsocheck_compact_filters_for_queryfor matching a pre-builtFilterQuery(check_compact_filters_for_elementsnow builds one and delegates).FiltersManagerkeeps a per-walletCachedWalletQuery(pruned scripts, bare elements, pre-groupedFilterQuery) keyed by that revision, plus a cached union query keyed by the sorted(wallet, revision)pairs it was assembled from. A batch scan reuses both until a revision moves or the behind set changes; during a quiet catch-up hundreds of consecutive batches share one assembled query, and the wallet lock is held only for the revision check.Correctness safety net: the revision is required to move on every scan-set change (real wallets bump account-level revisions on address/UTXO changes and
account_generationon account adds — pinned bytest_wallet_monitor_revision_tracks_scan_set_changes), and the existing #649 account-generation guard at commit time is unchanged.Testing
test_scan_batch_caches_query_by_wallet_revisionpins the cache contract: a scripts change without a revision bump keeps serving the cached query; a revision bump rebuilds it.test_wallet_monitor_revision_tracks_scan_set_changespins the revision semantics at the manager level.cargo test -p dash-spvwith dashd regtest integration tests passes; workspace clippy/fmt/pre-commit clean.query_assemblybench group quantifies the per-batch cost eliminated (~30 µs at 500 used addresses → ~165 µs at 6,000).Honest sizing: this is second-order next to #949 (assembly is ~5% of a 6k-address batch match); its value is that the per-batch cost no longer scales with wallet history at all, plus reduced lock hold and allocator churn on mobile.
🤖 Generated with Claude Code