Skip to content

feat(dash-spv): cache the filter-scan query keyed by wallet monitor revision - #951

Closed
QuantumExplorer wants to merge 2 commits into
claude/rust-dashcore-948-667282from
claude/rust-dashcore-948-cached-scan-query
Closed

feat(dash-spv): cache the filter-scan query keyed by wallet monitor revision#951
QuantumExplorer wants to merge 2 commits into
claude/rust-dashcore-948-667282from
claude/rust-dashcore-948-cached-scan-query

Conversation

@QuantumExplorer

Copy link
Copy Markdown
Member

Stacked on #949 (base is its branch; retarget to dev once #949 merges). Follow-up to #948.

Problem

Even with #949's pruning, scan_batch still re-collects every behind wallet's scan scripts and re-groups them into FilterQuerys on every batch — and the pruning pass itself walks all pool addresses, so that per-batch assembly is O(total historical addresses) per wallet (~165 µs at 6,000 historical CoinJoin addresses, measured by the new query_assembly bench). Over the ~1,160 batches of a full mainnet scan that work repeats identically while nothing in the wallet changed, all under the wallet read lock.

Change

Make the scan query persistent state that is modified on change instead of recreated per batch:

  • key-wallet-manager: new WalletInterface::wallet_monitor_revision(wallet_id) — per-wallet account revisions plus account_generation, which move exactly when an address is derived, an account is added, or a UTXO is created/spent (the only events that can change the scan set). Default falls back to the global monitor_revision() so existing implementations stay correct. Also check_compact_filters_for_query for matching a pre-built FilterQuery (check_compact_filters_for_elements now builds one and delegates).
  • dash-spv: FiltersManager keeps a per-wallet CachedWalletQuery (pruned scripts, bare elements, pre-grouped FilterQuery) keyed by that revision, plus a cached union query keyed by the sorted (wallet, revision) pairs it was assembled from. A batch scan reuses both until a revision moves or the behind set changes; during a quiet catch-up hundreds of consecutive batches share one assembled query, and the wallet lock is held only for the revision check.

Correctness safety net: the revision is required to move on every scan-set change (real wallets bump account-level revisions on address/UTXO changes and account_generation on account adds — pinned by test_wallet_monitor_revision_tracks_scan_set_changes), and the existing #649 account-generation guard at commit time is unchanged.

Testing

  • test_scan_batch_caches_query_by_wallet_revision pins the cache contract: a scripts change without a revision bump keeps serving the cached query; a revision bump rebuilds it.
  • test_wallet_monitor_revision_tracks_scan_set_changes pins the revision semantics at the manager level.
  • Full cargo test -p dash-spv with dashd regtest integration tests passes; workspace clippy/fmt/pre-commit clean.
  • New query_assembly bench group quantifies the per-batch cost eliminated (~30 µs at 500 used addresses → ~165 µs at 6,000).

Honest sizing: this is second-order next to #949 (assembly is ~5% of a 6k-address batch match); its value is that the per-batch cost no longer scales with wallet history at all, plus reduced lock hold and allocator churn on mobile.

🤖 Generated with Claude Code

…evision

scan_batch re-collected every behind wallet's scan scripts and re-grouped
them into FilterQuerys on every batch, even though the query only changes
when the wallet's monitored set does. The pruning pass added for #948 made
that per-batch assembly O(total addresses) per wallet (~165us at 6k
historical CoinJoin addresses), repeated across the ~1,160 batches of a
full mainnet scan while nothing changed.

Maintain the query as persistent state instead: FiltersManager keeps a
per-wallet CachedWalletQuery (scripts, bare elements, pre-grouped
FilterQuery) keyed by the new WalletInterface::wallet_monitor_revision —
per-wallet account revisions plus account_generation, which move exactly
when an address is derived, an account is added, or a UTXO is created or
spent. The union query over the behind set is cached the same way, keyed
by the sorted (wallet, revision) pairs it was assembled from. During a
quiet catch-up the revision never moves, so consecutive batches share one
assembled query and the wallet read lock is held only for the revision
check.

check_compact_filters_for_query lets callers match a pre-built
FilterQuery; check_compact_filters_for_elements builds one and delegates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 04e053b3-b668-43e6-a97d-6ab1c01f9231

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.73684% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.47%. Comparing base (53fd906) to head (1354236).

Files with missing lines Patch % Lines
dash-spv/src/sync/filters/manager.rs 95.32% 5 Missing ⚠️
key-wallet-manager/src/wallet_interface.rs 0.00% 3 Missing ⚠️
Additional details and impacted files
@@                         Coverage Diff                         @@
##           claude/rust-dashcore-948-667282     #951      +/-   ##
===================================================================
+ Coverage                            75.42%   75.47%   +0.04%     
===================================================================
  Files                                  328      328              
  Lines                                78630    78753     +123     
===================================================================
+ Hits                                 59305    59437     +132     
+ Misses                               19325    19316       -9     
Flag Coverage Δ
core 77.29% <ø> (ø)
ffi 49.04% <ø> (ø)
rpc 20.00% <ø> (ø)
spv 91.46% <95.32%> (+0.08%) ⬆️
wallet 77.52% <93.33%> (+0.04%) ⬆️
Files with missing lines Coverage Δ
key-wallet-manager/src/lib.rs 76.07% <ø> (ø)
key-wallet-manager/src/matching.rs 96.77% <100.00%> (+0.12%) ⬆️
key-wallet-manager/src/process_block.rs 92.84% <100.00%> (+0.53%) ⬆️
key-wallet-manager/src/wallet_interface.rs 8.57% <0.00%> (-0.81%) ⬇️
dash-spv/src/sync/filters/manager.rs 97.96% <95.32%> (+0.02%) ⬆️

... and 5 files with indirect coverage changes

…he cached scan query

scan_batch_query mirrors dash-spv's scan_batch for one behind wallet:
"rebuilt" re-collects the scan scripts and re-groups the queries every
batch (pre-cache shape), "cached" does a revision check and matches the
pre-assembled query. 512 filters, pruned query (555 scripts):

  used=500   rebuilt 5.54ms | cached 5.43ms  (~2% saved)
  used=2000  rebuilt 5.42ms | cached 5.28ms  (~2.5% saved)
  used=6000  rebuilt 5.71ms | cached 5.27ms  (~7.6% saved)

The cached path is flat regardless of wallet history; the rebuilt path
grows with total historical addresses because the pruning walk is
O(all pool addresses) per batch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@QuantumExplorer

Copy link
Copy Markdown
Member Author

Benchmarked the cache savings with a new scan_batch_query group that mirrors scan_batch for one behind wallet — rebuilt re-collects the scan scripts and re-groups the queries every batch (pre-cache shape), cached does a revision check and matches the pre-assembled query. 512 filters per batch, mixing-heavy wallet, pruned query (555 scripts) in both cases:

used CoinJoin addresses rebuilt / batch cached / batch saved
500 5.54 ms 5.43 ms ~2%
2,000 5.42 ms 5.28 ms ~2.5%
6,000 5.71 ms 5.27 ms ~7.6%

The key property: the cached path is flat regardless of wallet history, while the rebuilt path grows with total historical addresses — the pruning walk is O(all pool addresses) per batch, so pre-cache it was the last per-batch cost still scaling with mixing history. At 6,000 used addresses that's ~0.43 ms/batch, ~0.5 s of CPU across a full 2.32M-filter mainnet scan, plus the eliminated per-batch allocations and shorter wallet-lock hold. With larger real batches the relative share shrinks (matching grows, assembly doesn't) — consistent with the "second-order but structural" sizing in the PR description.

cargo bench -p key-wallet-manager --bench filter_scan -- scan_batch_query

🤖 Generated with Claude Code

@QuantumExplorer

Copy link
Copy Markdown
Member Author

Benchmark shows this isn't actually important.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant