Skip to content

backport: assumeutxo M3 — crash-safe background completion (bitcoin#25740, #27862, #28050, #27746) - #52

Open
PastaPastaPasta wants to merge 48 commits into
assumeutxo/m2-evodb-rolesfrom
assumeutxo/m3-background-completion
Open

backport: assumeutxo M3 — crash-safe background completion (bitcoin#25740, #27862, #28050, #27746)#52
PastaPastaPasta wants to merge 48 commits into
assumeutxo/m2-evodb-rolesfrom
assumeutxo/m3-background-completion

Conversation

@PastaPastaPasta

@PastaPastaPasta PastaPastaPasta commented Jul 12, 2026

Copy link
Copy Markdown
Owner

Issue being fixed or feature implemented

M2 (#51) wired up dual-chainstate EvoDB roles but left two integration
items open: promoting snapshot markers to the legacy key at completion, and
an independent (non-shared-cache) comparison of background vs. snapshot
state before a background chainstate is allowed to complete and replace the
snapshot. Without crash-safe completion, a power loss mid-promotion can
destroy a valid, already-validated snapshot. This is milestone 3/7. The
execution plan and per-unit ledger (M3 section) are tracked outside the repo
in a private gist, not checked in: https://gist.github.com/PastaPastaPasta/aa52b1f89fb74a0566ba3b5e15afab6a

PR 3/7 in the stacked series — base is M2 (assumeutxo/m2-evodb-roles).

What was done?

  • bitcoin#27862 (moved here from M1 since it depends on M3's
    SnapshotCompletionResult) and bitcoin#28050FAILURE_FATAL bypasses
    the reindex prompt in Dash's init else-if structure;
    FAILURE_INSUFFICIENT_DBCACHE intentionally absent until bitcoin#27596
    lands in M5.
  • bitcoin#25740 — crash-safe background-validation completion. Round 1
    adversarial review found four issues: crash-unsafe promote-before-swap
    ordering (including missing fsync gaps), MN-list comparison that wasn't
    actually independent of the shared cache, a nullopt base-height skip that
    could mask on-disk corruption, and unsynced activation metadata — all
    fixed. Dash adaptation: crash-safe completion lifecycle (renames +
    DirectoryCommit before a synced, idempotent PromoteSnapshotMarkers;
    RecoverSnapshotCleanup state machine run at load); independent MN-list
    verification via a per-connect NORMAL-identity captured hash slot compared
    against the activation-recorded snapshot hash at completion (not read from
    shared cache). Both M2 integration TODOs (marker promotion, holistic
    base-state comparison) are discharged here.
  • bitcoin#27746 (13 upstream commits) — per-chainstate candidate admission.
    Round 1 review found ResetBlockFailureFlags bypassed per-chainstate
    admission (background could admit non-base-path forks and starve the
    active set) and that InvalidateBlock's cache could reinsert
    ChainLock-conflicting candidates — both fixed, including a
    BLOCK_CONFLICT_CHAINLOCK exclusion in TryAddBlockIndexCandidate and a
    dual-chainstate regression test.

How Has This Been Tested?

Breaking Changes

None. Background validation completion is now fully wired (UTXO-hash +
MN-list-hash + EvoDB-marker checks, crash-safe promotion/cleanup with
load-time recovery) but is not yet reachable — loadtxoutset itself is
still unexposed until M5.

Checklist:

  • I have performed a self-review of my own code
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation (docs rewrite lands in M7/B8)
  • I have assigned this pull request to a milestone

@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1c0946c2-20ba-42fd-a86b-4ee0a0e5ccfd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch assumeutxo/m3-background-completion

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch from 552e404 to 2a37be9 Compare July 12, 2026 14:17
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 32d5966 to 4f9d716 Compare July 12, 2026 14:21
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch from 2a37be9 to 77a5910 Compare July 12, 2026 15:59
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 4f9d716 to f08bce5 Compare July 12, 2026 16:00
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch 4 times, most recently from 6a803ac to 014afb4 Compare July 13, 2026 22:45
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch 2 times, most recently from a6e49d6 to eb757a0 Compare July 14, 2026 05:10
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch 2 times, most recently from 06b5c6f to 1a95697 Compare July 14, 2026 17:01
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch 2 times, most recently from 2bc543c to 5b3d0d7 Compare July 20, 2026 16:59
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch 2 times, most recently from a3c4476 to 26c8718 Compare July 25, 2026 00:07
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch from 26c8718 to b6edcf8 Compare August 1, 2026 18:17
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch 3 times, most recently from 05723b8 to 667611e Compare August 1, 2026 22:27
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch from 11a31da to 5f3a876 Compare August 1, 2026 22:40
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch 2 times, most recently from 328d224 to f9f8fa6 Compare August 1, 2026 23:35
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch from 5f3a876 to d47d9b8 Compare August 2, 2026 18:42
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from f9f8fa6 to 63de2c1 Compare August 2, 2026 18:42
MacroFake and others added 2 commits August 3, 2026 01:44
…gnerImpl members public, rm temporaries, simplify

4bedfd7 refactor: remove unneeded temporaries in node/interfaces, simplify code (Jon Atack)
b27ba16 refactor: make all NodeImpl/ChainImpl/ExternalSignerImpl members public (Jon Atack)

Pull request description:

  - Make all `NodeImpl`, `ChainImpl` and `ExternalSignerImpl` class members `public` (and document why), to be consistent in all the `*Impl` classes in `src/node/interfaces.cpp` and `src/wallet/interfaces.cpp` and to help future reviewers and contributors.

  - Remove unneeded temporaries in `NodeImpl` and `ChainImpl` methods in `src/node/interfaces.cpp` and simplify, to make the code easier to read and understand and to improve performance by avoiding unnecessary move operations.

ACKs for top commit:
  ryanofsky:
    Code review ACK 4bedfd7. Changes since last review, applying suggested style & simplifiying first commit. Also avoiding another lock in second commit.

Tree-SHA512: 112f7cad5e2838c94c5b79d61328f42fe75fdb97f401ab49eccf696fc2c6a8a0c0ee55ec974c0602acf7423f78bb82e90eb8a0cc531e1d3347f73b7c83685504
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m2-evodb-roles branch from d47d9b8 to 7fabb1d Compare August 2, 2026 21:04
knst and others added 13 commits August 7, 2026 01:19
BACKPORT NOTE for Dash Core
This backport has been done in the past except one failing check.
That check had been failing due to double run of all checks in this functional
test: once for mempool, once for block.

This fix complete bitcoin#18628 backporting

----

fa4c29b test: Add various low-level p2p tests (MarcoFalke)

Pull request description:

ACKs for top commit:
  jonatack:
    ACK fa4c29b

Tree-SHA512: 842821b97359d4747c763398f7013415858c18a300cd882887bc812d039b5cbb67b9aa6f68434575dbc3c52f7eb8c43d1b293a59555a7242c0ca615cf44dc0aa

Co-authored-by: MarcoFalke <falke.marco@gmail.com>
c0a5fce test: Add test for erase orphan tx conflicted by block (Hennadii Stepanov)
fa45bb2 test: Add test for erase orphan tx included by block (Hennadii Stepanov)
5c04978 test: Add test for erase orphan tx from peer (Hennadii Stepanov)

Pull request description:

  This PR adds test coverage for the following cases:
  - erase orphan transactions when a peer is disconnected
  - erase an orphan transaction when it is included in a new tip block
  - erase an orphan transaction when it is conflicted with other transactions included in a new tip block

  Found useful while working on bitcoin#19374.

ACKs for top commit:
  aureleoules:
    tACK c0a5fce (`make check` and `test/functional/test_runner.py`).
  kouloumos:
    ACK c0a5fce with a nit per bitcoin#19393 (comment).
  pg156:
    Reviewed to bitcoin@c0a5fce. Concept ACK. Agree due to the lack of RPC calls to inspect orphan pool, using `assert_debug_log` to match strings in log is a reasonable way to test.

Tree-SHA512: 98f8deeee2d1c588c7e28a82e513d4a18655084198369db33fe2710458251eeaffed030626940072d7576f57fcbf7d856d761990129e2ca9e372d2ccbd86d07d

Co-authored-by: MacroFake <falke.marco@gmail.com>
e5c8384 fix(net): address ChainLock seen-cache follow-up (pasta)

Pull request description:

  ## Issue being fixed or feature implemented

  Follow-up to dashpay#7482 addressing post-merge review feedback.

  ## What was done?

  - Renamed limited-map thresholds so `max_size()` is the actual maximum and `cutoff_size()` is the post-prune size.
  - Removed the detailed ChainLock implementation comments and test-only threshold accessors.
  - Preserved the existing 1,024/2,048 ChainLock cache configuration.
  - Reworked the ChainLock test around a fixed 2,500-entry safety bound and recent CLSIG availability, without asserting internal pruning details.

  ## How Has This Been Tested?

  - `make -j1`
  - `./src/test/test_dash --run_test=limitedmap_tests/* --log_level=message`
  - `./src/test/test_dash --run_test=llmq_chainlock_tests/* --log_level=message`
  - `test/lint/lint-whitespace.py`
  - `test/lint/lint-includes.py`

  ## Breaking Changes

  None.

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have added or updated relevant unit/integration/functional/e2e tests

  This pull request was created by Codex.

ACKs for top commit:
  knst:
    utACK e5c8384

Tree-SHA512: e53fb00d1eee638de666c3c536093ee39c84c5cd33f0d7f6d68a07d1c9353676793f03aedede61eb6d091825ef6dd7760b159d874ffca65e4bddc4ea9c75b42b
…Context

d1b4f75 refactor: move CDSNotificationInterface into NodeContext (pasta)
2dcef69 refactor: pass CDeterministicMNManager by reference to CDSNotificationInterface (pasta)

Pull request description:

  ## Issue being fixed or feature implemented
  Moves `CDSNotificationInterface` from a raw global (`g_ds_notification_interface`) into `NodeContext` (`node.ds_notification_interface`).

  The global `g_ds_notification_interface` was previously constructed in `init.cpp` and held raw pointer references to various node services. Moving it into `NodeContext` aligns its lifecycle with other validation interface subscribers (such as `cj_walletman` and `active_ctx`).

  ## What was done?
  1. **Pass `CDeterministicMNManager` by reference to `CDSNotificationInterface`**:
     - Changed the constructor parameter from `const std::unique_ptr<CDeterministicMNManager>&` to `CDeterministicMNManager&`.
     - Removed the single-use `#include <util/check.h>` header in `src/dsnotificationinterface.cpp` after replacing `Assert(m_dmnman)` with `m_dmnman.UpdatedBlockTip(...)`.
     - Removed the stale `// todo:` note in `src/init.cpp`.

  2. **Move `CDSNotificationInterface` into `NodeContext`**:
     - Added `ds_notification_interface` (`std::unique_ptr<CDSNotificationInterface>`) to `NodeContext` in `src/node/context.h`. Placed it after all referenced managers to ensure correct C++ reverse-destruction order.
     - Included `<dsnotificationinterface.h>` in `src/node/context.cpp`.
     - Removed global `g_ds_notification_interface` extern declaration and definition.
     - Updated all call sites in `src/init.cpp` to use `node.ds_notification_interface`.

  ## How Has This Been Tested?
  - Unit tests (`./src/test/test_dash --run_test=getarg_tests`)
  - Functional tests: `feature_dip3_deterministicmns.py` and `feature_governance.py`
  - Static linters: `test/lint/all-lint.py`

  ## Breaking Changes
  None.

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have assigned this pull request to a milestone

  This pull request was created by Codex.

ACKs for top commit:
  knst:
    utACK d1b4f75

Tree-SHA512: e98da62223b4200d2d7e7cad7f246fb948a4aaf409cb39d40b4878ed659223863dd6cfd23524962869f0af69ff4ea64560887cd3deb7d19a47a4bf62daac8c4a
… stream size

aa1447f serialize: require SizedStream for ReadFixedBitSet instead of duck-typing it (pasta)
9f026a9 fix: bound DYNBITSET allocation against remaining stream size (pasta)

Pull request description:

  ## Issue being fixed or feature implemented

  `ReadFixedBitSet` allocated from a wire-declared `CompactSize` with no bound beyond `ReadCompactSize`'s 33,554,432 cap. Roughly five bytes on the wire — a `CompactSize` claiming millions of bits, with no payload following — therefore forced a `std::vector<bool>` resize plus a byte buffer totalling several MiB, all of which was only abandoned when the subsequent short read threw.

  `MAX_PROTOCOL_MESSAGE_LENGTH` does not help here, because the attack uses an *undersized* message.

  The primitive is reachable from an unauthenticated `QFCOMMITMENT` via `CFinalCommitment`'s `signers` and `validMembers` bitsets, and applies to every other `DYNBITSET` caller as well.

  This was split out of [dashpay#7523](dashpay#7523), where it was bundled with a much larger and more contentious DKG-intake change. The bound stands on its own, so it is offered separately for independent review.

  ## What was done?

  Bound the declared length against the bytes actually remaining in the stream, before allocating anything. A well-formed message always carries exactly the required bytes, so this rejects only claims that could never have been satisfied.

  The guard needs the stream to report how many bytes are still available, so `ReadFixedBitSet` is constrained on a `SizedStream` concept. That constraint is deliberately a hard requirement rather than a fallback: a stream without `size()` — `CHashVerifier`, for instance — now fails to compile here instead of silently deserializing an unbounded bitset. An earlier revision duck-typed the check with `if constexpr`, which would have let a future caller quietly lose the bound with no diagnostic.

  Nothing in tree is affected, because every stream that reaches this path already reports bytes *remaining*: `DataStream` (`vch.size() - m_read_pos`), `SpanReader`, and `OverrideStream`, which forwards to its wrapped stream. That "remaining, not total" requirement is what the bound rests on, so it is documented on the concept itself — a stream whose `size()` meant something else would weaken the bound silently rather than fail to compile.

  Note on what is deliberately *not* included: an earlier revision also checked `nbytes > MAX_SIZE`. That check is dead code — `size` comes from `ReadCompactSize`, which already caps at 33,554,432, so `(size + 7) / 8` can never exceed 4 MiB and the condition is always false.

  ## How Has This Been Tested?

  `src/test/serialize_tests.cpp` covers the amplification case (a declared length far exceeding the remaining stream must be rejected *before* allocating, not by a short read afterwards) and a maximum-size bitset (`MAX_SIZE`, 33,554,432 bits) round-tripping unchanged.

  Built locally and `test_dash --run_test=serialize_tests` passes. Full validation is delegated to CI.

  ## Breaking Changes

  None. Only messages declaring a bit count that the message could not possibly contain are affected, and those were never valid.

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have assigned this pull request to a milestone

ACKs for top commit:
  knst:
    utACK aa1447f

Tree-SHA512: 0b7da754ac61d1aa21e6c3fcf75c6986f69f6e0f90d51b9061cf45b03909bdf77d6b125f5f3665976aeca8defe361300bc465789347cac927895433c90a035ae
2f0bd0e test: restore complete spork21 signing coverage (pasta)

Pull request description:

  ## Issue being fixed or feature implemented

  PR dashpay#7497 merged the default and `--spork21` variants of `feature_llmq_signing.py` based on the assumption that the shared signing-session flow does not depend on spork 21.

  That assumption does not hold for the signature-share transport. Mainnet hardens `SPORK_21_QUORUM_ALL_CONNECTED` to value `1`, which keeps the all-connected concentrated `QSIGSHARE` path enabled for small quorums while LLMQ-100/400 quorums retain the legacy announce/request and `QBSIGSHARES` path. Both implementations are therefore live and need coverage for the complete signing-session lifecycle.

  The shortened mid-test spork section retained basic threshold recovery and recovery-member retry coverage, but no longer exercised conflict handling, quorum aging, recovered-signature cleanup, and session reuse through the spork-21 transport.

  ## What was done?

  - Restored the `--spork21` option and activation before the first quorum is mined.
  - Restored the original spork-specific `QSIGSHARE` submission, full-connectivity checks, and recovery-member isolation flow.
  - Removed the mid-test spork activation and shortened duplicate signing section.
  - Re-added `feature_llmq_signing.py --spork21` to the functional test runner.

  This is limited to the signing-test deduplication from dashpay#7497. Its wallet RPC coverage split and spork-23 test optimization remain unchanged.

  ## How Has This Been Tested?

  The following checks passed against a current local Dash Core build:

  - `test/functional/feature_llmq_signing.py`
  - `test/functional/feature_llmq_signing.py --spork21`
  - `python3 -m py_compile test/functional/feature_llmq_signing.py test/functional/test_runner.py`
  - `test/lint/lint-whitespace.py`
  - `git diff --check`

  `test/lint/lint-python.py` was also invoked, but skipped because `flake8` is not installed in the local environment.

  ## Breaking Changes

  None. Test-only change.

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation _(not applicable)_
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

  This pull request was created by Codex.

ACKs for top commit:
  knst:
    utACK 2f0bd0e

Tree-SHA512: ae777a38b1d61b0f999eb850812b51621c07ee4a6875c599c6ef44c4dab93a1fbea01aa35a77ebb3500274500094f723a9f81387e5433edef7a6189a2df65499
…5204, bitcoin#25689, bitcoin#25785, bitcoin-core/gui#613

452481a Merge bitcoin#19393: test: Add more tests for orphan tx handling (Konstantin Akimov)
92243fb Merge bitcoin#18628: test: Add various low-level p2p tests (Konstantin Akimov)
f95d19b fix: follow-up for bitcoin#21148 - orphanage should be cleaned by the first thing in the connected block (Konstantin Akimov)
8fc8f80 Merge bitcoin#25785: bench: Add a benchmark for descriptor expansion (Konstantin Akimov)
ce720b3 Merge bitcoin#25689: fuzz: Remove no-op SetMempoolConstraints (Konstantin Akimov)
a48944e partial Merge bitcoin-core/gui#613: Remove unnecessary wallet includes from rpcconsole.cpp (Konstantin Akimov)
13336e8 Merge bitcoin#25204: rpc: remove deprecated top-level fee fields from mempool entries (Konstantin Akimov)

Pull request description:

  ## What was done?
  Regular backports from Bitcoin Core v24

  ## How Has This Been Tested?
  Run unit & functional tests

  ## Breaking Changes
  Removed command line option `-deprecatedrpc=fees` and relevant RPC output.

  ## Checklist:
    _Go over all the following points, and put an `x` in all the boxes that apply._
  - [x] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

Top commit has no ACKs.

Tree-SHA512: 66113df43ae07ee30fd53b8967bfcb054e23e0d15a9600fdd31f29ee922f9918de3bd2e8cc593f79c61213b4934832336ce1c525b88cfaa6928b7bda258d1535
BACKPORT NOTE:
This PR doesn't actually have enabled anything relevant to anti-DoS header sync.

Pre-sync is deleted, not disabled - no HeadersSyncState, no commitment buffer,
no PRESYNC/REDOWNLOAD state.
Header acceptance hasn't changed path: CheckBlockHeader → ContextualCheckBlockHeader → ChainLock conflict check → checkpoints,
and headers are downloaded once.

This PR includes multiple useful refactorings and code changes to reduce conflicts for further backports and reduce divergency between
Dash Core codebase and Bitcoin Core.

Survived changes:
 - refactoring of GetLocator and its usages
 - changed setNumBlocks interface in qt code
 - changed interface of notification uiInterface.NotifyHeaderTip

---------------
3add234 ui: show header pre-synchronization progress (Pieter Wuille)
738421c Emit NotifyHeaderTip signals for pre-synchronization progress (Pieter Wuille)
376086f Make validation interface capable of signalling header presync (Pieter Wuille)
93eae27 Test large reorgs with headerssync logic (Suhas Daftuar)
3555473 Track headers presync progress and log it (Pieter Wuille)
03712dd Expose HeadersSyncState::m_current_height in getpeerinfo() (Suhas Daftuar)
150a548 Test headers sync using minchainwork threshold (Suhas Daftuar)
0b6aa82 Add unit test for HeadersSyncState (Suhas Daftuar)
83c6a0c Reduce spurious messages during headers sync (Suhas Daftuar)
ed6cddd Require callers of AcceptBlockHeader() to perform anti-dos checks (Suhas Daftuar)
551a8d9 Utilize anti-DoS headers download strategy (Suhas Daftuar)
ed47094 Add functions to construct locators without CChain (Pieter Wuille)
84852bb Add bitdeque, an std::deque<bool> analogue that does bit packing. (Pieter Wuille)
1d4cfa4 Add function to validate difficulty changes (Suhas Daftuar)

Pull request description:

  New nodes starting up for the first time lack protection against DoS from low-difficulty headers. While checkpoints serve as our protection against headers that fork from the main chain below the known checkpointed values, this protection only applies to nodes that have been able to download the honest chain to the checkpointed heights.

  We can protect all nodes from DoS from low-difficulty headers by adopting a different strategy: before we commit to storing a header in permanent storage, first verify that the header is part of a chain that has sufficiently high work (either `nMinimumChainWork`, or something comparable to our tip). This means that we will download headers from a given peer twice: once to verify the work on the chain, and a second time when permanently storing the headers.

  The p2p protocol doesn't provide an easy way for us to ensure that we receive the same headers during the second download of peer's headers chain. To ensure that a peer doesn't (say) give us the main chain in phase 1 to trick us into permanently storing an alternate, low-work chain in phase 2, we store commitments to the headers during our first download, which we validate in the second download.

  Some parameters must be chosen for commitment size/frequency in phase 1, and validation of commitments in phase 2. In this PR, those parameters are chosen to both (a) minimize the per-peer memory usage that an attacker could utilize, and (b) bound the expected amount of permanent memory that an attacker could get us to use to be well-below the memory growth that we'd get from the honest chain (where we expect 1 new block header every 10 minutes).

  After this PR, we should be able to remove checkpoints from our code, which is a nice philosophical change for us to make as well, as there has been confusion over the years about the role checkpoints play in Bitcoin's consensus algorithm.

  Thanks to Pieter Wuille for collaborating on this design.

ACKs for top commit:
  Sjors:
    re-tACK 3add234
  mzumsande:
    re-ACK 3add234
  sipa:
    re-ACK 3add234
  glozow:
    ACK 3add234

Tree-SHA512: e7789d65f62f72141b8899eb4a2fb3d0621278394d2d7adaa004675250118f89a4e4cb42777fe56649d744ec445ad95141e10f6def65f0a58b7b35b2e654a875

Co-authored-by: fanquake <fanquake@gmail.com>
…tidy fixup

6b24dfe CBlockLocator: performance-move-const-arg Clang tidy fixups (Jon Atack)

Pull request description:

  Fix Clang-tidy CI errors on master.  See https://cirrus-ci.com/task/4806752200818688?logs=ci#L4696 for an example.

ACKs for top commit:
  MarcoFalke:
    review ACK 6b24dfe
  vasild:
    ACK 6b24dfe

Tree-SHA512: 7a67acf7b42da07b63fbb392236e9a7be8cf35c36e37ca980c4467fe8295c2eda8aef10f41a1e3036cd9ebece47fa957fc3256033f853bd6a97ce2ca42799a0a

Co-authored-by: MacroFake <falke.marco@gmail.com>
…anti-dos header sync

30ab363 Merge bitcoin#25963: CBlockLocator: performance-move-const-arg Clang tidy fixup (Konstantin Akimov)
fbd895c partial Merge bitcoin#25717: p2p: Implement anti-DoS headers sync (Konstantin Akimov)

Pull request description:

  ## Issue being fixed or feature implemented
  This PR replaces dashpay#7318

  ## What was done?
  This PR doesn't actually have enabled anything relevant to anti-DoS header sync.

  Pre-sync is deleted, not disabled - no HeadersSyncState, no commitment buffer, no PRESYNC/REDOWNLOAD state.
  Header acceptance hasn't changed path: CheckBlockHeader → ContextualCheckBlockHeader → ChainLock conflict check → checkpoints, and headers are downloaded once.

  This PR includes multiple useful refactorings and code changes to reduce conflicts for further backports and reduce divergency between Dash Core codebase and Bitcoin Core.

  Survived changes:
     - refactoring of GetLocator and its usages
     - changed setNumBlocks interface in qt code
     - changed interface of notification uiInterface.NotifyHeaderTip

  ## How Has This Been Tested?
  Run unit & functional tests.

  ## Breaking Changes
  N/A

  ## Checklist:
  - [x] I have performed a self-review of my own code
  - [ ] I have commented my code, particularly in hard-to-understand areas
  - [ ] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation
  - [x] I have assigned this pull request to a milestone

Top commit has no ACKs.

Tree-SHA512: 718f7df78b16b0114d8f087317c298e657411e34ce41259cf8bbe217d89045f4f7965eaed7fe370fe6ec88d487d77f2679b13e6e32e687e54f941fe83c0570a0
…isolation and subsystem gating

066fca8 validation: discard snapshot state when the snapshot chainstate goes away (pasta)
6f2aabc evo: scope an open EvoDB transaction to the thread that began it (pasta)
311efbc validation: establish the connecting chainstate's BLS scheme (pasta)
5639e45 dash: only swallow unavailable-history errors from GetListForBlock (pasta)
a1fbefc dash: tolerate unavailable masternode list in scheduler-thread callbacks (pasta)
0e6172d evo: resolve transaction-less EvoDB access against the active chainstate (pasta)
7ff8231 evo: preserve EvoDB inconsistency classification across GetForBlock rethrow (pasta)
ec7ec1a evo: abort node on WriteDerived mismatch instead of consensus rejection (pasta)
2cb2a02 test: adapt AssumeUTXO fixtures after txindex removal (Pasta)
341972d fix: canonicalize serialization of block-derived EvoDB payloads (Pasta)
8010353 dash: refuse masternode duty on unvalidated snapshots (Pasta)
d3aea54 dash: guard serving unavailable snapshot history (Pasta)
f5c93ce validation: suppress background chainstate notifications (Pasta)
2fc3c51 dash: bind block validation to the calling chainstate (Pasta)
8cd55e6 test: add dual-chainstate EvoDB consistency coverage (Pasta)
3775ae4 evo: wire per-chainstate transactions into validation (Pasta)
990d950 evo: make CEvoDB consistency state per-chainstate (Pasta)

Pull request description:

  ## Issue being fixed or feature implemented

  [M1 (dashpay#7451)](dashpay#7451) added AssumeUTXO snapshot persistence, but Dash stores deterministic masternode, quorum, MNHF, and credit-pool state in a shared EvoDB. Running the snapshot and background chainstates concurrently therefore requires independent EvoDB transaction state and markers, chain-aware Dash validation, and protection against emitting or signing from the wrong chainstate.

  This is milestone 2 of the AssumeUTXO series. It supplies the Dash-specific multi-chainstate foundation required by the later background-completion and `loadtxoutset` milestones.

  ## What was done?

  - Added stable NORMAL and SNAPSHOT EvoDB identities, each with an independent transaction overlay, root batch, and best-block marker while retaining one physical EvoDB.
  - Added `WriteDerived` for immutable block-derived records. Independently derived values must serialize identically, including values pending in the other chainstate's overlay.
  - Wired block connect, disconnect, replay, verification, flushing, and snapshot activation to the calling chainstate's EvoDB identity. Snapshot activation now refuses a missing EvoDB marker instead of attaching snapshot coins to inconsistent Dash state.
  - Protected shared mined-quorum commitments from cross-chainstate erasure.
  - Passed the validating chainstate through Dash special-transaction and quorum processing. MNHF, asset-unlock, and quorum lookups now evaluate membership relative to the caller's chain instead of implicitly borrowing the active chain.
  - Suppressed active-tip, wallet, UI, and deterministic-masternode notifications from background validation. `BlockChecked` remains ungated because it reports validation results rather than active-tip changes.
  - Avoided penalizing peers when this snapshot-backed or pruned node cannot serve otherwise plausible masternode-list or quorum-rotation history.
  - Disabled DKG participation and quorum signing while an active snapshot remains unvalidated, including a final guard at the signature-share production boundary and clear RPC/status reporting.
  - Canonicalized serialization of block-derived masternode-list diffs and MNHF signals so logically identical values cannot produce false EvoDB mismatches.

  Review follow-ups (appended commits):

  - A `WriteDerived` mismatch is local EvoDB corruption, never evidence about the block. It now aborts the node with `M_ERROR` (matching the existing `EvoDbInconsistencyMessage` convention) instead of marking the block `BLOCK_CONSENSUS`-invalid and penalizing the relaying peer. A typed `EvoDbInconsistencyError` preserves that classification through the catch blocks on the miner, RPC, and MNHF-recomputation paths.
  - EvoDB access outside a `BeginTransaction` scope previously always bound to the NORMAL identity, so transaction-less consumers (RPC, mempool, miner, P2P serving) could not see snapshot-chain records pending in the SNAPSHOT overlay. `CEvoDB` now tracks a default identity that snapshot activation sets to SNAPSHOT and `ResetChainstates` resets; the background-completion milestone must reset it to NORMAL at marker promotion (TODO noted in code).
  - `GetListForBlockInternal` no longer fabricates an empty "initial snapshot" masternode list when a diff for a DIP3-active block is missing; it throws instead. The message deliberately carries the `IsBlockDataUnavailableError` sentinel and is deliberately a plain `runtime_error` rather than `EvoDbInconsistencyError`: at that layer a missing diff can be benign (pending in the other chainstate's unflushed overlay, e.g. while serving historical `mnlistdiff`), so it is reported as unavailable history without penalizing the requesting peer, and only definite mismatches abort the node. Scheduler-thread consumers (`CActiveMasternodeManager::UpdatedBlockTip`, governance trigger creation) catch it and skip the update, since an uncaught exception there would terminate the node.

  Second review round (appended commits):

  - `bls::bls_legacy_scheme` is process-wide, but the correct value belongs to the block being validated. `ConnectBlock` only saved and committed the flag; it never established it, and `ProcessSpecialTxsInBlock` switches legacy to basic only when crossing V19 forward. With an active post-V19 snapshot, the background chainstate therefore validated pre-V19 blocks under the basic scheme, and a background disconnect across V19 could commit legacy onto the active chain's consumers. `ConnectBlock` now enters under the scheme the block's parent left behind, and `ConnectTip`/`DisconnectTip` commit it only from the active chainstate.
  - The scheduler-thread guards caught every `std::exception`, so the plain `runtime_error`s that `CDeterministicMNList::ApplyDiff` raises for missing removals or updates, duplicate masternodes, and duplicate unique properties were logged as benign unavailable history while the active masternode stayed READY. The unavailable-history condition now has its own `BlockDataUnavailableError` type and is the only thing those guards (and `BuildSimplifiedMNListDiff`) catch; everything else propagates exactly as before the guards existed.
  - `CEvoDB::GetCurrentIdentity()` resolved to a process-wide `active_transaction`, so an open background NORMAL transaction redirected every concurrent transaction-less read away from the active snapshot's overlay. An open transaction is one validation execution context, not a process mode, so it now resolves only for the thread that began it.
  - Snapshot state could outlive the chainstate it describes in two ways, both fixed: `PopulateAndValidateSnapshot` commits the SNAPSHOT best-block and dual-chainstate markers as its last step, so abandoning activation afterwards (e.g. `WriteSnapshotBaseBlockhash` on an unwritable datadir) left a single-chainstate node carrying both; and `-reindex`/`-reindex-chainstate` wiped the shared EvoDB, erasing the SNAPSHOT marker while `chainstate_snapshot` stayed on disk, so startup failed on the erased marker with advice the user had already followed. Markers are now rolled back on abandoned activation, and the persisted snapshot chainstate's on-disk artifacts are discarded alongside the EvoDB wipe -- the Dash-shaped equivalent of the reindex-time cleanup in bitcoin#27596/bitcoin#29726, which runs before any chainstate has coins views and so needs no mempool transfer or leveldb-lock dance.

  The batch contains the original eight focused commits, a rebase fixture adaptation after the txindex removal, and eight review follow-up commits. The partial Bitcoin Core `BlockInfo` and `ChainstateRole` prerequisites were intentionally moved to the later `loadtxoutset` milestone where their APIs are first consumed.

  ## How Has This Been Tested?

  - Added focused EvoDB unit coverage for overlay isolation, tombstones, cross-overlay derived-value checks, independent marker flushing, disk mismatch rejection, and preservation of the legacy NORMAL marker key.
  - Added dual-chainstate integration coverage for restart consistency, snapshot-only flushing, shared quorum erasure, chain-aware commitment/quorum lookup, cache-order isolation, and suppression of background notifications.
  - Added canonical serialization coverage for deterministic masternode-list diffs.
  - Added `transaction_less_access_uses_default_identity` covering default-identity resolution of transaction-less reads and writes. Known gap: the DIP3-active missing-diff throw has no end-to-end unit test because regtest activates DIP3 at height 432, above the unit-test chain heights; it is covered by the serving-path catch and review.
  - Before the stack-only relocation of the two unused Bitcoin prerequisite commits, `make check` passed and the following functional subset passed: `feature_mnehf.py`, `feature_asset_locks.py`, `feature_dip3_deterministicmns.py` (both modes), `feature_llmq_signing.py` (both modes), `feature_llmq_rotation.py`, and `rpc_quorum.py`.
  - After the review follow-up commits: full rebuild plus `evo_db_tests`, `evo_deterministicmns_tests`, `evo_mnhf_tests`, `evo_assetlocks_tests`, and `validation_chainstatemanager_tests` pass (20 cases), and `lint-circular-dependencies` is clean.
  - After the second review round (rebased onto develop at e829261): full rebuild clean and the complete `test_dash` suite passes (815 cases). Each new regression test was negative-controlled -- `chainstate_connectblock_bls_scheme` fails on both halves of the BLS fix independently, `open_transaction_does_not_capture_other_threads` fails without the thread scoping, and `chainstatemanager_snapshot_discarded_on_reindex` reproduces the unrecoverable reindex before the fix.
  - Every rewritten commit passes `git diff-tree --check`.

  ## Breaking Changes

  No released-interface changes. Internally, `ActivateExistingSnapshot` becomes fallible so startup can reject a snapshot chainstate whose EvoDB marker is missing. Two RPC-visible additions while a snapshot is active and unvalidated: `masternode status` gains a `quorumParticipation` field and appends a "DKG participation and quorum signing disabled" clause to `status`, and `quorum sign` fails with an explanatory error. `AbstractEHFManager::Signals` changed from `std::unordered_map` to `std::map` (canonical serialization); the on-disk encoding is unchanged, but iteration order of EHF signals in `getblockchaininfo` output is now sorted.

  ## Checklist:

  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas
  - [x] I have added or updated relevant unit/integration/functional/e2e tests
  - [ ] I have made corresponding changes to the documentation (the AssumeUTXO documentation update is part of a later milestone)
  - [ ] I have assigned this pull request to a milestone

Top commit has no ACKs.

Tree-SHA512: 9acaf1a789ba06aa30b3ac2a0d8e5b8380f8e03ac1aecda14115b2c7c75af1838ba355eda91a5b8ccc91a5f9054dfb0dbfec45054ae6a9375afa88ddc7aa7f38
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch 2 times, most recently from a9e8b57 to 88c0091 Compare August 7, 2026 18:56
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch 2 times, most recently from 488db89 to 14994f4 Compare August 7, 2026 23:07
PastaPastaPasta and others added 11 commits August 7, 2026 18:56
Discard Dash snapshot lifecycle markers only after the invalid snapshot directory rename succeeds. If the rename fails, preserving the markers keeps the existing restart recovery state recognizable while the upstream rename error is propagated to the fatal shutdown message.
Keep Dash’s mock shutdown callback while asserting the expected fatal diagnostic. The default callback reaches StartShutdown(), whose unit-test guard aborts the process before this Dash test can complete.
a733dd7 Remove unused function `reliesOnAssumedValid` (Suhas Daftuar)
d4a11ab Cache block index entry corresponding to assumeutxo snapshot base blockhash (Suhas Daftuar)
3556b85 Move CheckBlockIndex() from Chainstate to ChainstateManager (Suhas Daftuar)
0ce805b Documentation improvements for assumeutxo (Ryan Ofsky)
768690b Fix initialization of setBlockIndexCandidates when working with multiple chainstates (Suhas Daftuar)
d43a1f1 Tighten requirements for adding elements to setBlockIndexCandidates (Suhas Daftuar)
d0d40ea Move block-storage-related logic to ChainstateManager (Suhas Daftuar)
3cfc753 test: Clear block index flags when testing snapshots (Suhas Daftuar)
272fbc3 Update CheckBlockIndex invariants for chains based on an assumeutxo snapshot (Suhas Daftuar)
10c0571 Add wrapper for adding entries to a chainstate's block index candidates (Suhas Daftuar)
471da5f Move block-arrival information / preciousblock counters to ChainstateManager (Suhas Daftuar)
1cfc887 Remove CChain dependency in node/blockstorage (Suhas Daftuar)
fe86a7c Explicitly track maximum block height stored in undo files (Suhas Daftuar)

Pull request description:

  This PR proposes a clean up of the relationship between block storage and the chainstate objects, by moving the decision of whether to store a block on disk to something that is not chainstate-specific.  Philosophically, the decision of whether to store a block on disk is related to validation rules that do not require any UTXO state; for anti-DoS reasons we were using some chainstate-specific heuristics, and those have been reworked here to achieve the proposed separation.

  This PR also fixes a bug in how a chainstate's `setBlockIndexCandidates` was being initialized; it should always have all the HAVE_DATA block index entries that have more work than the chain tip.  During startup, we were not fully populating `setBlockIndexCandidates` in some scenarios involving multiple chainstates.

  Further, this PR establishes a concept that whenever we have 2 chainstates, that we always know the snapshotted chain's base block and the base block's hash must be an element of our block index. Given that, we can establish a new invariant that the background validation chainstate only needs to consider blocks leading to that snapshotted block entry as potential candidates for its tip. As a followup I would imagine that when writing net_processing logic to download blocks for the background chainstate, that we would use this concept to only download blocks towards the snapshotted entry as well.

ACKs for top commit:
  achow101:
    ACK a733dd7
  jamesob:
    reACK a733dd7 ([`jamesob/ackr/27746.5.sdaftuar.rework_validation_logic`](https://github.com/jamesob/bitcoin/tree/ackr/27746.5.sdaftuar.rework_validation_logic))
  Sjors:
    Code review ACK a733dd7.
  ryanofsky:
    Code review ACK a733dd7. Just suggested changes since the last review. There are various small things that could be followed up on, but I think this is ready for merge.

Tree-SHA512: 9ec17746f22b9c27082743ee581b8adceb2bd322fceafa507b428bdcc3ffb8b4c6601fc61cc7bb1161f890c3d38503e8b49474da7b5ab1b1f38bda7aa8668675
Preserve ChainLock candidate exclusions in the new admission wrapper and keep Dash background-notification and EvoDB fixtures consistent with the tightened multi-chainstate candidate invariants.
Peer-penalty exemption for unavailable history hinged on three files repeating one literal string that IsBlockDataUnavailableError() then substring-matched; rewording any copy would silently revert those paths to Misbehaving. Define the suffix once next to BlockDataUnavailableError and use it at every producer and in the matcher.
Both chainstates carried a live mempool pointer after snapshot activation, so background ConnectTip called removeForBlock and removeExpiredAssetUnlock with historical blocks and lower heights. Follow the bitcoin#27596 shape: only the active chainstate keeps the mempool. Runtime activation transfers it to the snapshot chainstate, restart activation clears it from the background chainstate, and the invalid-snapshot revert hands it back.
The background chainstate in this fixture is reset to genesis before activation, so the base MN list is not derivable; a capture at activation would fabricate an empty list and poison the shared list cache (thepastaclaw review finding).
@PastaPastaPasta
PastaPastaPasta force-pushed the assumeutxo/m3-background-completion branch from 14994f4 to e3a8989 Compare August 8, 2026 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants