Skip to content

Feat/pipeline parallelize fastsync - #59

Open
secretnamebasis wants to merge 2 commits into
DEROFDN:community-devfrom
secretnamebasis:feat/pipeline-parallelize-fastsync
Open

Feat/pipeline parallelize fastsync#59
secretnamebasis wants to merge 2 commits into
DEROFDN:community-devfrom
secretnamebasis:feat/pipeline-parallelize-fastsync

Conversation

@secretnamebasis

Copy link
Copy Markdown

Description

bootstrap_chain()'s --fastsync state-tree fetch (both the balance-tree phase and the smart-contract-tree phase) fetched one Peer.TreeSection chunk at a time, waiting for each round-trip before requesting the next. This PR pipelines both phases: up to 16 concurrent Peer.TreeSection requests are in flight at once, with results drained in completion order rather than strict index order.

Draining in completion order also fixes a real bug in a naive "just add concurrency" approach: waiting on requests in strict index order means one slow chunk head-of-line-blocks visibility into other chunks that already finished, which mostly cancels out the benefit of firing them concurrently in the first place. Chunk writes are commutative (each writes a disjoint key range into the same tree), so there's no ordering requirement — only the resume checkpoint needs to stay correct, which is now tracked via a low-water-mark instead of the last-fired index.

Only the network round-trips (connection.Client.Call) run inside worker goroutines. All graviton Tree/Snapshot reads and writes (GetTree, Put, Commit) stay on the single calling goroutine as results are drained — there is no concurrent access to graviton objects at any point, since that thread-safety was never something I could verify.

Live-tested against a real mainnet peer:

  • Balance-tree phase: previously serial at ~11-12s/chunk; now completes in a small fraction of a second per test run (effectively bound by the peer's own response time under a full concurrent window, not by client-side waiting).
  • Smart-contract-tree phase: ~2.9x faster (0.159%/sec vs ~0.056%/sec baseline), measured over 98 seconds of continuous progress against a real peer.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update
  • This will require a HardFork to be enabled

Which part is impacted?

  • Wallet
  • Daemon
  • Miner
  • Explorer
  • Simulator
  • Misc (documentation, comments, text...)

Checklist:

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings

License

I am contributing & releasing the code under DERO Research License (which can be found here).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant