Skip to content

fix(funding): wait for message pool readiness for all funded accounts - #207

Merged
beck-8 merged 2 commits into
mainfrom
beck-8/fix-multicall3-deploy-race
Sep 18, 2026
Merged

beck-8 merged 2 commits into
mainfrom
beck-8/fix-multicall3-deploy-race

Conversation

@beck-8

@beck-8 beck-8 commented Sep 17, 2026

Copy link
Copy Markdown
Member

Summary

  • probe the deployer with cast nonce --block pending before forge create
  • retry only the read-only probe: up to 6 attempts, 4s apart

Rationale

Deploy Multicall3 Contract intermittently fails during broadcast:

Error: server returned an error response: error code 1: failed to look up actor state
nonce: resolution lookup failed (t410fa34eb7lgf6xovtgfnsnoe6sz67ca2zjkbwi22ly): resolve
address t410fa34eb7lgf6xovtgfnsnoe6sz67ca2zjkbwi22ly: actor not found: validation failure

Funding verification uses lotus wallet balance, which reads the chain head, while the
message pool's tipset (mp.curTs) can lag by the coalescer's 2-6s delay.

--block pending is essential: it alone routes through MpoolGetNonce to the same
getStateNonce(ctx, addr, mp.curTs) lookup used by broadcast. Every other block
parameter returns 0, nil for a missing actor, silently hiding the race.

The probe is retried because it is a read-only query; forge create stays outside the
retry, as src/utils/retry.rs requires these helpers never wrap transactions or
deployments.

Both deployments start in the same parallel epoch, but MockUSDFC spends ~7s installing
dependencies and compiling before it broadcasts, which clears the coalescer window.
Multicall3 compiles in ~130ms and broadcasts immediately. In nightly run 35198060438 the
broadcast failed 0.6s after funding verification had reported DEPLOYER_MULTICALL3: 10 FIL
for that same account: the balance was already visible on the head while the pool still
could not resolve the actor.

Confirmed in 24 jobs across 20 of the 100 nightly runs between June 27 and September 17,
hitting one or two matrix legs at a time. Earlier occurrences are likely but no longer
verifiable, as Actions logs from before late June have expired. No issues were ever
filed for it, because Start cluster uses continue-on-error: true and the step that
marks the job failed runs after issue reporting has already decided not to file.

Verification

  • cargo fmt --all -- --check
  • cargo clippy --all-targets --all-features -- -D warnings
  • cargo test --all-targets --all-features (50 passed)

Not verified end to end against a live devnet. On a clean run the log will show
✓ Message pool resolves the nonce for 0x06f840fd662faeeaccc56c9ae27a59f7c40d652a. If
the race is hit, Multicall3 deployer nonce lookup failed (attempt 1/6), retrying in 4 seconds... appears first; the deployment proceeds only once a probe succeeds, and six
consecutive failures abort the step instead.

🤖 Generated with Claude Code

`forge create` intermittently failed while broadcasting the deployment:

  failed to look up actor state nonce: resolution lookup failed
  (t410fa34eb7lgf6xovtgfnsnoe6sz67ca2zjkbwi22ly): actor not found

The funding step sleeps for a fixed interval and then verifies the transfers
with `lotus wallet balance`, which reads the chain head. The message pool
tracks its own tipset (`mp.curTs`), updated through a coalescer with a 2-6s
delay (HeadChangeCoalesceMinDelay/MaxDelay), so it can still trail the head
while the balance is already visible. `eth_sendRawTransaction` resolves the
sender's nonce against that lagging view, and fails.

Deploy Multicall3 and Deploy MockUSDFC share a parallel epoch, but MockUSDFC
spends ~27s installing dependencies before it broadcasts, so only Multicall3
lands inside the window — it compiles in ~130ms and broadcasts immediately.

Probe with `cast nonce --block pending`, the only query that goes through the
same `getStateNonce(ctx, addr, mp.curTs)` call as the broadcast; every other
block parameter silently returns 0 for a missing actor. The probe is read-only
so it is retried; the deployment itself still runs exactly once.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@beck-8
beck-8 requested a review from rvagg as a code owner September 17, 2026 14:34
@FilOzzy FilOzzy added this to FOC Sep 17, 2026
@github-project-automation github-project-automation Bot moved this to 📌 Triage in FOC Sep 17, 2026
@FilOzzy FilOzzy moved this from 📌 Triage to 🔎 Awaiting review in FOC Sep 17, 2026
@rvagg

rvagg commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Good find but I think it needs to be broadened. Every account funded at epoch 4 has this same problem and it's just multicall3 that's the first to need the funds. So instead, it seems we should move this check into ETHAccFundingStep::post_execute and do it per-account there next to the balance verification, and it could be a reusable helper to do this. The check is then a generic "mpool will accept a send from this account".

@beck-8 beck-8 changed the title fix(multicall3): wait for message pool nonce readiness before deploying fix(funding): wait for message pool readiness for all funded accounts Sep 18, 2026
@github-project-automation github-project-automation Bot moved this from 🔎 Awaiting review to ✔️ Approved by reviewer in FOC Sep 18, 2026
@beck-8
beck-8 merged commit fc02d6d into main Sep 18, 2026
3 checks passed
@beck-8
beck-8 deleted the beck-8/fix-multicall3-deploy-race branch September 18, 2026 06:57
@github-project-automation github-project-automation Bot moved this from ✔️ Approved by reviewer to 🎉 Done in FOC Sep 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🎉 Done

Development

Successfully merging this pull request may close these issues.

3 participants