Skip to content

fix(net): backport the joined-head FETCH race fix to release - #7680

Merged
kixelated merged 4 commits into
releasefrom
backport/fill-head-race
Nov 5, 2026
Merged

kixelated merged 4 commits into
releasefrom
backport/fill-head-race

Conversation

@kixelated

@kixelated kixelated commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

On release, a cache-miss FETCH can outrun the subscription's joining fill. The fetched group then occupies the cache, the live tail is refused as a duplicate, and live readers wait for an end that never arrives.

Approach

Hand-port the settled relay fix from moq-dev/moq#7135. Only the group named by SUBSCRIBE_OK waits for its own head. The cache answers an arriving head; a settled fill with a remaining miss permits a FETCH. Reader cancellation and track termination release the wait. The PUBLISH_DONE grace also polls queued misses, so a head that settles there can release its FETCH.

The subscriber coverage uses release's paused Tokio harness. No main-only range-fill or subscription-widening machinery is ported because release has neither. Main quest bookkeeping stays in moq-dev/moq#7669.

Impact

  • Public API: unchanged; the request completion helper is crate-private.
  • Wire: unchanged; this suppresses redundant FETCH requests without changing their encoding.
  • No package version changes, tags, or publication.

Alternatives

Cherry-picking would import main's incompatible track-fill model. Delaying all cache misses would stall older history behind an unrelated head, so only the joined group waits.

Validation

  • Reproduced on unfixed release: both joined-head miss tests emitted an extra FETCH; the older-history control passed.
  • All 13 joining-fetch tests pass, including six added cases for waiting, fallback, cache delivery, older history, cancellation, track abort, and PUBLISH_DONE grace.
  • The strict release CI test half passes: 5,188 default tests, 387 capture tests, and 133 CLI play/capture tests.
  • cargo semver-checks check-release -p moq-net --baseline-rev origin/release --release-type patch --all-features: 229 checks pass; no semver update required.
  • The strict release CI check half passes, including WASM, selected device configurations, and repository lints. Checks run through release's Nix shell with BASH_ENV unset inside it, so a cached Codex shell snapshot cannot replace the pinned toolchain.
  • After merging origin/release (d6ffbbb69b) conflict-free into c59489c5da: cargo clippy -p moq-net --all-targets --all-features -D warnings is clean, and cargo test -p moq-net --all-features passes 1,613 tests, including all 13 joining-fetch tests.

Review

  • Head c59489c5da only adds a conflict-free merge of origin/release. git diff origin/release...c59489c5da is byte-identical to git diff origin/release...87e8ef9c8e, so the round 2 record below covers it.
  • Latest: Claude Code (quest review), Claude Opus 5.5, medium effort, round 2 against origin/release, reviewed 87e8ef9c8e: no findings.
  • Round 1 dispositions, accepted in round 2:
    • Missing-head fallback: refuted. The settled design deliberately leaves an absent head costing only its own group, until its reader cancels or the subscription ends; see the original review response.
    • Older-history ownership redesign: refuted for this backport. Older history must FETCH immediately, as the quest Plan specifies. The review's Absolute-join collision is pre-existing and degrades a later join rather than causing the joined-head live-reader hang. It is reported as a suggested main follow-up.
    • PUBLISH_DONE grace: fixed by polling queued misses during settlement. The new paused-time test reproduced the missing FETCH before this fix, and now passes.

Main-first prerequisite

Cleared 2026-11-05 by moq-dev/moq#7903, merged as edc7ef73ca. It adds the same group_fetches.poll(waiter) to main's PUBLISH_DONE grace loop, just before poll_settled, matching this PR's poll. It also adds a_miss_fetches_during_the_publish_done_grace, which failed on unfixed main. Its other change drives a lingering copy's fetches during resumed SUBSCRIBE setup. That has no release counterpart, because release creates its fetch set after SUBSCRIBE_OK and never lingers.

Follow-ups

  • The main quest (quest/m0/backport-fill-head-race.md) keeps the maintainer's merge into release and the moq-net patch publication.
  • Suggested follow-up quest: verify and fix the pre-existing collision between an Absolute joining fill and an older-history FETCH on main. The local reviewer identified a possible degraded join; the joined-largest-group hang is covered here.
  • Main bookkeeping also records a test-discovery follow-up: Bun filename filters discovered the API checker's generated release source tree while checking main.
  • Suggested follow-up quest: prevent a cached Codex shell snapshot from overriding an explicit Nix shell after switching branches. The cached toolchain lacked the WASM target; the declared release toolchain has it.

(Written by GPT-6.1 Sol and Claude Opus 5.5)

kixelated and others added 3 commits November 2, 2026 21:45
Hand-port the relay fix from #7135 onto release, with paused-time coverage for the head wait, fallback, older history, cancellation, and track abort.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Poll pending cache misses while PUBLISH_DONE waits for reordered data streams, so a settled joining head can release its miss to FETCH during the grace. Cover that lifecycle in the paused-time subscriber harness.

Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated
kixelated marked this pull request as ready for review November 5, 2026 20:15

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit c59489c5da6ead7395894ec11d1d821639d1c44f against release base d6ffbbb69b29e492734bd04da07efe3c3666c8ea. No earlier published review was present on this PR.

No actionable findings. Direction: keep this narrow backport. Waiting only for the named joined group addresses the competing-producer race without importing main's range-fill machinery or blocking unrelated history. The cancellation and unrelated-history concerns from #7135 are addressed here: the wait observes reader withdrawal and terminal track state, and cache resolution retires the request without rejecting its readers. The added PUBLISH_DONE poll keeps already queued misses moving during settlement and matches merged #7903.

The new helper remains crate-private, with no wire-format change. Tests cover waiting/fallback, cache delivery, older-history progress, cancellation, track abort, and grace-period progress.

Verification: GitHub-only static review of the changes, surrounding lifecycle/cache code, tests and prior findings, with an independent cancellation/locking cross-check. Tests and interoperability were not run; the new head-arrival test injects internal fill state. Author-reported local results were not independently reproduced. GitHub Check/Test are still running; an earlier release run has a failed gate following a cancelled Scope job.

(Written by OpenAI)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Backport of the joined-head FETCH race fix (#7135, with #7903's PUBLISH_DONE poll) onto release at c59489c5da, including the conflict-free merge of origin/release (d6ffbbb69b). The diff matches the claimed scope exactly: two files, no stray changes from the merge.

  • Joined-head gate — wait_group_head parks a cache-miss FETCH for the group named by SUBSCRIBE_OK's Largest until the subscription's own fill settles, then re-checks the cache and only runs run_group_fetch for a remaining miss. Older history, a missing subscription, a settled fill, and a departed reader all fall through without waiting.
  • Request resolution — new crate-private group::Request::poll_resolved reports when the cache or a terminal track state already answers the request and retires the attempt without writing a rejection, so readers that resolve through the track are not failed.
  • PUBLISH_DONE grace — the grace loop now polls group_fetches before poll_settled, so a head that settles during settlement can release its FETCH instead of waiting for teardown.
  • Tests — JoinRun::subscribe factored out of start, plus six paused-time cases covering the wait, cache delivery, older history, reader cancellation, track abort, and the grace.

I re-ran the suite locally and mutation-checked the two load-bearing hunks: cargo test -p moq-net --all-features --lib joining_fetch passes 26 tests; returning Some(request) immediately from wait_group_head fails five of the new cases (the older-history control still passes), and deleting the new group_fetches.poll(waiter) from the grace loop fails the grace case.

The outstanding behavior — a fill that never settles parks that one group's miss until the reader cancels or the subscription ends — is the design settled on main, and the Absolute-join collision below Largest remains pre-existing with a follow-up already recorded. No new evidence to reopen either.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

@kixelated
kixelated merged commit dc3478a into release Nov 5, 2026
34 of 36 checks passed
@kixelated
kixelated deleted the backport/fill-head-race branch November 5, 2026 21:08
@pullfrog pullfrog Bot mentioned this pull request Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant