Repository navigation
quest: plan the land fixer epic from the handback audit #354
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| # The land fixer handles every mechanical failure itself | ||
|
|
||
| ## Goal | ||
|
|
||
| When a land batch fails or a queued PR conflicts, the gate and its fixer | ||
| resolve every mechanical cause themselves and land the result, so a | ||
| maintainer sees only behavior and design decisions. Nothing leaves the queue | ||
| for a conflict or a failure: a PR the fixer is working on keeps its | ||
| auto-merge and its place, sits out batches until the fix is in, and leaves | ||
| the queue only when the fixer hands it back. The fixer blames the failing PR | ||
| before anything else happens. The gate merges locally instead of | ||
| trusting GitHub's mergeability, reruns a failed module before blaming, and | ||
| disables a flaky test with a follow-up quest. The fixer builds and tests | ||
| with access to package registries, edits protected paths when the edit is | ||
| mechanical, and spends a pick's single attempt only on a fix that failed. | ||
| When GitHub refuses its push to a fork, it hands back a ready commit. | ||
|
|
||
| Non-goals: design conflicts. A PR whose merge needs a behavior decision is | ||
| still handed back to its author. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-01 (user, /quest-plan), from an audit of the 406 fixer | ||
| handbacks posted 2026-10-30 to 11-01 on moq-dev/moq, moq-dev/moq.pro, | ||
| moq-dev/rust2, moq-dev/rind and moq-dev/quest. | ||
|
|
||
| | Count | Cause | Owner | | ||
| |---|---|---| | ||
| | 108 | fixer crashed (exit 101, EAGAIN) | fixed by [moq-dev/quest#320](https://github.com/moq-dev/quest/pull/320) and [#332](https://github.com/moq-dev/quest/pull/332); none after 2026-10-31 21:00Z | | ||
| | 16 | fixer disappeared before recording an exit status | fixed on the host; none after 2026-10-31 04:00Z | | ||
| | 7 | fixer repository exceeds its size cap | fixed by #332 | | ||
| | 6 | AGENTS.md "pull the latest origin changes" read as an injection | fixed by [#325](https://github.com/moq-dev/quest/pull/325) and [#328](https://github.com/moq-dev/quest/pull/328); none after 2026-10-31 18:46Z | | ||
| | 98 | "already used its mechanical fix attempt" after a collateral requeue or an earlier conflict fix; each needed a manual re-pick | [Attempt budget](/quest/m0/land-fixer/attempts.md), [Batch triage](/quest/m0/land-fixer/triage.md) | | ||
| | ~75 | "requeued unchanged": collateral of a batch another PR broke | [Batch triage](/quest/m0/land-fixer/triage.md) | | ||
| | 36 | picked again while fixing; the newer pick is kept, so nothing is lost | not planned | | ||
| | 17 | conflicts in protected paths (`flake.nix`, `.github/`), mostly mechanical; resolved by hand on moq-dev/moq#7344, #7315, #7318 | [Protected edits](/quest/m0/land-fixer/protected.md) | | ||
| | 17 of the last 40 jobs | `cargo` fails offline ("failed to get `uniffi`"), so Rust fixes were requeued uncompiled: moq-dev/moq#7357 sank three batches, and #6985's lockfile could not be regenerated | [Registry network](/quest/m0/land-fixer/network.md) | | ||
| | 2 | GitHub refused the push to a fork because the merge touches workflows: moq-dev/moq#7439, #7467 | [Fork commits](/quest/m0/land-fixer/fork-commit.md) | | ||
| | several batches | a flaky ffmpeg timeout ([moq-dev/moq#7480](https://github.com/moq-dev/moq/pull/7480)) ejected whole batches | [Flaky tests](/quest/m0/land-fixer/flaky.md) | | ||
| | ~15 | design conflicts, correctly handed back | none | | ||
|
|
||
| Decision prompts, selected options marked ✅: | ||
|
|
||
| - Goal: "the fixer resolves every mechanical failure on its own" | ||
| - Yes, as stated (triage and local mergeability stay separate) | ||
| - Narrower: only the attempt budget and offline builds | ||
| - ✅ Broader: also fold [Batch triage](/quest/m0/land-fixer/triage.md) | ||
| and [Local mergeability](/quest/m0/land-fixer/mergeability.md) into | ||
| this epic | ||
| - How attempts count: see [Attempt budget](/quest/m0/land-fixer/attempts.md). | ||
| - How the fixer gets dependencies: see [Registry network](/quest/m0/land-fixer/network.md). | ||
| - Protected paths: see [Protected edits](/quest/m0/land-fixer/protected.md). | ||
| - Fork pushes GitHub refuses: see [Fork commits](/quest/m0/land-fixer/fork-commit.md). | ||
| - Flaky tests: see [Flaky tests](/quest/m0/land-fixer/flaky.md). | ||
| - Milestone for the new quests: ✅ m0 · m1 | ||
| - The user, starting the work: "The fixer is meant to handle conflicts and | ||
| failures without kicking stuff out of the queue." A conflict or a failing | ||
| check therefore never ejects a PR; the gate parks it (no auto-merge change, | ||
| no ejection comment) while its fixer job runs, and only a handback | ||
| disables auto-merge. [Batch triage](/quest/m0/land-fixer/triage.md) and | ||
| [Local mergeability](/quest/m0/land-fixer/mergeability.md) follow this | ||
| rule. | ||
|
|
||
| The README's landing section and `CONTRIBUTING.md` list what the fixer | ||
| resolves itself and what it hands back; that docs update finishes the epic. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Attempt budget](/quest/m0/land-fixer/attempts.md) - only a fix that fails again spends a pick's single attempt | ||
| - [Registry network](/quest/m0/land-fixer/network.md) - the fixer fetches dependencies, so it builds and tests what it requeues | ||
| - [Batch triage](/quest/m0/land-fixer/triage.md) - the fixer blames the failing PR before ejection, and a red base halts the queue | ||
| - [Flaky tests](/quest/m0/land-fixer/flaky.md) - the gate reruns a failed module, disables a flaky test in the next batch, and files a follow-up | ||
| - [Local mergeability](/quest/m0/land-fixer/mergeability.md) - `quest land` merges with the quest driver instead of ejecting on GitHub's conflict flag | ||
| - [Protected edits](/quest/m0/land-fixer/protected.md) - the fixer edits protected paths when it judges the edit mechanical, and the fix lands | ||
| - [Fork commits](/quest/m0/land-fixer/fork-commit.md) - a fork push GitHub refuses hands back a commit a maintainer session applies as is | ||
|
|
||
| ## Related | ||
|
|
||
| - [Merge through a host gate](/quest/m0/host-land/README.md) - the gate and fixer this epic extends |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # [S] Only a failed fix spends a pick's attempt | ||
|
|
||
| ## Goal | ||
|
|
||
| A maintainer pick's single fixer attempt is spent only when a head the fixer | ||
| pushed fails again, or when the fixer hands back. A "requeued unchanged" | ||
| verdict, and a fix that only merged the base (cleanly or with resolved | ||
| conflicts) leave the attempt unspent. A PR that conflicts again as the base | ||
| moves is then fixed again without a re-pick. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-01 (user, /quest-plan of the land fixer audit; see the | ||
| [epic](/quest/m0/land-fixer/README.md)). In 2026-10-30 to 11-01, 98 handbacks | ||
| said only "This maintainer pick already used its mechanical fix attempt", and | ||
| each needed a manual re-pick. Most had spent the attempt on a requeue-unchanged | ||
| verdict as collateral of another PR's failure, or on an earlier conflict fix | ||
| before the base moved again. | ||
|
|
||
| - How attempts count: | ||
| - ✅ Only failed fixes count: requeue-unchanged and base-merge fixes leave | ||
| the attempt unspent; it is spent when a pushed fix fails again, or on a | ||
| handback | ||
| - One attempt per head | ||
| - N attempts per pick (for example 3) | ||
| - Keep one per pick | ||
|
|
||
| The attempt record is `fix-attempt-<n>.json` (`src/cli/land-fixer.ts`): its | ||
| `pick`, `requeued`, and `head` fields, read by `handbackRepeated`. Add an | ||
| `outcome` field (`unchanged`, `merged`, or `fixed`) and have | ||
| `handbackRepeated` count only `fixed`. A record written before this field | ||
| existed has no `outcome` and counts as spent, as today, so a deploy never | ||
| hands a pick a second attempt it already used. A head the fixer changed beyond a base merge that then fails its | ||
| checks still hands back, so a bad fix cannot loop. | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| [Batch triage](/quest/m0/land-fixer/triage.md) keeps a collateral ejection | ||
| from reaching the fixer at all; this quest owns the counting rule for every | ||
| outcome, including the ones triage still produces. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Batch triage](/quest/m0/land-fixer/triage.md) - blames one PR, so collateral PRs are not ejected | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| # [M] The gate reruns a failure, then disables a flaky test | ||
|
|
||
| ## Goal | ||
|
|
||
| When a batch's check fails, `quest land` reruns the failing check modules | ||
| once on the same tip before blaming or ejecting anything. A test that failed | ||
| and then passed, or whose failure the fixer judges nondeterministic (for | ||
| example a timeout under load, or a race), is flaky. The gate lands the batch, | ||
| adds a commit to the next batch that disables the test with a reason, and | ||
| files a follow-up quest to fix or delete it in the repository's m0 `flaky` | ||
| epic. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-01 (user, /quest-plan of the land fixer audit; see the | ||
| [epic](/quest/m0/land-fixer/README.md)). A flaky ffmpeg render timeout in | ||
| moq's interop tests ejected whole batches on 2026-11-01. The fixer then | ||
| recognized it as flaky, but only after the batch's PRs had been ejected and | ||
| their attempts spent. The fix ([moq-dev/moq#7480](https://github.com/moq-dev/moq/pull/7480)) | ||
| was written and landed by hand. | ||
|
|
||
| - What the gate does with a flaky failure: | ||
| - ✅ Rerun the failed module once | ||
| - The fixer names the flake | ||
| - Nothing new | ||
| - User follow-up: "make a rule to immediately disable any flakey tests and | ||
| create a follow-up to fix/delete them". | ||
| - Who disables: | ||
| - The fixer opens and queues its own PR | ||
| - The fixer opens a PR, and a maintainer picks it | ||
| - ✅ Inside the next batch: the gate adds the skip commit to the batch it | ||
| lands next, tested with it | ||
| - What counts as flaky: | ||
| - Only a rerun that passes | ||
| - ✅ A rerun that passes, or a failure the fixer judges nondeterministic | ||
| - Where the follow-up goes: | ||
| - ✅ A standing `flaky` epic in the repository's m0, one quest per | ||
| disabled test | ||
| - The earliest milestone, ungrouped | ||
| - m1 | ||
|
|
||
| How the gate reruns one module is up to the repository. `quest land` passes | ||
| the failed module names to the check command (for example through | ||
| `QUEST_LAND_RERUN`). A check command that ignores them reruns whole. moq's | ||
| `sh/dispatch.sh` already reports `ci: <action> <module> failed`. | ||
|
|
||
| The fixer writes the skip in the test framework's own form, with the reason | ||
| naming the follow-up quest: `#[ignore = "flaky: /quest/m0/flaky/<name>.md"]`, | ||
| `test.skip`, `pytest.mark.skip`, and so on. A disabled test that is a quest's | ||
| acceptance test still counts as its quest's work. The follow-up quest's Goal | ||
| says to fix the test or delete it, and links the batch logs that showed the | ||
| flake. | ||
|
|
||
| Flaky verdicts never spend a PR's attempt | ||
| ([Attempt budget](/quest/m0/land-fixer/attempts.md)), and a batch whose only | ||
| failure was flaky lands without ejecting anyone | ||
| ([Batch triage](/quest/m0/land-fixer/triage.md)). | ||
|
|
||
| ## Related | ||
|
|
||
| - [Batch triage](/quest/m0/land-fixer/triage.md) - blame runs after the rerun |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| # [S] A refused fork push hands back a ready commit | ||
|
|
||
| ## Goal | ||
|
|
||
| When GitHub refuses the fixer's push to a fork because the commit touches | ||
| `.github/workflows` (the App has no `workflows` permission), the handback | ||
| carries the fixer's validated resolution. A maintainer session applies and | ||
| pushes it without redoing the merge. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-01 (user, /quest-plan of the land fixer audit; see the | ||
| [epic](/quest/m0/land-fixer/README.md)). On | ||
| [moq-dev/moq#7439](https://github.com/moq-dev/moq/pull/7439) and | ||
| [#7467](https://github.com/moq-dev/moq/pull/7467) the fixer's merge of the | ||
| base brought in the base's workflow changes, and GitHub refused the push. A | ||
| maintainer session recovered #7439's commit from the daemon's | ||
| `repository.git` through a temporary ref and a bundle over SSH. | ||
|
|
||
| - What happens on a refused fork push: | ||
| - Grant moq-bot the `workflows` permission | ||
| - ✅ Hand back a ready commit | ||
| - Leave it manual | ||
|
|
||
| In any repository where the App lacks the `workflows` permission, GitHub | ||
| refuses the App's push of such a commit, so the App cannot publish it on a | ||
| fork or an upstream ref. The commit itself can still travel through Git, as | ||
| the bundle recovery showed. The handback comment records the PR head SHA, | ||
| the base SHA, the expected tree hash, and the fixer's resolution as a patch | ||
| against the merge's conflicted result. A command such as `quest land apply | ||
| <pr>` stops if the PR's head is no longer the recorded one, then merges that | ||
| base, applies the patch, checks the tree hash, and leaves a commit for the | ||
| maintainer to push. Size the comment to GitHub's limit and | ||
| hand back plainly when the patch does not fit. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Protected edits](/quest/m0/land-fixer/protected.md) - the fixer's own workflow edits |
8 changes: 5 additions & 3 deletions
8
quest/m0/land-mergeability.md → quest/m0/land-fixer/mergeability.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # [S] The fixer reaches package registries | ||
|
|
||
| ## Goal | ||
|
|
||
| The fixer container can fetch dependencies from the package registries a | ||
| repository's lockfiles name, so `cargo`, `bun`, `uv`, and `go` build and test | ||
| a fix before the fixer reports `fixed` or `requeue`. It does not use GitHub's | ||
| API or push, and the only credential in the container stays the gate's | ||
| CLIProxyAPI client key ([Fixer proxy access](/quest/m0/host-land/fixer.md)). | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-01 (user, /quest-plan of the land fixer audit; see the | ||
| [epic](/quest/m0/land-fixer/README.md)). In 17 of moq's last 40 fix jobs, | ||
| `cargo` failed offline with "failed to get `uniffi` as a dependency", a git | ||
| dependency the container cannot fetch. The fixer requeued Rust fixes it never | ||
| compiled ([moq-dev/moq#7357](https://github.com/moq-dev/moq/pull/7357) sank | ||
| three batches), and could not regenerate | ||
| [moq-dev/moq#6985](https://github.com/moq-dev/moq/pull/6985)'s lockfile. | ||
|
|
||
| - How the fixer gets dependencies: | ||
| - Host prefetches the job's lockfile dependencies and mounts them read-only | ||
| - ✅ Allow registry network | ||
| - Leave it | ||
|
|
||
| Found while starting the work (2026-11-01): the host is not what blocks it. | ||
| kixelated/nixos `modules/ci-egress.nix` gives the fixer container | ||
| (`qland-fix`) the same egress as the check containers, the public internet | ||
| without private ranges, and the fixer runs Codex with | ||
| `--dangerously-bypass-approvals-and-sandbox`. The fixer chooses `--offline` | ||
| itself because `assets/land-fix.md` says "Never push or contact GitHub", and | ||
| a git dependency's fetch from `github.com` is contact. So the change is the | ||
| prompt: fetching dependencies from package registries and from | ||
| `github.com`/`codeload.github.com` for git dependencies is allowed, while | ||
| GitHub's API, pushing, and any other contact stay forbidden. | ||
| [Fixer proxy access](/quest/m0/host-land/fixer.md)'s "network access only to | ||
| CLIProxyAPI" describes the intent, not the host as built; correct it in the | ||
| same PR. | ||
|
|
||
| `assets/land-fix.md` then tells the fixer to build and test the packages its | ||
| fix touches, and to report what it ran in the result's reason. A fix it could | ||
| not verify hands back with the command's output. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Fixer proxy access](/quest/m0/host-land/fixer.md) - the container's existing network rules |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.