[kixelated]

Completes quest/m0/queue-watcher/flaky.md: a flaky failure is re-enqueued unchanged (which the watcher already did), and a test seen flaking twice is now disabled in a separate PR with a follow-up quest. The watcher enqueues that PR with no human pick.

Changes

• Per-repository flaky record (src/cli/land-watch-flaky.ts): the watcher stores each flaky verdict in flaky.json in its state directory. Verdicts are grouped by the fixer's failure signature, across every PR, and keep the failing merge_group job URLs. A fixer round's job file now carries those URLs (runs).

• Fixer input: before each failed-build round, the watcher writes the tests that have flaked and are not yet disabled to .git/quest-land-flaky.json in the fixer's checkout. That file is bounded to 50 KB, most recently seen first, with each test's last 10 verdicts' jobs. quest land fix reads it (bounded, no symlinks) and appends it to the agent's input as Known flaky tests: [...]. The nixos fixer client already tars the whole checkout, .git/ included, so the host needs no change.

• Prompt (assets/land-fix.md, new ## Flaky tests section): when a listed test flakes again, the fixer reuses its signature and commits, on a disable branch from base:

  • the skip, in the framework's own form, naming the follow-up quest (#[ignore = "flaky: /quest/m0/flaky/<name>.md"], test.skip, pytest.mark.skip);
  • the quest quest/m0/flaky/<name>.md, whose Goal is to fix the test or delete it, linking the jobs that showed the flake;
  • its link in the quest/m0/flaky/README.md epic, creating the epic if it is missing.

  It returns the commit as "disable": "<sha>" on its requeue entry. parseFixResult accepts disable only on a requeue.

• Validation (recordFlaky): the round's verdict is recorded first. A disable is taken only once the test has 2 verdicts; one verdict alone is ignored. A taken disable is imported and checked with validateFix against the base, so it must descend from the base and leave every protected path untouched, and it must add exactly one quest/m0/flaky/<name>.md. A refused disable waits for the test's next flake.

• Publication (advanceDisables in src/cli/land-watch-fix.ts): the watcher lease-pushes the disable to quest-land/flaky-<sha12> and opens a PR on the base it was built on. The PR body lists the verdicts with their job links and a ## Review record of the mechanical change. The watcher then enqueues the PR (or arms auto-merge) through the same requeue it uses for fixed PRs. Each step is recorded, so a restart repeats none. A failed push or a GitHub error is retried on the next pass. An enqueue GitHub refuses is reported on the disable PR, which then waits for a maintainer. If the queue later drops the disable PR, the watcher handles it like any other PR.

• Docs and tree: the README's watcher section describes the record. The quest is deleted, and its links in the queue-watcher and land-fixer epics now point here.

Choices made without asking

• One round, not a second job. The watcher tells the fixer the known flaky tests ahead of time, so the round that gives the second verdict also returns the disabling commit. A separate disable job would cost another container run and a new mode signal through the nixos wrapper.
• What counts as a verdict. Only applied rounds count, including one a stop rule then hands back. A round whose pick ended while it ran is not counted.
• What is validated. Beyond validateFix's protected paths, the watcher only checks that the disable adds exactly one follow-up quest. It does not check the skip itself.
• Branch name. The disable branch is named by commit, quest-land/flaky-<sha12>, so it never collides with a stale branch.
• No retry after a decision. A disable PR that a maintainer closes, or whose enqueue GitHub refuses, is not proposed again for that test.
• Acceptance tests. I read "a disabled test that is a quest's acceptance test still counts as its quest's work" as: leave that quest as it is. The prompt says so.

Tests

• test/land-watch.test.ts: an end-to-end run over three PRs. The first verdict's disable is ignored, the second's edits flake.nix and is refused, and the third's is pushed, opened as PR 8 with its Review, and enqueued; a restart does not repeat it. A second test checks the bound on the known-flaky input.
• test/land.test.ts: disable is accepted only on a requeue, and quest land fix passes the known list to the agent and rejects a malformed one.
• just check and just test pass locally: 938 CLI tests and 474 worker tests.

Suggested follow-ups

• Verify on moq. Once the watcher runs there, confirm the first real disable PR lands through moq's queue and its quest/m0/flaky/ epic passes quest check. The epic's end-to-end run covers the flaky re-enqueue but not the disable.
• Gate parity. The host gate (quest land serve) still requeues flaky failures without recording them. This was left out because gate work is frozen until the gate is retired.

Review

mechanical fix by codex / Codex: Merged base and re·solved documentation/test-fixture conflicts preserving both sides; bun run build, bun run check, bun run test (942 passed, 2 skipped), and node node_modules/vitest/vitest.mjs run test/land-watch.test.ts test/land.test.ts (211 passed) passed. (head d60228f)

Codex gpt-6.1-sol, medium effort, via quest review --base origin/main, head 73b45f1: round 2 reported no findings.

Round 1 (head 9662061) raised two findings, both fixed:

• The known-flaky input was unbounded and could exceed the 100 KB the fixer reads. It is now bounded to 50 KB, most recent first, with each test's last 10 verdicts; a regression test was added.
• A failed disable push was recorded as final. It is now retried on the next pass.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

[coderabbitai]

Important

• Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: d46a13b3-20ca-4094-944c-8ac4b82f690e

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

[kixelated]

Two follow-ups from the /quest-spawn coordinator's evidence check. Both are relevant to this PR's flaky-disable rule, so they belong here:

1. Deterministic failures are recorded as flaky. On main, src/cli/land-watch-fix.ts:473-481 labels every unchanged requeue of a failed build "flaky", and this PR records each one in flaky.json, keyed on the fixer's free-form failure. Observed cases that weren't flaky:

   • moq-dev/quest#324 (EROFS on cgroup.kill): a runner environment gap, fixed in kixelated/nixos#19.
   • moq-dev/quest#324 (missing flock/script): the fixer itself wrote "environment dependency issue, not a flaky test", yet it was requeued as flaky. These were four real tests failing, which a mechanical test.skip would "fix".
   • moq-dev/moq#7560 and moq-dev/moq#7221: "unused import Io" in code already on main was requeued as flaky. The two signatures differ only by backticks, so the same failure didn't even match.

   Under this PR, two such verdicts could auto-disable real tests. Suggested fix: split the requeue result into flaky and not-this-PR (base or environment) and record only flaky. Tell the fixer in assets/land-fix.md that a failure which reproduces deterministically, on the base or from the runner, is not flaky. Optionally, require the check to have passed in a merge group between the two verdicts, and normalise signatures (strip backticks and quoting) before matching.

2. End-to-end verification. The epic's end-to-end condition (quest/m0/queue-watcher/README.md) covers the flaky re-enqueue but not the disable. Add that moq-dev/moq's first disable PR lands through its queue and its new quest/m0/flaky epic passes quest check. moq has one genuine flaky verdict so far (moq-dev/moq#7606).

(Written by Claude Opus 5.5)

[moq-bot]

quest land watch: the merge queue removed this pull request at 73b45f1 because it conflicts with main (merge_conflict).

The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.

(written by quest land watch)

[moq-bot]

quest land watch: armed auto-merge on this pull request at d60228f, so it joins the back of the merge queue once its required checks pass, with the land fixer's fix (codex / Codex): Merged base and re·solved documentation/test-fixture conflicts preserving both sides; bun run build, bun run check, bun run test (942 passed, 2 skipped), and node node_modules/vitest/vitest.mjs run test/land-watch.test.ts test/land.test.ts (211 passed) passed..

(written by quest land watch)