Skip to content

quest: the queue concurrency trial ran out of disk - #7970

Merged
kixelated merged 1 commit into
mainfrom
quest-plan/queue-trial-disk
Nov 5, 2026
Merged

kixelated merged 1 commit into
mainfrom
quest-plan/queue-trial-disk

Conversation

@kixelated

@kixelated kixelated commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Records the end of the merge-queue concurrency trial in quest/m0/ci/queue-concurrency.md.

  • What happened: at 17:12Z the second queue runner, nixos-moq-ci2, failed a merge-group build with No space left on device. That dropped fix(watch,publish): keep audio across declared ends; repair browser media tests #7716.
  • Cause: every CI runner on nixos shares one 667 GiB XFS project quota, and each moq queue runner keeps its own Cargo targets and mbx caches inside it.
  • Decision: the maintainer ended the trial early on 2026-11-05. Ruleset 2420853 went back to max_entries_to_build: 1 at 18:02Z, and the scheduled revert was cancelled.
  • Before a retry: size the per-runner caches or the quota in kixelated/nixos for two concurrent builds.

Public API: none. Wire: none.

Review

d1debdec5f skipped as trivial: a quest-only edit that records the trial's outcome, written by Claude Opus 5.5. quest check passes.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 22 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d00d285a-fa55-4d37-a43e-71a6f5731313
Commits

Reviewing files that changed from the base of the PR and between 67f56cb and d1debde.

Files selected for processing: 1

  • quest/m0/ci/queue-concurrency.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated
kixelated added this pull request to the merge queue Nov 5, 2026

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit d1debde. No actionable findings. Recording the aborted experiment without claiming a performance winner and requiring warm-cache capacity before another trial is a sound direction. Minor chronology: the linked run started at 17:12Z; its first logged no-space error was at 17:16:35Z, so quest/m0/ci/queue-concurrency.md:42–43 could distinguish those times.

Verification: GitHub-only static review of the plan, linked job logs and configuration. I did not run tests or independently verify the host storage snapshot.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — two accuracy points inline, neither blocks.

Reviewed changes

This run reviewed the single quest-only commit that records how the merge-queue concurrency trial ended.

  • Trial ended early — the plan text now says the maintainer stopped the max_entries_to_build: 2 window after ~2.2 hours, records the ruleset returning to 1 at 18:02:15Z, and cancels the scheduled 2026-11-07 revert.
  • Failure recorded — nixos-moq-ci2 ran out of space during a merge-group build for #7716, attributed to the shared 667 GiB XFS project quota plus each queue runner's own Cargo targets and mbx caches.
  • Retry prerequisite — sizing the per-runner caches or the quota in kixelated/nixos is now a stated condition before another two-build attempt, with the trial window declared too short and disturbed to compare.

The core facts check out: run 42072368255 is #7716's merge group on nixos-moq-ci2, and #7716 re-merged at 18:02:23Z. The quest stays open, and no other quest file depends on the trial's state.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏


It ran out of disk the same day, so the maintainer ended it early
(decided 2026-11-05, maintainer, /quest-complete; ruleset back to 1 at
2026-11-05T18:02:15Z, scheduled revert cancelled). At 17:12Z

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The merge-group build did not fail at 17:12Z: the job started 17:12:49Z, hit error: couldn't create a temp dir: No space left on device at 17:16:35Z, and concluded 17:16:42Z. Since this record uses second-level timestamps everywhere else, 17:12Z reads as the failure time and will mislead anyone reconstructing the incident.

Comment on lines +48 to +49
hard; 578 GiB used afterwards), and each moq queue runner keeps its own
Cargo targets and mbx caches inside it (`modules/ci-runners.nix`: MBX target

@pullfrog pullfrog Bot Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says each queue runner keeps its own mbx caches, while the Method paragraph above (line 31-33) still warns that "the shared mbx cache" could erase the gain and tells the retry to watch for it. In this failing job mbx state lived in /var/cache/github-runner-home/.cache/mbx on moq-ci2, and .github/actions/rust-cache/action.yml notes runners on one host may share a home but never a workspace, so the per-runner wording here looks right and the shared-cache premise is stale. Mark that earlier sentence superseded (or say which holds) so the record and the retry guidance agree.

Merged via the queue into main with commit 1c140ac Nov 5, 2026
6 of 9 checks passed
@kixelated
kixelated deleted the quest-plan/queue-trial-disk branch November 5, 2026 18:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant