Skip to content

ci: publish @moq/libav and its sources to nix.moq.dev - #7902

Merged
kixelated merged 5 commits into
mainfrom
quest/m0/ci/libav-sources
Nov 4, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m0/ci/libav-sources

Conversation

@kixelated

@kixelated kixelated commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

Check failed twice on 2026-11-03/04 because ffmpeg.org was unreachable: moq-dev/moq#7685 and moq-dev/moq#7760 (run 41777407147: "cannot download ffmpeg-9.0.tar.xz from any mirror").

Root cause: moq-libav was never on nix.moq.dev. The root postinstall (js/libav/build.ts) runs nix build .#moq-libav outside the dev shell, and sh/gh/cache.sh pushes only the dev shell's closure. So every hosted Check compiled ffmpeg to WebAssembly from source and downloaded ffmpeg-9.0.tar.xz, opus-1.6.1.tar.gz, emfiberthreads-1.3.tar.gz, the libav.js tag from GitHub, and terser's npm tarballs. On 2026-11-04, before this change, https://nix.moq.dev/<hash>.narinfo returned 404 for both moq-libav and its wasm derivation on x86 Linux, ARM Linux and ARM macOS. Run 41777407147's log shows it building ffmpeg-9.0.tar.xz.drv, libav.js-opus-aac-af-6.10.9.0.drv and moq-libav-0.1.0.drv.

Approach

  • js/libav/package.nix adds a sources passthru: a linkFarm over every download the wasm build makes (three tarballs, the libav.js source, and the importNpmLock deps). Its closure is about 36 MB.
  • sh/gh/cache.sh builds .#moq-libav, .#moq-libav.wasm and .#moq-libav.sources after it pushes the dev shell, then pushes all three to nix-ci-moq-dev on every Cache matrix leg.
    • Pushing the built package lets Check substitute it instead of compiling it.
    • moq-libav copies the wasm build's files instead of referencing them, so its closure is just its own path. Pushing wasm separately means a change to the package alone (a version bump, the README, index.d.ts) still substitutes the ffmpeg compile.
    • Pushing the sources covers a PR that changes the build, and the window after a nixpkgs bump before Cache republishes: fixed-output paths don't depend on nixpkgs, so they still substitute.
  • The comments in infra/README.md, .github/justfile and cache.yml now mention it.
  • Deletes the quest.

Impact

  • No public API or wire changes. The moq-libav derivation and its output are unchanged; only its passthru grows.
  • Cache now builds moq-libav on each leg whenever its derivation changes. Once that's done, hosted Check skips the ffmpeg wasm compile entirely.

Choices made without asking

  • I push from the existing dev-shell matrix instead of a new job, so every platform that publishes a dev shell also publishes moq-libav. Cache runs on main pushes only, and the dev shell is pushed first, so a libav failure still fails the job without blocking the dev shell.
  • I publish the sources as one linkFarm passthru instead of listing each fetcher in cache.sh, so a new download in package.nix is covered automatically.
  • I delete the quest in this PR. Its "Done when" can only be observed after merge: the first main Cache run pushes the paths, and later Checks substitute them. Check afterwards with curl -sI https://nix.moq.dev/$(nix eval --raw .#packages.aarch64-linux.moq-libav.outPath | cut -d/ -f4 | cut -d- -f1).narinfo (expect 200).

Validation

  • nix build .#moq-libav .#moq-libav.wasm .#moq-libav.sources built all three on x86 Linux. The out-links are libav (the package), libav-1 (wasm) and libav-2 (sources), and cache.sh pushes them all with "$roots"/libav*. nix path-info -r on the package lists only itself, which confirms that wasm needs its own push. The sources closure is 24 paths, about 36 MB: the three tarballs, the libav.js source, and the npm tarballs.
  • shellcheck sh/gh/cache.sh, just gh check (actionlint, zizmor, test-inventory), quest check --base origin/main: pass. cache.sh has no tests of its own.
  • quest gate nix develop --command just ci origin/main: biome, js, markdown, gh, workers, quest, shell, and the text/toml/nix/justfile lints all pass. check flake fails building the runner image: unshare: write failed /proc/self/uid_map: Operation not permitted. This is pre-existing. checks.x86_64-linux.github-runner-vm-image evaluates to the same derivation (zp08za9k...-vm-test-run-github-runner-vm-image.drv) on this branch and on its base f37d964, and quest/m2/audit-ci.md already records it as reproducing on main.
  • I couldn't test the actual push here: Cache runs only on main, with the cache-write Environment.

Overlap

moq-dev/moq#7662 (draft) adds a separate runtime job to cache.yml and its own sh/gh/cache-runtime.sh. The work doesn't overlap, but both PRs edit the same infra/README.md paragraph, so whichever lands second needs a simple text merge.

Follow-ups

None.

Review

Codex, gpt-6.1-sol, medium effort, reviewed head 9cc461c8978095bedce76fe6620f17dbac16f87d against origin/main (round 2). It reported no findings. Round 1, on 671bf39c37, also reported none.

pullfrog's inline finding (the wasm build wasn't published) was valid. It is fixed in 9cc461c and the thread is resolved.

(Written by Claude Opus 5.5)

kixelated and others added 3 commits November 4, 2026 10:01
Check's `bun install` builds `moq-libav` outside the dev shell, so the
Cache workflow never pushed it and every hosted Check compiled it from
source, fetching ffmpeg, opus and emfiberthreads from their upstream
mirrors. Push the built package and a link farm of everything its build
downloads alongside the dev shell.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title CI never downloads libav's sources from upstream mirrors ci: publish @moq/libav and its sources to nix.moq.dev Nov 4, 2026
@kixelated
kixelated marked this pull request as ready for review November 4, 2026 18:14
@coderabbitai

coderabbitai Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 24 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3a258c37-05b4-4a65-be76-3992823971c7
Commits

Reviewing files that changed from the base of the PR and between 735cc0c and 9cc461c.

Files selected for processing: 7
  • .github/justfile
  • .github/workflows/cache.yml
  • infra/README.md
  • js/libav/package.nix
  • quest/m0/ci/README.md
  • quest/m0/ci/libav-sources.md
  • sh/gh/cache.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 671bf39

No actionable code findings. The source farm retains the three upstream tarballs, libav.js source and npm dependencies, and the cache script pushes both the package and source closures after the dev shell. The existing main-only cache-write workflow remains in place.

Overall direction: the implementation addresses the missing cache roots. Deployment verification remains outstanding: a successful main Cache run and a subsequent clean Check substituting these paths are needed to establish the deleted quest's no-upstream-fetch outcome; a narinfo HTTP 200 alone does not establish that outcome.

Verification limits: GitHub source review only, including the pinned Nix helper implementations. I did not run Nix builds, upload to the cache, or exercise substitution on any platform.

(Written by review (OpenAI))

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one minor suggestion inline.

Reviewed changes

  • js/libav/package.nix — adds a sources passthru linkFarm over the three fetched tarballs, the libav.js source, and the importNpmLock deps.
  • sh/gh/cache.sh — after the dev-shell push, builds moq-libav and moq-libav.sources and pushes both out-links to nix-ci-moq-dev.
  • cache.yml / .github/justfile / infra/README.md — renames the step and documents that Cache now publishes moq-libav and its sources.
  • quest/m0/ci/ — deletes the completed libav-sources quest and its index bullet.

The mechanism checks out: Check runs on aarch64-linux (ubuntu-24.04-arm), Cache has a matching leg, wasm.src is forced at evaluation by importNpmLock, and the fixed-output paths are nixpkgs-independent, so substituting them works across a flake.lock bump.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

Comment thread sh/gh/cache.sh Resolved
kixelated and others added 2 commits November 4, 2026 10:34
moq-libav copies the wasm build's files instead of referencing them, so
pushing its closure left the wasm derivation behind. A change to the
package alone (version bump, README, index.d.ts) would then recompile
ffmpeg in Check. Build and push .#moq-libav.wasm alongside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed commit: 9cc461c

No new actionable findings. The update addresses the missing wasm cache root: sh/gh/cache.sh now builds the package, wasm output and source farm, then passes all three out-links to the existing closure-upload recipe. Because the package copies rather than references the wasm files, publishing that output separately preserves reuse across package-only changes. The glob is confined to the fresh temporary directory and excludes the dev-shell links.

Overall direction: the targeted fix closes the cache-root omission. The earlier deployment-verification limit remains: confirm a successful main Cache run and clean Check substitution, including a package-only rebuild, before treating the operational outcome as proven.

Verification limits: GitHub-only source/delta review; no Nix build, cache upload or substitution was run.

(Written by review (OpenAI))

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • sh/gh/cache.sh — builds .#moq-libav.wasm alongside the package and sources, then pushes all three out-links with the "$roots"/libav* glob. This closes the prior finding: the package copies ${wasm}/*, so its closure never references wasm, and a package-only change now still substitutes the ffmpeg compile.
  • infra/README.md — documents that Cache publishes moq-libav, its wasm build, and its downloaded sources.

The out-link numbering (libav, libav-1, libav-2) matches the corrected push, and a fresh mktemp -d per run means the glob can't pick up stale links. shellcheck and bash -n pass. The earlier inline thread is resolved.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit dfbcdc4 Nov 4, 2026
9 of 10 checks passed
@kixelated
kixelated deleted the quest/m0/ci/libav-sources branch November 4, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant