Skip to content

fix(size): trace Nightly growth and identify comparison revisions - #7813

Merged
kixelated merged 7 commits into
mainfrom
quest/m0/nightly/shipping-size
Nov 4, 2026
Merged

kixelated merged 7 commits into
mainfrom
quest/m0/nightly/shipping-size

Conversation

@kixelated

@kixelated kixelated commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The scheduled 2026-11-03 shipping-size job failed nine 5% comparisons. This PR makes the report identify its current and baseline commits, documents how failed reports advance the daily baseline, and records attribution for the remaining acceptance decisions.

Approach

  • Keep the 5% daily and 10% thirty-day guards unchanged. No history is deleted and no artifact is exempted.
  • Include commit links in the summary and failure lines, refusing malformed revision metadata before writing history.
  • Test the full failed-report retention, next-day comparison, and surviving thirty-day guard.
  • Check all 57 open PRs by title, body and changed files before claiming. No duplicate shipping-size implementation. Parent only overlaps Nightly orchestration; bigint merged after the failing Nightly revision and is excluded as a cause.

Measurements

Recorded runs: previous, commit dde50c699f, to failed, commit 801fd9a2f2. Byte counts below are the retained reports and the downloaded shipping archives, not estimates from another platform.

Failing measurement Previous bytes Failed bytes Growth
js/net/raw 444,692 467,348 5.09%
js/net/gzip 136,324 143,266 5.09%
js/watch/raw 587,025 617,504 5.19%
js/watch/gzip 180,712 190,262 5.28%
js/watch/brotli 155,165 162,948 5.02%
moq-relay-aarch64-apple-darwin archive 9,467,927 10,201,659 7.75%
moq-relay-aarch64-unknown-linux-gnu archive 9,686,833 10,408,881 7.45%
moq-relay-x86_64-pc-windows-msvc archive 9,928,808 10,907,583 9.86%
moq-relay-x86_64-unknown-linux-gnu archive 10,301,723 11,302,331 9.71%

JavaScript: exact attribution

Rebuilt both endpoints with the locked dependencies and the original build/bundle scripts, then every one of the 18 intervening commits touching js/, package.json, or bun.lock. All 21 endpoint measurements (including unrelated and lazy bundles) reproduce exactly. Intermediate steps build only the measured entries and their workspace dependencies, using the unchanged package build scripts and sh/size/bundles.ts; the final step reproduces all 21 failed-run values exactly too.

Byte changes below are per commit, relative to the preceding measured point. 6 intervening commits have zero change in all five failing JS measurements and are omitted. Compressed-size deltas include changes in generated bundle contents and chunk names.

Commit net raw net gzip watch raw watch gzip watch Brotli
62ccb69bac perf(net): insert replayed JS groups in order instead of re-sorting +61 +16 +61 +7 +18
103e98e362 fix(net): lite-05/06 close waits for the subscriber's FIN; drop the legacy close quest -120 +14 -120 -9 -77
5e327b55f8 fix(watch): cache audio support across unmute +0 +0 +76 -132 -43
e3e1cc8841 feat(net)!: routes carry an instance and a per-instance epoch +2,780 +671 +3,300 +782 +640
20955c09e9 fix(net): wait for subscription-owed groups in JS ordered reads +1,184 +276 +1,184 +267 +433
2738588cfc feat(js)!: the browser publisher and player report stats and feedback +0 +0 +6,787 +2,524 +1,925
555390791f chore(deps): remove redundant build, schema, and codec helpers +0 +0 +0 +8 +19
5ab878666c feat(net)!: subscriptions take a track Delay { min, max } +1,056 +338 +1,127 +378 +201
945aec5a53 refactor(net)!: return Following from origin follow +208 +23 +191 +31 -40
1777a2d6f9 feat(net)!: support JavaScript subscription ranges +17,090 +5,431 +17,111 +5,390 +4,543
1661c554cb perf(hang): keep a bulk max-delay skip linear in buffered groups +0 +0 +365 +134 +173
d98b0ce53b perf(net): sample the send-rate estimate only while a reservation reads it +397 +173 +397 +170 -9
Total +22,656 +6,942 +30,479 +9,550 +7,783

The largest cause is subscription ranges: +17,090 net raw bytes and +17,111 watch raw bytes. The endpoint metafile attributes +9,018 raw bytes to window.js, +4,874 to track.js, and +2,767 to the net bundle's lite subscriber. Watch also adds +6,787 raw bytes for stats and feedback. Routing instances, delay bounds, ordered-read correctness and idle stats sampling explain the rest as measured above.

Accepted (maintainer, 2026-11-04; see Decisions): the JS step is the measured cost of these retained features and correctness fixes. Reverting range semantics, stats, or route identity to meet a size threshold would undo their consumers. No avoidable added dependency or duplicated copy of these implementations appeared in the metafile. The per-commit table above is the acceptance record the size README asks for; the retained November 3 point is already the next daily baseline, so no tooling change, exemption, or history edit implements it.

Relay: shipping payloads and native attribution

All downloaded binaries have larger code sections. Original executable bytes and code-section sizes:

Target Executable before Executable after Code before Code after
aarch64 macOS 20,261,208 22,738,664 15,420,356 17,551,648
aarch64 Linux 20,188,592 22,638,384 15,069,164 17,168,364
x86_64 Windows MSVC 25,871,872 29,446,656 20,171,230 23,286,510
x86_64 Linux 23,452,144 26,963,848 18,497,306 21,548,314

Code is ELF/PE .text, or Mach-O __TEXT,__text. The failed binaries on all four targets contain rind-0.1.2 source paths absent from all previous binaries, including Windows paths normalized from backslashes. This is corroboration of new linked code, not a per-platform commit-cost measurement.

The x86_64 Linux profiling builds use the same Rust 1.98.1 as the Linux release artifact. They preserve symbols for cargo bloat --locked --profile profiling -p moq-relay --bin moq-relay --crates -n 0 --message-format json; their sizes are diagnostic and do not replace the shipping archives. Across the recorded endpoints, .text grows 3,054,848 bytes. Attribution estimates add 2,357,825 bytes of rind, 415,911 of rind_rustls, 204,169 of moq_net, and 118,853 of std. AWS-LC remains at 1,363,342 attributed bytes because transport still uses it. Bloat attributes optimized symbols approximately; those crate counts are not an exact decomposition of compressed bytes.

Paired x86_64 Linux profiling builds isolate the auth migration itself:

Revision Diagnostic .text bytes Change from preceding point
dde50c699f, previous Nightly 18,440,889 baseline
53553b8195, migration's parent 18,649,337 +208,448
855dc617ab, auth on rind 21,478,905 +2,829,568
801fd9a2f2, failed Nightly 21,495,737 +16,832

The single rind-auth commit accounts for 92.6% of the endpoint diagnostic code growth. moq_net attribution is identical across that pair, while rind and rind-rustls appear. Earlier commits collectively add 208,448 code bytes and later ones another 16,832; they are not individually bisected here. These figures establish the dominant Linux cause, not exact compressed-archive deltas or a full native per-commit decomposition.

The rind auth migration deliberately moves standalone moq-auth away from C crypto; a relay still carries AWS-LC for transport, so the intermediate migration ships both. Switching the relay's transport or deleting its supported auth cryptography is a separate API/behavior decision, not a mechanical size fix. The existing transport quest owns moving that stack. Native per-commit macOS, aarch64 Linux, and Windows archive attribution has not been measured here. Both reachable NixOS hosts are x86_64. A macOS measurement needs the pinned Nix toolchain rather than kixtop's ambient Rust 1.97.0, and no listed host runs Windows MSVC. Those historical builds remain work, not claimed results. Recommendation: retain the intended rind auth migration, but keep relay growth acceptance open until the missing native attribution is supplied or the maintainer explicitly accepts this evidence limit. The four actual archive deltas are measured; treating the Linux cause as an exact byte attribution on the other targets would overstate the evidence.

Baseline behavior

report.py writes history before exiting 1 for threshold failures; Nightly uploads it with if: !cancelled(). The failed run's history actually contains the November 3 point. Therefore the next scheduled run compares daily against that failed point, not the last successful check. The retained history currently starts October 22, so this failure had no thirty-day anchor. Tests demonstrate that an older thirty-day anchor continues to fail after daily advancement, and that a step over 10% clears that guard only when a post-jump point becomes the thirty-day anchor. Recording acceptance does not waive this guard; a large step can keep Nightly red for thirty days. Invalid/missing-artifact reports still write no history. Routine acceptance records the cause and reason in a PR and keeps the retained series; deleting every history artifact is a destructive full reset, not needed here.

Impact

  • Public library API: none.
  • Wire: none.
  • Internal size report text now names the measured and regressed baseline revisions. History JSON and exit status semantics are unchanged.

Alternatives

  • Raising the threshold or resetting history: rejected, since either hides growth without attributing it.
  • Reverting approved JS features or choosing crypto providers based on the size check: left to the maintainer as behavior decisions, not silently done here.
  • Add an acceptance manifest or another baseline workflow: dropped. Daily advancement already exists; documenting and testing it is simpler. Thirty-day growth is still reported.

Follow-ups

  • Existing rind transport owns eliminating the remaining AWS-LC transport stack, and release size audit owns the release-time view. No new follow-up quest is created by this sub-agent.

Decisions

Decided 2026-11-04 by the maintainer:

  • JS growth (js/net and js/watch, +5.02–5.28%):
    • ✅ Accept the exactly attributed feature and correctness cost, recorded by the per-commit table above.
    • Require further reduction while retaining those behaviors.
  • Guards:
    • ✅ Keep the 5% daily and 10% thirty-day guards unchanged.
    • Raise or exempt the guards for this step.

This PR lands as a checkpoint. Relay archive acceptance stays open until native per-commit attribution for macOS, aarch64 Linux, and Windows MSVC is supplied (or the maintainer explicitly accepts the evidence limit), and the next scheduled Nightly shipping-size job on main still needs inspection. Both stay in the quest's ## Waiting, so the quest survives this merge.

Validation

  • After merging origin/main and recording the decisions: nix develop --command quest gate just ci origin/main passed every selected module (Biome, Markdown, gh tooling tests, quest, Python, shell, large files, text, TOML, Nix, justfile).
  • python3 sh/size/report.test.py: 13 tests pass; nightly.test.py and release.test.py pass. The thirty-day boundary test checks the large step on day N, N+1, N+29 and N+30 with fixed dates, without wall-clock waits.
  • Replayed the real failed sizes against the real prior history with the updated reporter: exit 1 and the same nine failures, now with the measured and baseline commit links. No failure was suppressed.
  • Historical JS builds and native profiling builds ran through quest gate. Both JS endpoints and the final intermediate point match all 21 retained measurements exactly.
  • quest check --base origin/main: 601 documents pass.
  • No full Nightly was dispatched. No scheduled main pass is claimed. No hosted checks or reviews are being monitored.

Review

Codex / GPT-6.1 Sol (gpt-6.1-sol), medium effort, quest review --base origin/main --new, session 01a1a550-8a87-7dd2-98d5-dc7f292418ba, round 1. Reviewed head 6ece0c5ed5 against 17a81d6c64: no findings.

Earlier rounds (Claude Code / Claude Opus 5.5, medium, head b28b9a5987) fixed the ambiguity about an accepted large step clearing Nightly after one failure, then reported no findings. The new head adds only the origin/main merge and the recorded decisions in the quest.

Investigation by GPT-6.1 Sol; recorded decisions, origin/main merge, and review update by Claude Opus 5.5.

(Written by Claude Opus 5.5)

kixelated and others added 5 commits November 3, 2026 21:01
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
Co-Authored-By: GPT-6.1 Sol <noreply@openai.com>
@kixelated kixelated changed the title Nightly's shipping-size check passes fix(size): trace Nightly growth and identify comparison revisions Nov 4, 2026
kixelated and others added 2 commits November 3, 2026 22:45
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The maintainer accepted the exactly attributed js/net and js/watch growth
and kept the 5% daily and 10% thirty-day guards unchanged. Relay archive
acceptance and the scheduled Nightly check remain in the quest's Waiting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 4, 2026 06:49
@coderabbitai

coderabbitai Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a3fdf593-f503-477c-87b6-46df5c8ec5e7
Commits

Reviewing files that changed from the base of the PR and between 17a81d6 and 6ece0c5.

Files selected for processing: 4
  • quest/m0/nightly/shipping-size.md
  • sh/size/README.md
  • sh/size/report.py
  • sh/size/report.test.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Reporter revision labels: source(row) names the measured commit in the summary and appends the baseline's date and commit to each failure line, falling back to the bare date for older reports without revision.
  • Revision validation: compare() now rejects a present revision that is not a full 40-character lowercase SHA, before history is written. Missing revisions stay accepted, so retained history from before the field was added still loads.
  • Tests: covers the large-step monthly lifecycle, invalid revision metadata, failed-report retention followed by a passing next day with the 30-day guard still firing, and baseline revision text.
  • Docs and quest: README explains daily baseline advancement, the surviving 30-day guard, and how to accept a measured step without resetting history; the quest records the JS acceptance and leaves relay native attribution plus the next scheduled Nightly inspection in ## Waiting.

I replayed the real failed run against the previous run's retained history with the new reporter: exit 1, exactly the nine documented failures, each naming 2026-11-02 [dde50c699f](https://github.com/moq-dev/moq/commit/dde50c699f69507ced243754c2f700a6184d1576), and the summary naming 2026-11-03 [801fd9a2f2](https://github.com/moq-dev/moq/commit/801fd9a2f2f5c88371fa833d805620d97d372465). The real history artifact starts on 2026-10-22 (so no thirty-day anchor), includes the 2026-11-03 point, and every retained row carries a full SHA, so the new validation cannot break the next load. All three size test files pass.

Pullfrog  | View workflow run | Using deepseek-v4.1-flash-1017 (free via Pullfrog for OSS) | 𝕏

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 6ece0c5. No actionable issue found in revision validation, comparison links, or integration with the size-history collector. Older reports without revision metadata remain supported.

Verification: GitHub-only static inspection of the reporter, tests, collector and Nightly integration. I did not run tests or replay stored artifacts. Native relay attribution on the other platforms and the next scheduled Nightly remain open as noted in the PR.

(Written by review (OpenAI))

@kixelated
kixelated added this pull request to the merge queue Nov 4, 2026
Merged via the queue into main with commit 5790218 Nov 4, 2026
11 checks passed
@kixelated
kixelated deleted the quest/m0/nightly/shipping-size branch November 4, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant