Skip to content

feat(bench): measure Apple M4 natively in the bench report - #435

Merged
kixelated merged 5 commits into
mainfrom
quest/m1/bench-apple-m4
Nov 7, 2026
Merged

kixelated merged 5 commits into
mainfrom
quest/m1/bench-apple-m4

Conversation

@kixelated

@kixelated kixelated commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Works on quest/m1/bench-apple-m4.md: adds kixtop (Apple M4, macOS) to BENCHMARKS.md, and rewrites the quest to what remains, a quiet desktop re-collect and kixtop's build cost, both waiting on the maintainer's hosts going quiet.

Changes

  • benchmarks/data/kixtop.json: the second of two native kixtop collections, at 93f91f48, which measured 0.68 other busy CPUs during the run (not noisy). The first, at cc2f73f6, ran under 4.36 other busy CPUs and most rows came out 30-40% slower, so it is not used. fix(bench-report): hold each host with quest gate --host, and measure macOS noise #399's macOS path worked unchanged; the ssh host key was already accepted.
  • benchmarks/gaps.toml: two kixtop entries, both reasons:
    • 19 short-message and packet-size rows (AES-GCM 32 B and 1350 B, ChaCha20-Poly1305 32 B, all three header protections, SHA-256/512, HMAC, HKDF), 1.11x to 12.6x: DIT. rind toggles it around every call that handles a secret, about 35-55 ns a call on the M4, and no other library sets it. A probe build on kixtop at the same commit with DIT denied (--cfg rind_cpu_disable, RIND_CPU_DISABLE_dit) put every one within 10% of the fastest library except AES header protection (1.2x, 0.6 ns) and HMAC-SHA-256 64 B (1.15x RustCrypto). Per-call DIT is the maintainer's decision from #387's findings, and #431's rind::cpu::Dit::hold() pays it once per batch.
    • xchacha20poly1305/32B 1.26x: noise. The probe's rerun and the first run both measured 1.04x.
  • admin/bench-report: a kixtop host note (in daily use, has DIT), and "in the Nix dev shell on Linux" becomes "in the Nix dev shell", since macOS uses it too since #399.
  • README.md: the gaps example adds AES-GCM and short messages on Apple silicon, so "matches or beats both" stays true with kixtop in the headline.
  • quest/m1/bench-apple-m4.md: rewritten to the remaining work, with a Waiting item for the maintainer. quest mv refuses on a claimed quest, so it keeps its path. quest/m3/bench-report-refresh.md now points here instead of at the quest.

Verified table

Rechecked against kixtop's features (aes, dit, neon, pmull, sha2, sha3); no change needed. With sha3, rind runs the SHA-512 instructions, so the !sha3 SHA-512 and Ed25519 entries correctly give no ✓. X25519, P-256 ECDH and ECDSA, and ChaCha20 header protection run the _alt routines and the same Rust their aarch64 entries cover, so they keep theirs.

Why desktop isn't here

collect refused desktop three times on 2026-11-07 (16.8, 4.7, and 5.3 of 24 CPUs busy, limit 2.4), all from other sessions' ungated builds. One attempt's host hold waited an hour: the global quest on desktop predates moq-dev/quest#526 and has no host-hold logic, so its gated commands were admitted past the hold. Both kixtop runs' build costs were refused the same way (11.6 and 6.6 of 10 busy), so its build-cost table reads "Not measured".

Choices made without asking

  • Reported kixtop's quiet second run rather than averaging in the noisy first.
  • Recorded the DIT rows as a reason, not a quest, since keeping DIT per call is already settled.
  • Landed kixtop now and moved desktop to the quest instead of holding the PR open for a quiet window that may not come soon.

Impact

No public API or wire change. Report data, its renderer's host note, README wording, and quests only.

Validation

  • nix develop --command just check origin/main: pass on the pushed head.
  • admin/bench-report render --check: pass.
  • quest check --base origin/main: 70 documents ok.

Follow-ups

  • Time rind under rind::cpu::Dit::hold() on DIT cores in the report or nightly bench. feat(cpu): hold DIT across a batch of calls, putting M1 AES-GCM ahead of aws-lc-rs #431 skipped a held case because no report host had DIT; kixtop does, and MoQ will hold the guard, so the report shows a cost MoQ won't pay. Recommend: plan it.
  • aarch64 HMAC-SHA-256 at 64 B, 1.15x RustCrypto with DIT denied. Recommend: drop; it is about 15 ns, and a handshake runs only a few HMACs against 130 µs or more.
  • AES header protection on aarch64, 1.2x aws-lc-rs with DIT denied. Recommend: drop; 0.6 ns a packet.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, high effort

Review

Rewritten by quest review after each round.

  • Reviewer: Codex, gpt-6.1-sol, medium effort
  • Session: 01a1b43d-ccb7-7090-b831-b32b0ebf82f3, round 1
  • Head: b44c0c6
  • Base: origin/main at e7570f6
  • Score: 10/10, The specified diff adds no cryptographic paths, the report and charts pass freshness validation, all eight rendering tests pass, and the missing M4 build costs are explicitly disclosed.
  • Findings: none

kixelated and others added 4 commits November 6, 2026 17:56
Quest-Claim: 5c6ac16208b97925
Adds kixtop (Apple M4, macOS) to BENCHMARKS.md from its quiet run at
93f91f4, and records why rind trails on short messages there: the M4
has DIT, which rind toggles around every call that handles a secret.
A probe with DIT denied puts all but two of those rows within 10%.

Rewrites the quest to the quiet desktop re-collect and kixtop's build
cost, which both wait on the maintainer's hosts going quiet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title The bench report measures Apple M4 and a quiet Arrow Lake feat(bench): measure Apple M4 natively in the bench report Nov 7, 2026
@kixelated
kixelated marked this pull request as ready for review November 7, 2026 04:23
@pullfrog

pullfrog Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Your OpenCode Go account is out of credit. Top it up, then re-trigger Pullfrog.

OpenCode Go's response
provider error: Insufficient account funds

Top up OpenCode Go →

Pullfrog  | Rerun failed job ➔ | View workflow run | via Pullfrog | Using glm-5.3

@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 5 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 64155bf6-ec1b-4020-9455-b35e4a12c796
Commits

Reviewing files that changed from the base of the PR and between e7570f6 and 58faaf5.

Files ignored due to path filters: 6

  • benchmarks/headline-dark.svg is excluded by !**/*.svg
  • benchmarks/headline.svg is excluded by !**/*.svg
  • benchmarks/kixtop-dark.svg is excluded by !**/*.svg
  • benchmarks/kixtop.svg is excluded by !**/*.svg
  • benchmarks/overview-dark.svg is excluded by !**/*.svg
  • benchmarks/overview.svg is excluded by !**/*.svg
Files selected for processing: 9

  • BENCHMARKS.md
  • README.md
  • admin/bench-report
  • benchmarks/data/kixtop.json
  • benchmarks/gaps.toml
  • quest/m1/README.md
  • quest/m1/bench-apple-m4.md
  • quest/m1/bench-dit-held.md
  • quest/m3/bench-report-refresh.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head b44c0c6 against base e7570f6.

No actionable findings in the reviewed report/data changes. The M4 collection identifies commit 93f91f4, records 0.68 other busy CPUs against a limit of 1, and explicitly records the failed build-cost collection rather than inventing a measurement. Its 357 result IDs are unique and each estimate lies within its recorded interval. Recomputing the >10% comparisons from those results gives exactly the 19 DIT-listed operations plus the XChaCha20 32 B row, including the reported 12.62x maximum. The M4's sha3 feature excludes the !sha3 SHA-512/Ed25519 verification entries. The remaining quiet Arrow Lake collection and M4 build costs stay open in the quest.

Direction: the README now acknowledges Apple's short-message gaps. A separate held-DIT measurement remains useful for batch consumers, but keep it distinct from this per-call baseline and from the DIT-disabled diagnostic probe. Preserving the source commit and load qualification is important when newer kernel or dispatch changes land.

Limits: GitHub-only review of the report, raw JSON, gap entries, small renderer changes, verification predicates and quest updates. I did not rerun collection, the DIT-disabled probe, renderer freshness tests, CI, or proofs, and did not visually render the SVGs. The noise and causal explanations rely on the additional runs described in the PR, not independently reproduced measurements. This COMMENT is not merge approval.

…d rows

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated enabled auto-merge November 7, 2026 04:25
@kixelated
kixelated added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 10d344f Nov 7, 2026
5 checks passed
@moq-bot moq-bot Bot mentioned this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant