Skip to content

chore: benchmark the downgraders with CodSpeed - #47

Merged
dinwwwh merged 2 commits into
mainfrom
claude/focused-rubin-sc626c
Oct 2, 2026
Merged

dinwwwh merged 2 commits into
mainfrom
claude/focused-rubin-sc626c

Conversation

@dinwwwh

@dinwwwh dinwwwh commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

Mirrors the setup in middleapi/standard-server: pnpm bench runs Vitest benchmarks through @codspeed/vitest-plugin, and a CodSpeed workflow runs them in simulation mode on pushes to main and on pull requests, so a performance change shows up on the pull request that causes it.

Benches

packages/downgrader/benches covers the spec and schema converters of each step, plus the chained 3.2 → 3.1 → 3.0 path, on:

  • The official corpus the tests already validate against.
  • A generated CRUD API at 10 and 100 resources that uses everything each step rewrites: path items reused from components.pathItems, $defs, an $id resource, webhooks, mutual TLS, multipart bodies, and in 3.2 query operations, itemSchema, components.mediaTypes, Tag and Server fields, and dataValue examples. It builds a fresh tree with no shared objects, like a document parsed from JSON.
  • Worst-case reference graphs: shared diamonds, cyclic callback graphs, and a 200-hop path item $ref chain. These are the inputs that earlier fixes (fix(downgrader): convert shared schemas once in dereferenced documents #23, fix(downgrader): convert each Path Item hop once when merging $ref chains #33) made linear, so a regression there costs orders of magnitude.
  • Standalone schemas: one nullable property (the per-call overhead), an order schema with $defs and a recursive tree, and a 64-level dereferenced diamond.

The generated API, the diamonds, and the chain pass the official schemas both before and after conversion. The callback graphs are checked on output only, as the existing tests do.

Other changes

  • vitest.config.ts: adds the CodSpeed plugin, which only switches on in benchmark mode, and the benchmark include/exclude globs.
  • Package tsconfig.jsons exclude **/*.bench.* from the build, like their tests. The root tsc still type-checks the benches.
  • READMEs: CodSpeed badge in all three, pnpm bench in the root Development section, and a short Performance section in the downgrader README.
  • pnpm-lock.yaml: the CodSpeed plugin only supports vite up to 7, so vitest now resolves vite 7.3.6 instead of 8.2.2, as in standard-server.

After merging

CodSpeed already picked up all 19 benchmarks on this branch. Pull requests will show performance changes once the workflow has run on main, which happens when this merges.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LjCihASFt61Eqjq9jfAoE5

claude added 2 commits October 2, 2026 13:07
Mirror middleapi/standard-server: `pnpm bench` runs Vitest benchmarks
through the CodSpeed plugin, and a CodSpeed workflow runs them in
simulation mode on pushes to main and on pull requests, so a
performance change shows up on the pull request that causes it.

The benches in packages/downgrader/benches cover both spec and schema
converters of each step, plus the chained 3.2 to 3.0 path, on:
- the official corpus the tests validate against
- a generated CRUD API at 10 and 100 resources that uses every feature
  each step rewrites (removed parts to inline, $defs, $id, webhooks,
  query operations, item schemas, ...)
- the shared and cyclic reference graphs that once made conversion
  exponential, so a regression there costs orders of magnitude

READMEs gain the CodSpeed badge, the bench command, and a short
performance section for the downgrader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LjCihASFt61Eqjq9jfAoE5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LjCihASFt61Eqjq9jfAoE5
@pullfrog

pullfrog Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

your Pullfrog Router balance is empty, and this repo has no provider key to fall back on, so the agent never ran.

To fix, any one of: add a payment method or top up your Router balance · add a provider API key (GitHub Actions secret or Pullfrog secret) · switch this repo to a free model.

Top up Router → · Model settings → · Setup docs → · Ask in Discord →

Pullfrog  | Rerun failed job ➔ | View workflow run | via Pullfrog | 𝕏

@dinwwwh dinwwwh changed the title Add comprehensive benchmarks for OpenAPI spec downgrades chore: benchmark the downgraders with CodSpeed Oct 2, 2026
@codecov

codecov Bot commented Oct 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

dinwwwh commented Oct 2, 2026

Copy link
Copy Markdown
Member Author

The failing pullfrog check isn't caused by this PR. Pullfrog reports that its Router balance is empty and there's no provider key, so the agent never ran. It fails the same way on #48, which doesn't share any changes with this one. No code change can fix it: it needs a top-up, a provider key, or a switch to a free model in Pullfrog's settings. I haven't re-run it, because a re-run would hit the same billing error.

Everything else in CI passes on this head: lint_and_typecheck and test_matrix on Node 20, 22, 24, and 26.


Generated by Claude Code

@codspeed

codspeed Bot commented Oct 2, 2026

Copy link
Copy Markdown

Congrats! CodSpeed is installed 🎉

🆕 19 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


Open in CodSpeed

@dinwwwh
dinwwwh merged commit c4640c6 into main Oct 2, 2026
8 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants