Skip to content

perf: highlight with fewer grammars by default - #1235

Open
ovflowd wants to merge 1 commit into
mainfrom
perf/fewer-grammars
Open

perf: highlight with fewer grammars by default#1235
ovflowd wants to merge 1 commit into
mainfrom
perf/fewer-grammars

Conversation

@ovflowd

@ovflowd ovflowd commented Nov 3, 2026

Copy link
Copy Markdown
Member

Description

This PR highlights code with fewer grammars by default, as each one takes its own memory on every thread, and time to register. JavaScript (js, mjs, cjs, javascript) now uses TypeScript's grammar, which already highlights types, and C++ (cpp, c++) uses C's. JavaScript's grammar alone is ~60MB on every thread: on Node.js's docs this saves ~190MB of peak memory and 2-7% of build time.

In the light theme JavaScript comes out the same, except one block of console output marked as js. Nord, the dark theme, loses a few colors it only gives JavaScript's scopes, and C++-only keywords lose theirs. langAlias takes precedence, so { cpp: 'cpp', 'c++': 'cpp' } brings C++'s own grammar back.

doc-kit now resolves aliases itself, since Shiki replaced configured ones with a grammar's own as it registered. Aliases also follow other aliases and match in any case. resolveLanguage, and transformers, get the language code is highlighted as (typescript for mjs).

Validation

Added highlights JavaScript as TypeScript, and C++ as C, takes aliases over the default ones, follows configured aliases, and finds languages named in any case and highlights with a language of langs an alias leads to. node --run test, lint and format:check are clean.

The memory and time are medians on Node.js's docs on Node.js 24.

Related Issues

Split out of #1205.

Check List

  • I have read the Contributing Guidelines and made commit messages that follow the guideline.
  • I have run node --run test and all tests passed.
  • I have check code formatting with node --run format:check & node --run lint.
  • I've covered new added functionality with unit tests if necessary.

@ovflowd
perf: highlight with fewer grammars by default
2b215f2 
Each grammar costs every thread its own memory, about 60MB for
JavaScript's on Oniguruma, and time to register. Types and signatures
are already highlighted as TypeScript, whose grammar highlights
JavaScript too, so `js`, `mjs`, `cjs` and `javascript` code is now
highlighted with it by default, and C++ (`cpp`, `c++`) with C's. On
Node.js's docs that saves ~190MB of peak memory and 2-7% of build time.

In the light theme, the default, JavaScript comes out the same but for
one block of console output marked as `js`. Nord, the dark theme, has
a few colors only for JavaScript's scopes. C++-only keywords lose their
color.

`langAlias` takes precedence, and doc-kit now resolves aliases itself:
Shiki replaced configured aliases with a grammar's own as it
registered. Aliases now also lead on through other aliases and are
found in any case. Transformers, and the language `resolveLanguage`
returns, are the one code is highlighted as.

Assisted-by: Claude Opus 5.5 <noreply@anthropic.com>
Assisted-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
api-docs-tooling Ready Ready Preview Nov 3, 2026 7:34pm UTC

Request Review

@codecov

codecov Bot commented Nov 3, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.19%. Comparing base (c8b228e) to head (2b215f2).

Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #1235      +/-   ##
==========================================
+ Coverage   94.14%   94.19%   +0.05%     
==========================================
  Files         291      292       +1     
  Lines       28950    29065     +115     
  Branches     2944     2961      +17     
==========================================
+ Hits        27256    27379     +123     
+ Misses       1672     1664       -8     
  Partials       22       22

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ovflowd ovflowd mentioned this pull request Nov 3, 2026
Closed
4 tasks done
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 3, 2026

Copy link
Copy Markdown

🚀 Deploying Preview to Cloudflare 🚀

Preview Deployments by commit

Status Deployment URL Commit Updated (UTC) See this deployment's details
  • Build: Success ✅

View logs ↗		 No Preview URL. Enable ↗ 2b215f2 2026-11-03T19:34:36.267Z View logs ↗

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

api-links Generator

Performance estimate (single CI run)

  • Generation time: 3.8% faster (1.05 s → 1.01 s)
  • Peak memory: 0.1% lower (293.49 MB → 293.07 MB)

json Generator

Performance estimate (single CI run)

  • Generation time: 40.9% faster (9.48 s → 5.60 s)
  • Peak memory: 0.8% higher (997.38 MB → 1005.43 MB)

legacy-html Generator

Output size: 10 files changed · net -52.57 KB

File size details
File Main PR Change
addons.html 374.23 KB 328.87 KB -45.36 KB (-12.1%)
embedding.html 64.38 KB 57.78 KB -6.60 KB (-10.3%)
async_context.html 247.17 KB 245.10 KB -2.08 KB (-0.8%)
tls.html 297.30 KB 298.85 KB +1.54 KB (+0.5%)
n-api.html 822.59 KB 822.23 KB -364.00 B (-0.0%)
module.html 364.54 KB 364.76 KB +224.00 B (+0.1%)
readline.html 219.10 KB 219.27 KB +168.00 B (+0.1%)
worker_threads.html 417.80 KB 417.64 KB -168.00 B (-0.0%)
repl.html 226.64 KB 226.75 KB +112.00 B (+0.0%)
test.html 1009.73 KB 1009.67 KB -56.00 B (-0.0%)

Performance estimate (single CI run)

  • Generation time: 25.6% slower (12.36 s → 15.52 s)
  • Peak memory: 9.5% lower (1.52 GB → 1.38 GB)

legacy-json Generator

Performance estimate (single CI run)

  • Generation time: 15.8% slower (7.23 s → 8.37 s)
  • Peak memory: 7.4% higher (934.98 MB → 1004.21 MB)

llms-txt Generator

Performance estimate (single CI run)

  • Generation time: 1.3% slower (8.43 s → 8.54 s)
  • Peak memory: 4.4% higher (864.12 MB → 902.01 MB)

orama-db Generator

Performance estimate (single CI run)

  • Generation time: 15.6% faster (8.82 s → 7.44 s)
  • Peak memory: 8.2% higher (992.37 MB → 1.05 GB)

web Generator

Output size: 46 files changed · net -108.25 KB

File size details
File Main PR Change
all.html 34.78 MB 34.73 MB -54.12 KB (-0.2%)
addons.html 389.73 KB 361.93 KB -27.80 KB (-7.1%)
embedding.html 73.02 KB 68.39 KB -4.63 KB (-6.3%)
process.html 1.20 MB 1.19 MB -4.16 KB (-0.3%)
child_process.html 706.47 KB 704.23 KB -2.24 KB (-0.3%)
tls.html 550.15 KB 552.33 KB +2.18 KB (+0.4%)
fs.html 2.36 MB 2.35 MB -1.32 KB (-0.1%)
crypto.html 2.03 MB 2.03 MB -1.21 KB (-0.1%)
assert.html 673.00 KB 671.89 KB -1.11 KB (-0.2%)
http.html 1.23 MB 1.23 MB -1.06 KB (-0.1%)
util.html 1.26 MB 1.26 MB -964.00 B (-0.1%)
async_hooks.html 293.14 KB 292.26 KB -898.00 B (-0.3%)
http2.html 1.31 MB 1.31 MB -788.00 B (-0.1%)
readline.html 380.82 KB 380.05 KB -788.00 B (-0.2%)
buffer.html 1.82 MB 1.82 MB -736.00 B (-0.0%)
dgram.html 313.49 KB 312.78 KB -728.00 B (-0.2%)
vm.html 656.77 KB 656.11 KB -680.00 B (-0.1%)
webstreams.html 565.36 KB 564.70 KB -674.00 B (-0.1%)
perf_hooks.html 910.85 KB 910.24 KB -620.00 B (-0.1%)
cluster.html 317.74 KB 317.14 KB -616.00 B (-0.2%)
worker_threads.html 662.16 KB 661.58 KB -588.00 B (-0.1%)
zlib.html 1.00 MB 1023.86 KB -504.00 B (-0.0%)
test.html 1.47 MB 1.47 MB -458.00 B (-0.0%)
ffi.html 204.00 KB 203.56 KB -448.00 B (-0.2%)
globals.html 302.35 KB 301.96 KB -394.00 B (-0.1%)
vfs.html 213.64 KB 213.31 KB -340.00 B (-0.2%)
url.html 533.89 KB 533.56 KB -338.00 B (-0.1%)
stream_iter.html 795.19 KB 794.90 KB -296.00 B (-0.0%)
v8.html 574.47 KB 574.19 KB -294.00 B (-0.0%)
async_context.html 329.94 KB 329.71 KB -234.00 B (-0.1%)
n-api.html 1.05 MB 1.05 MB -230.00 B (-0.0%)
dns.html 442.03 KB 441.81 KB -224.00 B (-0.0%)
sqlite.html 542.98 KB 542.81 KB -174.00 B (-0.0%)
domain.html 131.23 KB 131.06 KB -168.00 B (-0.1%)
quic.html 986.03 KB 985.87 KB -168.00 B (-0.0%)
stream.html 1.27 MB 1.27 MB -122.00 B (-0.0%)
repl.html 314.01 KB 313.90 KB -116.00 B (-0.0%)
modules.html 238.01 KB 237.90 KB -112.00 B (-0.0%)
tty.html 113.74 KB 113.63 KB -112.00 B (-0.1%)
module.html 533.05 KB 532.96 KB -84.00 B (-0.0%)
bench.html 213.44 KB 213.38 KB -60.00 B (-0.0%)
cli.html 678.78 KB 678.73 KB -56.00 B (-0.0%)
permissions.html 94.66 KB 94.60 KB -56.00 B (-0.1%)
esm.html 196.06 KB 196.05 KB -8.00 B (-0.0%)
single-executable-applications.html 170.64 KB 170.64 KB -2.00 B (-0.0%)
wasi.html 83.98 KB 83.98 KB -2.00 B (-0.0%)

Performance estimate (single CI run)

  • Generation time: 37.7% faster (23.90 s → 14.89 s)
  • Peak memory: 7.6% higher (1.48 GB → 1.59 GB)

@ovflowd
ovflowd marked this pull request as ready for review November 3, 2026 20:39
@ovflowd
ovflowd requested a review from a team as a code owner November 3, 2026 20:39
bmuenzenmeyer
bmuenzenmeyer reviewed Nov 3, 2026
View reviewed changes
Comment thread packages/core/src/plugins/shiki/__tests__/highlighter.test.mjs

assert.ok(!loaded().includes('python'));
assert.ok(!loaded().includes('javascript'));
assert.ok(!loaded().includes('rust'));

@bmuenzenmeyer bmuenzenmeyer Nov 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is rust here? do we have this in the codebase? its not in the release note or PR

This branch was successfully deployed

1 active deployment
Preview – api-docs-tooling — 2b215f2d Deployed Nov 3, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bmuenzenmeyer bmuenzenmeyer bmuenzenmeyer left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ovflowd @bmuenzenmeyer