[sderossi]

PR to generate Markdown version of each HTML page at build time, to serve to AI Agents, with no active code component. Just Markdown aot generation.

Also remove inline CSS from all pages - reduces page size by 50%+, and makes real content of the page far more accessible to AIs.

Why

Every docs page carries about 155 KB of markup (inline CSS, sidebar, inline SVG) before <main>, and only 7–9 KB of text. Agents that read raw HTML truncate before the content or spend tens of thousands of tokens on CSS. Claude Code's WebFetch sends Accept: text/markdown, text/html, */*, so it would take Markdown if the site offered it. Today /docs/<page>.md, /docs/llms.txt and /docs/llms-full.txt all return 404.

What the build now writes

URL	Contents
/docs/<page>.md	Markdown version of each docs page (43 of them), with title and source (the HTML URL) in frontmatter
<link rel="alternate" type="text/markdown">	Added to each page's <head>, pointing at its .md
/docs/llms.txt	The pages in sidebar order with their descriptions, linking the .md versions, plus a pointer to the CheerpJ skill. Copyright notices go under ## Optional
/docs/llms-full.txt	The pages llms.txt lists outside ## Optional, in one file (37 pages, about 228 KB)

How

• packages/astro-theme/markdown.ts: after the build, converts each page's <main> with rehype-remark. Converting the built HTML rather than the sources handles MDX components, replaced variables, callouts and Expressive Code blocks as the reader sees them:
  • code blocks keep their language, with the filename label as a line above;
  • both callout markups ([!note] and <Callout>) become blockquotes led by a bold title;
  • links between docs pages point at the .md versions; other links and images become absolute URLs.
• packages/astro-theme/index.ts: an opt-in markdown option, off by default, so the other labs sites don't change.
• sites/cheerpj: turns markdown: true on. src/pages/llms.txt.ts builds llms.txt from the same nav data as the sidebar. public/_headers serves .md files as text/markdown and the llms files as text/plain.
• Dependencies: unified, rehype-parse, rehype-remark, remark-gfm, remark-stringify and hast-util-to-text added to the theme. All but rehype-remark were already in the lockfile.

Stylesheets as files (all docs sites)

packages/astro-theme/index.ts sets inlineStylesheets to "auto" (Astro's default, which inlines only stylesheets under 4 KB) instead of "always". That had put the same ~109 KB of CSS before <main> on every page, sent again with each page because the HTML isn't cached. This change applies to every site using the theme.

Getting started: Java app	Before	After
<main> starts at	155.5 KB	47.8 KB
Page size	193 KB	85 KB
Compressed	~42 KB	~22 KB
Shared CSS	inline in every page	one 103 KB file (18 KB compressed), downloaded once

Screenshots of 4 CheerpJ pages (Java app, Overview, cheerpjInit, FAQ) at 1440 and 390 px are pixel-identical to the live site. The CheerpX site also builds, with its CSS now external.

Verified locally

• pnpm run build in sites/cheerpj writes 43 .md files, llms.txt and llms-full.txt.
• No raw HTML outside code fences in the output.
• Of 343 internal links in the output, all resolve except /docs/community, which the HTML page links to as well and _redirects handles.

Not in this PR

• Inline SVG: the CheerpJ wordmark is inlined twice on every page, about 16 KB.

[cloudflare-workers-and-pages]

Deploying labs-browserpod-previews with    Cloudflare Pages

Latest commit:	8d5695a
Status:	✅  Deploy successful!
Preview URL:	https://678c3ca8.labs-browserpod-previews.pages.dev
Branch Preview URL:	https://cheerpj-docs-markdown.labs-browserpod-previews.pages.dev

View logs

[cloudflare-workers-and-pages]

Deploying labs-cheerpj-previews with    Cloudflare Pages

Latest commit:	8d5695a
Status:	✅  Deploy successful!
Preview URL:	https://7b8292a2.labs-cheerpj-previews.pages.dev
Branch Preview URL:	https://cheerpj-docs-markdown.labs-cheerpj-previews.pages.dev

View logs

[alexp-sssup]

I think this is backwards, for two reasons:

1. The content is already markdown to being with. A saner strategy would be to copy existing source markdown files or to reference them directly from the meta tag.

2. The reason these pages are massive is that CSS style and SVGs are inline. They should be place in separate files. It's also likely that the inline stylesheet could be pruned from unused styles, I don't know how this should be done but it seems all quite doable.

As I was writing I had an analysis in the background. Apparently this could already slice the HTML side in half (untested):

packages/astro-theme/index.ts:136 sets inlineStylesheets: "always"

[sderossi]

I am aware of the CSS inlining blowing up size 2x for no reason, I didn't fix that as I didn't have time to test, and it touches all docs sites.

In any case, this does not solve the root issue.

1. Markdown is the source of 1/2 of the pages, 1/2 are mdx
2. Markdown is the source, but it does not represent the content of the build 1-to-1
3. links and components need to be re-rendered to markdown

Yes, there is a case that even if the alternative markdown file is not exactly showing the same content of the HTML, it's all good. But there is also a case that the Astro 'renderer' has already run, it shows what it shows in html, and it is very cheap to get back to markdown from it.

[GabrielaReyna]

Approving the approach. On the design question raised: converting the built HTML is doing real work. For example, <AppletRunnerButton /> comes through as [CheerpJ Applet Runner](https://cheerpj.com/extensions/) with its text
intact. Copying the source .mdx would have produced nothing there. 13 of the 38 doc sources are .mdx, including the overview and three of four getting-started pages, and source links point at HTML routeThe round trip earns its keep.

Before merging please fix the callout bugs and let's consider the action to be taken about cheerpj.com/llms.txt. Either suggested option is fine, but it shouldn't be forgotten or agents won't find the Markdown twins.

I also suggested to add some comments on areas where is not a problem today, but could be an issue when merging the docs site to the main site.

[GabrielaReyna]

There is a loss of information for the .md twin on the callouts:
Only the title and the first paragraph is kept, but everything else is lost (2nd, 3rd... paragraphs).

Suggested solution:
To keep the blockquote’s other children too, instead of only .callout-content’s.”:

const contentBox = find(quote, (e) => classes(e).includes("callout-content"));
const content = quote.children.flatMap((c) =>
    c === titleBox ? [] : c === contentBox ? contentBox.children : [c]
);

[GabrielaReyna]

The convention expects this file at the root of the domain
(cheerpj.com/llms.txt). Because the docs site is built with base: "/docs/",
it lands at cheerpj.com/docs/llms.txt instead, where agents won't look.
cheerpj.com/llms.txt already exists (on leaningtech.com repo I think) and points to
/docs/index.html, not to any of this.

We got two options that depends on how fast we want this to be out:

• a) Add two lines to llms.txt in the cheerpj.com repo pointing here, so it
  works as soon as this merges.
• b) Leave it until the repos merge and fix it there.
  Let's decide which approach is best.

[GabrielaReyna]

Note for the future merge of the docs site into cheerpj.com, no action needed now. This assumes every .md link in llms.txt is a docs page we generated. After the merge that stops being true: the root llms.txt will also list marketing pages and the skill, which have no .md here, and the build will
crash on them.

Leaving this here so it's easy to find when it happens. Consider adding a warning comment on code for the future merger.

[GabrielaReyna]

Minor thing: This file doesn't explain itself. Worth adding some further comments here.

[GabrielaReyna]

Worth a note here for whoever handles the repo merge: this converts built HTML, so it depends on class names owned by other packages: expressive-code, callout-title / callout-content, data-pagefind-body.
A major bump of any of them, or re-resolved versions after a repo move, can quietly degrade the output without failing the build. I simulated an expressive-code rename and every code block kept its text
but lost its language tag.

Suggested addition after line 8:

// This depends on class names owned by other packages such as
// `expressive-code` (code frames), `callout-title` / `callout-content`
// (remark-obsidian-callout), `data-pagefind-body` (pagefind). A major bump of
// any of them, or a move to another repo where versions re-resolve, can quietly
// degrade the output. After upgrading, check a page with code and a callout.