Repository navigation
Conversation
Every site listed /docs in its sitemap. That URL is the injected docs index, a meta refresh page (a 301 on cheerpj.com), so crawlers were given a redirect instead of a page. The theme's sitemap() now leaves it out. cheerpj.com's own sitemap index points to https://cheerpj.com/docs/sitemap.xml, which 404'd: @astrojs/sitemap only writes sitemap-index.xml. A small build:done step copies it to sitemap.xml on every site. On labs, the /cheerpj stub page (a meta refresh to /cheerpj3) is gone: the _redirects 301 to cheerpj.com always answered first, and the stub only put a redirect in the sitemap. The /sitemap.xml -> /sitemap-index.xml redirect (a 302) is gone too, now that the file exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deploying labs-browserpod-previews with
|
| Latest commit: |
1d5183d
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://a7f479be.labs-browserpod-previews.pages.dev |
| Branch Preview URL: | https://fix-sitemap.labs-browserpod-previews.pages.dev |
Deploying labs-cheerpj-previews with
|
| Latest commit: |
1d5183d
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://fd6185c4.labs-cheerpj-previews.pages.dev |
| Branch Preview URL: | https://fix-sitemap.labs-cheerpj-previews.pages.dev |
GabrielaReyna
left a comment
GabrielaReyna
left a comment
There was a problem hiding this comment.
Nice catch on this one, rebuilt all five sites and your numbers match exactly,
all 319 sitemap URLs come back 200 👌
One important thing before merging:
cheerpj.com/sitemap.xmlis already a<sitemapindex>, and this makes/docs/sitemap.xmlanother one. Google doesn't do nested indexes,
so the 404 goes away but Search Console probably still skips the docs URLs.
Copyingsitemap-0.xmlinstead gives a plain<urlset>, one word at
index.ts:40.
Also, I suggested some comment for the future site merger (docs into the main site)
| await copyFile( | ||
| new URL("sitemap-index.xml", dir), | ||
| new URL("sitemap.xml", dir) | ||
| ); |
There was a problem hiding this comment.
I think this lands us in a nested sitemap index. cheerpj.com/sitemap.xml is already a <sitemapindex>, and this makes /docs/sitemap.xml one too. Google doesn’t support an index inside an index (Mueller confirms), so GSC may still reject it. Could we copy sitemap-0.xml instead?
| sitemap({ | ||
| // /docs is the injected docs index: a meta refresh page (and a 301 on | ||
| // cheerpj.com) that sends visitors to the first docs page. Sitemaps | ||
| // list only the pages that answer 200 with content. | ||
| filter: (page) => !/^\/docs\/?$/.test(new URL(page).pathname), | ||
| }), |
There was a problem hiding this comment.
Can be addressed in this PR or as a follow-up:
/docs/404 is still listed on labs, cheerp and cheerpx, and it answers 200 with “page not found” content. (cheerpj and browserpod escape it because their 404 sits at the base root, which Astro drops on its own.) Could the regex cover it too (e.g. !/^/docs(/404)?/?$/)?
| name: "@leaningtech/astro-theme/sitemap-alias", | ||
| hooks: { | ||
| "astro:build:done": async ({ dir }) => { | ||
| await copyFile( |
There was a problem hiding this comment.
Small robustness thing, non blocking: if @astrojs/sitemap doesn’t write the index (no site set, or everything filtered out) it only warns, and this copyFile then throws ENOENT(Error: NO ENTry) and fails the build.
Since this is in the shared theme, could we skip quietly when the source file isn’t there? Something like wrapping it in existsSync, or catching ENOENT and logging a warning.
| // /docs is the injected docs index: a meta refresh page (and a 301 on | ||
| // cheerpj.com) that sends visitors to the first docs page. Sitemaps | ||
| // list only the pages that answer 200 with content. | ||
| filter: (page) => !/^\/docs\/?$/.test(new URL(page).pathname), |
There was a problem hiding this comment.
we already compute base on line 51, using it instead of hardcoding /docs would keep this working if a site ever uses a different base.
| theme?: string | ExpressiveCodeTheme; // Shiki theme name, or a custom ExpressiveCodeTheme instance, defaults to material-theme-darker | ||
| }; | ||
|
|
||
| // @astrojs/sitemap only writes sitemap-index.xml, but cheerpj.com's own sitemap |
There was a problem hiding this comment.
Could we extend this comment with a line about the upcoming site merge (docs to the main site)? The copy is safe right now only because docs sit under /docs/ and the main site’s own sitemap.xml is at the root. If docs ever move to the root, the two collide at the same address and one silently wins. A sentence here would save whoever does the merge from finding out the hard way.
| // @astrojs/sitemap only writes sitemap-index.xml, but cheerpj.com's own sitemap | |
| // @astrojs/sitemap only writes sitemap-index.xml, but cheerpj.com's own sitemap | |
| // index (and the habit of most crawlers) asks for sitemap.xml. Serve the same | |
| // index under both names. | |
| // | |
| // This is safe only while the docs are served from a subfolder (cheerpj.com/docs/): | |
| // the copy lands at /docs/sitemap.xml, clear of the main site's own /sitemap.xml. | |
| // When the docs merge into cheerpj.com, if they move to the site root this copy | |
| // and the main site's sitemap.xml claim the same address and one silently wins. | |
| // Drop this alias at that point, or rename the file. |
| filter: (page) => !/^\/docs\/?$/.test(new URL(page).pathname), | ||
| }), | ||
| sitemapAlias(), // must come after sitemap(): it copies its output | ||
| robotsTxt(), |
There was a problem hiding this comment.
Comment for the future site merge
| robotsTxt(), | |
| // Only effective on sites served from their own domain root (labs, cheerp, | |
| // cheerpx). On cheerpj and browserpod the docs sit under /docs/, and crawlers | |
| // read robots.txt only at the domain root, so /docs/robots.txt, and the | |
| // Sitemap: line in it, is never fetched. Those two rely on the root domain's | |
| // robots.txt (another repo) pointing at the docs sitemap. When the docs merge | |
| // into the main site, move that line into the single root robots.txt and | |
| // drop this integration for the merged site. | |
| robotsTxt(), |
The docs sitemaps list every real page, but cheerpj.com's sitemap index pointed crawlers at a file that doesn't exist, and every site listed a redirect.
Problems
https://cheerpj.com/sitemap.xml(from the leaningtech.com repo) listshttps://cheerpj.com/docs/sitemap.xml, which 404s:@astrojs/sitemaponly writessitemap-index.xml./docs, a redirect. It's the injected docs index, a meta-refresh page with no content. On cheerpj.com it's also a 301 to/docs/overview. Listed on cheerpj.com, cheerpx.io, cheerp.io, browserpod.io and labs.leaningtech.com./cheerpj.src/pages/cheerpj.astrois a meta-refresh stub to/cheerpj3, but the_redirects301 to cheerpj.com always answers first, so the stub can never be served.Changes
packages/astro-theme/index.ts:sitemap({ filter })leaves/docsout on every site.astro:build:donestep, added aftersitemap(), copiessitemap-index.xmltositemap.xml. Once deployed,https://cheerpj.com/docs/sitemap.xmlresolves and cheerpj.com's index works without a change in leaningtech.com./cheerpjstub page./sitemap.xml→/sitemap-index.xmlredirect (a 302), since the file now exists.Checked locally
All five sites build. Each sitemap lost only the bad entries:
/docs/docs,/cheerpj/docs/docs/docsNo URL left in any sitemap is a redirect page, and every
sitemap.xmlmatches itssitemap-index.xmlbyte for byte. The CheerpJ docs sitemap still lists all 43 real docs pages; only the 404 page is left out.After deploy
https://cheerpj.com/docs/sitemap.xmlanswers 200 with the sitemap index.https://cheerpj.com/docs/sitemap-0.xmlno longer listshttps://cheerpj.com/docs.https://labs.leaningtech.com/sitemap.xmlanswers 200 directly, with no 302.https://cheerpj.com/sitemap.xmlin Google Search Console.Not changed
cheerpx's sitemap has a URL with a space in it,
/docs/reference/CheerpX.Linux/event%20callbacks. It works; a hyphenated slug would be cleaner.🤖 Generated with Claude Code