From 5f7e3a4b7b1beaf1e9132d0d2097de1334c30f38 Mon Sep 17 00:00:00 2001 From: Marco Walz Date: Wed, 16 Sep 2026 12:33:52 +0200 Subject: [PATCH 01/17] feat: sync certified-assets docs into Frontends Publishes the eight user-doc pages of dfinity/certified-assets as a synced tree under guides/frontends/static-site/, with the repo staying the single source of truth. Fetch-based rather than a submodule: the build resolves no file from that repo, only markdown links, so the pin is a ref in .sources/upstream.json under a new `synced` group and the pages are fetched at that ref. The script exits non-zero instead of publishing something broken: missing frontmatter, an absolute link to this site it cannot map, a relative link that does not resolve, or a banned character that survived normalization. The tree is deliberately not exempt from the validator. Pinned to a commit rather than the v0.3.3 tag: the only two commits past that release are the upstream docs PRs, and the tag predates the frontmatter contract this sync requires. Refs #396 --- .agents/upstream-tracking.md | 73 ++++- .github/workflows/sync-static-site.yml | 155 ++++++++++ .sources/upstream.json | 31 +- AGENTS.md | 36 ++- .../static-site/access-protection.md | 200 ++++++++++++ docs/guides/frontends/static-site/headers.md | 137 +++++++++ .../frontends/static-site/how-it-works.md | 173 +++++++++++ docs/guides/frontends/static-site/overview.md | 152 +++++++++ .../guides/frontends/static-site/redirects.md | 114 +++++++ docs/guides/frontends/static-site/routing.md | 134 ++++++++ .../frontends/static-site/site-files.md | 71 +++++ .../static-site/verifying-contents.md | 131 ++++++++ package.json | 1 + scripts/sync-static-site.mjs | 291 ++++++++++++++++++ 14 files changed, 1686 insertions(+), 13 deletions(-) create mode 100644 .github/workflows/sync-static-site.yml create mode 100644 docs/guides/frontends/static-site/access-protection.md create mode 100644 docs/guides/frontends/static-site/headers.md create mode 100644 docs/guides/frontends/static-site/how-it-works.md create mode 100644 docs/guides/frontends/static-site/overview.md create mode 100644 docs/guides/frontends/static-site/redirects.md create mode 100644 docs/guides/frontends/static-site/routing.md create mode 100644 docs/guides/frontends/static-site/site-files.md create mode 100644 docs/guides/frontends/static-site/verifying-contents.md create mode 100644 scripts/sync-static-site.mjs diff --git a/.agents/upstream-tracking.md b/.agents/upstream-tracking.md index d4a4ec11..7b906de3 100644 --- a/.agents/upstream-tracking.md +++ b/.agents/upstream-tracking.md @@ -2,14 +2,14 @@ How this repo stays current with the projects it documents. -Upstream repos fall into three groups, and the group decides the procedure. +Upstream repos fall into four groups, and the group decides the procedure. -| | Vendored (submodule) | Watched | Reference | -|---|---|---|---| -| Which | `motoko`, `internetidentity`, `examples` | the `watched` array in `.sources/upstream.json` | the `reference` array | -| Why | the build opens their files | a release can silently invalidate a lot of published content | drawn on too lightly, or another check already catches the drift | -| Pin | the gitlink | `pinned` in `upstream.json` | none; verify against the latest release | -| Release issue | only `examples`, and only when a quoted file moved | yes | no | +| | Vendored (submodule) | Synced | Watched | Reference | +|---|---|---|---|---| +| Which | `motoko`, `internetidentity`, `examples` | the `synced` array in `.sources/upstream.json` | the `watched` array | the `reference` array | +| Why | the build opens their files | their own pages are published here as-is | a release can silently invalidate a lot of published content | drawn on too lightly, or another check already catches the drift | +| Pin | the gitlink | `pinned` in `upstream.json` | `pinned` in `upstream.json` | none; verify against the latest release | +| Release issue | only `examples`, and only when a quoted file moved | no; its own workflow opens the bump PR | yes | no | Deciding between the last two is a judgment about blast radius, and the `why` field on each `reference` entry records the footprint that decided it. Promote an @@ -236,6 +236,65 @@ git -C .sources/ fetch --unshallow git -C .sources/ checkout ``` +## Synced trees + +A synced tree is a set of pages this repo publishes but does not write. Upstream +owns the prose; a sync script fetches it at a pinned ref, adapts the few things +that only make sense on this site, and writes the result into `docs/`. + +| Tree | Upstream | Script | Workflow | +|---|---|---|---| +| `docs/guides/frontends/static-site/` | `dfinity/certified-assets` `docs/` | `scripts/sync-static-site.mjs` | `.github/workflows/sync-static-site.yml` | + +`motoko` and `internetidentity` are synced too, but they are submodules because +the build also opens their files, so they follow the vendored procedure above. +This group is for the case where nothing is resolved at build time, only +markdown links, which is why there is no submodule to hold the pin. + +### How the pin moves + +The workflow runs weekly, resolves the latest release tag, and opens the bump PR +only when `docs/` actually changed in that range. A release that ships canister +changes without touching `docs/` produces nothing here; the recipe version +readers type is covered separately by the `static-site` entry under `watched`. + +To sync by hand, or to trial a ref before pinning it: + +```bash +npm run sync:static-site # uses the pin in .sources/upstream.json +node scripts/sync-static-site.mjs --ref main +``` + +The script exits non-zero rather than publishing something broken: missing +frontmatter, an absolute link to this site it cannot map, a relative link that +does not resolve, or a banned character that survived normalization. Fix the +cause, do not hand-edit the output. + +### The pin may sit ahead of the latest release + +`.sources/VERSIONS` forbids pinning a submodule past its latest release, so that +docs cannot describe behavior users cannot run yet. A synced tree can need the +opposite: a docs-only fix upstream is published before the next release, and the +pages must be syncable now. `certified-assets` started exactly there, pinned to a +commit because the `v0.3.3` tag predated the frontmatter contract the sync +requires (`certified-assets#124`). + +The rule that matters is the one behind it: never pin past a commit that +documents unreleased behavior. Docs-only commits are safe, so check what the +range contains before pinning past a tag, and record why in the entry's +`$comment`. Move the pin back onto release tags as soon as one includes the +change. + +### On bump, check + +1. The page diffs, for content changes; the tree is regenerated wholesale +2. Whether a behavior change contradicts our own pages: `certification.md` + describes what this canister certifies, `asset-canister.md` contrasts the two + recipes +3. Whether prose naming the recipe version needs bumping with it +4. Whether a normalization the script reports should be fixed upstream instead, + so it becomes a no-op + ## Build dependencies (npm) The packages in `package.json` are a separate axis: they decide whether the site diff --git a/.github/workflows/sync-static-site.yml b/.github/workflows/sync-static-site.yml new file mode 100644 index 00000000..c296df22 --- /dev/null +++ b/.github/workflows/sync-static-site.yml @@ -0,0 +1,155 @@ +name: Sync static-site docs + +on: + schedule: + - cron: '0 9 * * 3' # Weekly on Wednesday + workflow_dispatch: + +jobs: + sync: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + fetch-depth: 0 + + - name: Create GitHub App Token + uses: actions/create-github-app-token@1b10c78c7865c340bc4f6099eb2f838309f1e8c3 # v3.1.1 + id: app-token + with: + client-id: ${{ vars.PR_AUTOMATION_BOT_PUBLIC_CLIENT_ID }} + private-key: ${{ secrets.PR_AUTOMATION_BOT_PUBLIC_PRIVATE_KEY }} + + # certified-assets is not a submodule: the build resolves no file from it. + # A blobless clone is only needed to reason about refs (is the latest + # release already contained in the pin, did docs/ change), which the + # contents API cannot answer. + - name: Clone certified-assets + run: git clone --filter=blob:none --no-checkout https://github.com/dfinity/certified-assets.git /tmp/certified-assets + + - name: Resolve the ref to sync + id: check + run: | + PIN=$(node -p "require('./.sources/upstream.json').synced.find(e => e.repo === 'dfinity/certified-assets').pinned") + echo "pin=$PIN" >> $GITHUB_OUTPUT + + TAG=$(git -C /tmp/certified-assets tag --sort=-version:refname | grep '^v[0-9]' | head -1) + echo "Pinned: $PIN. Latest release: $TAG." + echo "tag=$TAG" >> $GITHUB_OUTPUT + + # The pin is allowed to sit ahead of the latest release while a docs + # fix has shipped but a release has not (the state this sync started + # in). Syncing the tag then would publish older prose. + if git -C /tmp/certified-assets merge-base --is-ancestor "$TAG" "$PIN"; then + echo "Pin already contains $TAG. Nothing to sync." + echo "needed=false" >> $GITHUB_OUTPUT + exit 0 + fi + + BRANCH="infra/sync-static-site-${TAG}" + if git ls-remote --exit-code origin "refs/heads/${BRANCH}" > /dev/null 2>&1; then + echo "Branch $BRANCH already exists, so a PR is likely open. Skipping." + echo "needed=false" >> $GITHUB_OUTPUT + exit 0 + fi + + # A release that ships canister changes without touching docs/ leaves + # the synced pages byte-identical. The recipe version readers type is + # covered by the `static-site` entry under `watched`, which opens its + # own issue, so there is nothing to review here. + CHANGED=$(git -C /tmp/certified-assets diff --name-only "${PIN}..${TAG}" -- docs/) + if [ -z "$CHANGED" ]; then + echo "No docs/ changes between $PIN and $TAG. Skipping." + echo "needed=false" >> $GITHUB_OUTPUT + else + echo "Changed upstream pages:" + echo "$CHANGED" + echo "needed=true" >> $GITHUB_OUTPUT + echo "changed_files<> $GITHUB_OUTPUT + echo "$CHANGED" >> $GITHUB_OUTPUT + echo "EOF" >> $GITHUB_OUTPUT + fi + + - name: Move the pin to the release tag + if: steps.check.outputs.needed == 'true' + run: | + node -e ' + const fs = require("fs"); + const file = ".sources/upstream.json"; + const config = JSON.parse(fs.readFileSync(file, "utf8")); + const entry = config.synced.find((e) => e.repo === "dfinity/certified-assets"); + entry.pinned = process.argv[1]; + fs.writeFileSync(file, JSON.stringify(config, null, 2) + "\n"); + ' "${{ steps.check.outputs.tag }}" + + - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4.4.0 + if: steps.check.outputs.needed == 'true' + with: + node-version: 22 + cache: npm + + - name: Install dependencies + if: steps.check.outputs.needed == 'true' + run: npm ci + + - name: Run static-site docs sync + if: steps.check.outputs.needed == 'true' + run: npm run sync:static-site + env: + GITHUB_TOKEN: ${{ steps.app-token.outputs.token }} + + # The synced tree is deliberately not in the validator's SYNCED allowlist: + # the sync normalizes and rewrites so that the pages pass the same checks + # as a hand-written page, and an exemption would hide the day they stop. + - name: Validate + if: steps.check.outputs.needed == 'true' + run: npm run validate + + - name: Initialize examples submodule (required for build) + if: steps.check.outputs.needed == 'true' + run: git submodule update --init --depth 1 .sources/examples + + - name: Build check + if: steps.check.outputs.needed == 'true' + run: npm run build + + - name: Create PR + if: steps.check.outputs.needed == 'true' + run: | + git config user.name "pr-automation-bot-public[bot]" + git config user.email "pr-automation-bot-public[bot]@users.noreply.github.com" + + BRANCH="infra/sync-static-site-${{ steps.check.outputs.tag }}" + git checkout -b "$BRANCH" + git add .sources/upstream.json docs/guides/frontends/static-site + git commit -m "chore: sync static-site docs to dfinity/certified-assets ${{ steps.check.outputs.tag }}" + git push -u origin "$BRANCH" + + CHANGED="${{ steps.check.outputs.changed_files }}" + { + echo "## Summary" + echo "" + echo "Automated sync of the static-site (certified-assets) user docs." + echo "" + echo "**Release:** \`${{ steps.check.outputs.tag }}\` (pinned from \`${{ steps.check.outputs.pin }}\`)" + echo "" + echo "**Changed upstream files:**" + while IFS= read -r f; do + [ -n "$f" ] && echo "- \`$f\`" + done <<< "$CHANGED" + echo "" + echo "- Ran \`npm run sync:static-site\`, regenerating \`docs/guides/frontends/static-site/\`" + echo "- Validator and build passed" + echo "" + echo "## Checklist" + echo "" + echo "- [ ] Review the page diffs for content changes" + echo "- [ ] Check whether a behavior change contradicts our own Frontends pages (\`certification.md\`, \`asset-canister.md\`)" + echo "- [ ] Check whether the recipe version named in \`icp.yaml\` examples needs bumping with it" + } > /tmp/pr-body.md + + gh pr create \ + --title "chore: sync static-site docs to dfinity/certified-assets ${{ steps.check.outputs.tag }}" \ + --body-file /tmp/pr-body.md + env: + GH_TOKEN: ${{ steps.app-token.outputs.token }} diff --git a/.sources/upstream.json b/.sources/upstream.json index 152852f5..3162f5c7 100644 --- a/.sources/upstream.json +++ b/.sources/upstream.json @@ -1,11 +1,17 @@ { "$comment": [ - "Every upstream this repo tracks, in three groups.", + "Every upstream this repo tracks, in four groups.", "", "`watched` are repos where a release can silently invalidate a lot of published", "content, so they get a pinned ref and a weekly issue when they move. Verify", "claims against the pinned ref, not against main, so a review is reproducible.", "", + "`synced` are repos whose own pages are copied into docs/ by a sync script. The", + "pin is recorded here because there is no gitlink to hold it: the build resolves", + "no file from them, so they are not submodules. Their sync workflow does its own", + "release check and opens the bump PR, the way sync-motoko.yml and sync-ii-spec.yml", + "do, so check-upstream-releases.mjs ignores this group.", + "", "`vendored` are submodules whose pin has no sync workflow of its own. Their pin", "is the gitlink, so git stays the single source of truth. `motoko` and", "`internetidentity` are absent on purpose: sync-motoko.yml and sync-ii-spec.yml", @@ -39,7 +45,28 @@ "", "Fields (vendored): path, repo, track 'branch', branch, affects, and an optional", "pathFilter. 'snippets' reports only when the range touched a file quoted by a", - "`snippet=` in docs/, so an unrelated commit on an active repo stays silent." + "`snippet=` in docs/, so an unrelated commit on an active repo stays silent.", + "", + "Fields (synced): repo, pinned, source, target, script, and affects." + ], + "synced": [ + { + "repo": "dfinity/certified-assets", + "pinned": "d9cb7df", + "source": "docs/", + "target": "docs/guides/frontends/static-site/", + "script": "scripts/sync-static-site.mjs", + "affects": "The synced pages are regenerated wholesale, so review the diff rather than the pages. Behavior changes also reach our own Frontends pages: certification.md describes what this canister certifies, and asset-canister.md contrasts the two recipes. The recipe version readers type is tracked separately as the `static-site` entry under `watched`, and the two move in lockstep.", + "$comment": [ + "Pinned to a commit, not the v0.3.3 tag, and one commit past the latest", + "release. .sources/VERSIONS forbids pinning past a release so that docs", + "cannot describe unshipped behavior; this is the case that rule is not", + "aimed at. The only commits past v0.3.3 are certified-assets#125 and #126,", + "both docs-only, and the tag predates the frontmatter contract this sync", + "requires (certified-assets#124), so v0.3.3 cannot be synced at all.", + "Move the pin to a release tag at the next certified-assets release." + ] + } ], "vendored": [ { diff --git a/AGENTS.md b/AGENTS.md index 00684b0d..0711bfec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -162,7 +162,7 @@ EOF - Removing existing pages - Changing the frontmatter schema - Modifying sidebar configuration (`astro.config.mjs`) -- Adding a new `.sources/` submodule +- Adding a new `.sources/` submodule or a new synced tree ## Never @@ -170,7 +170,7 @@ EOF - Use `mo:base` — use `mo:core` instead. Critical replacements: `Buffer` → `List`, `HashMap`/`TrieMap`/`Trie`/`RBTree` → `Map`, `Deque` → `Queue`, `OrderedMap` → `pure/Map`, `OrderedSet` → `pure/Set` - Create `.mdx` without a clear need for interactive components - Duplicate content that lives in external docs (icp-cli site, JS SDK docs, the IC skills) -- Edit synced files directly (`docs/languages/motoko/`, `docs/references/internet-identity-spec.md`, `docs/references/verifiable-credentials-spec.md`) +- Edit synced files directly (`docs/languages/motoko/`, `docs/guides/frontends/static-site/`, `docs/references/internet-identity-spec.md`, `docs/references/verifiable-credentials-spec.md`) - Nest sidebar items more than 3 levels deep - Add `Co-Authored-By` or any AI attribution to commits or PR descriptions - Link to `internetcomputer.org/docs/` (retired) or `learn.internetcomputer.org` (content is now in this repo under `docs/concepts/`) @@ -187,8 +187,9 @@ EOF - `docs/` — All documentation (`.md` by default). `src/content/docs/` symlinks here. - `docs/languages/motoko/` — Auto-synced from `caffeinelabs/motoko` (do not edit directly) +- `docs/guides/frontends/static-site/` (synced from `dfinity/certified-assets`, do not edit directly) - `docs/references/internet-identity-spec.md`, `docs/references/verifiable-credentials-spec.md` — Synced from `dfinity/internet-identity` (do not edit directly) -- `.sources/` — Vendored submodules, read-only, plus `upstream.json` (watched repos) and `VERSIONS` (submodule pins) +- `.sources/` — Vendored submodules, read-only, plus `upstream.json` (watched and synced repos) and `VERSIONS` (submodule pins) - `.claude/skills/` — Skills. IC skills are mirrored by `.claude/sync-ic-skills.sh` and not committed; `icp-brand-design`, `icp-brand-voice`, and `technical-documentation` are maintained here - `.agents/upstream-tracking.md` — How upstreams are tracked and bumped (maintainer use) @@ -215,7 +216,7 @@ docs/ ## Source material -Upstream repos are tracked two ways. Which one decides where you read from. +Upstream repos are tracked three ways. Which one decides where you read from. **Vendored as submodules** — three repos, because the build opens their files. Read them from disk; do not edit them. @@ -236,6 +237,32 @@ Pinned versions: [`.sources/VERSIONS`](.sources/VERSIONS). `motoko` and `snippet=` quotes; the build already fails if such a path or `#region` marker stops resolving. Canister IDs and code patterns are in the skills (see "Skills"). +**Synced without a submodule** is for a repo whose own pages are published here +as-is. Nothing about them is resolved at build time, only markdown links, so the +pin is a ref in [`.sources/upstream.json`](.sources/upstream.json) under `synced` +and the sync script fetches the pages at that ref. + +| Topic | Repo | Synced tree | +|-------|------|-------------| +| static-site (certified-assets) canister behavior | `dfinity/certified-assets` | `docs/guides/frontends/static-site/` | + +Rules for every synced tree, this one included: + +- **Never hand-edit a page in it.** Content changes belong upstream, and the next + sync overwrites anything else. The pages carry a do-not-edit marker and record + `source_repo` / `source_ref` in their frontmatter. +- **They are exempt from the directory-based Diataxis rule and from + `## Next steps`.** Upstream owns one flat set of pages that mixes how-to, + reference, and explanation, and filing them by type would scatter one source of + truth across three directories. No synced page in this repo carries + `## Next steps`. Our own pages around the tree still follow both rules. +- **They are not exempt from the validator.** Only `docs/languages/motoko/` is, + for historical reasons. A sync script normalizes and rewrites so that its + output passes the same checks as a hand-written page, and it exits non-zero + instead of publishing something that does not. +- Sidebar placement comes from the sync target directory plus upstream's + `sidebar.order`, so adding a page upstream needs no change here. + **Watched** are the repos where a release can silently invalidate a lot of what is published, so [`.sources/upstream.json`](.sources/upstream.json) records the ref the docs are verified against and a weekly workflow opens an issue when one @@ -370,6 +397,7 @@ For design work (CSS, UI, marketing copy), also load `icp-brand-design`. - `guides/` — task-oriented how-to - `references/` — precise lookups, no tutorials - End every page with a `## Next steps` section. +- Synced trees are exempt from the two rules above; see "Source material". ## Linking rules diff --git a/docs/guides/frontends/static-site/access-protection.md b/docs/guides/frontends/static-site/access-protection.md new file mode 100644 index 00000000..14025d3b --- /dev/null +++ b/docs/guides/frontends/static-site/access-protection.md @@ -0,0 +1,200 @@ +--- +title: "Access protection" +description: "Put a login page in front of a private or preview site with revocable, expiring access tokens" +sidebar: + order: 6 +source_repo: "dfinity/certified-assets" +source_ref: "d9cb7df" +--- + +By default every deployed app is public. **Access protection** puts a login screen +in front of it: unauthenticated visitors get a certified redirect to your login +page (or a `401` for non-page assets) instead of your content. It's meant to +**deter casual/public access** to in-progress or preview work, not to provide +confidentiality against a determined attacker (see [Threat model](#threat-model)). + +Access is a set of **labeled tokens**. The cookie *is* the token: a visitor presents +a token, the canister sets it as a cookie, and every request re-validates that cookie +against the token store. A classic single "password" is just one long-lived token. + +When protection is off, the app is **completely unchanged**: no gate, no extra +headers, no cost difference. Everything below applies only once you enable it. + +## Quick start + +Protection is configured through controller-only canister methods, so you call them +with `icp canister call` as the canister's controller. + +```sh +# 1. Add a login page to your site (see "The login page" below) and deploy it. +icp deploy + +# 2. Turn the gate on, naming your login page. +icp canister call frontend enable_protection '("/login.html")' + +# 3. Mint a credential. Here, a chosen "password" valid for ~1 year. +icp canister call frontend issue_token \ + '(record { label = "owner"; ttl_secs = 31536000 : nat32; value = opt "my-passphrase" })' +``` + +That's it: `https://.icp0.io/` now redirects strangers to +`/login.html`, and visitors who present `my-passphrase` get in. + +> **Enable before the first sync for a brand-new private app.** Enabling on an empty +> canister, *then* syncing, means there is never a window where your assets are +> world-readable. Enabling on an already-public app necessarily had a public period. + +## Managing tokens + +| Method | Call | Effect | +|---|---|---| +| Issue | `issue_token '(record { label = "