diff --git a/.agents/skills/bump-version/SKILL.md b/.agents/skills/bump-version/SKILL.md index 640105e..cfab96e 100644 --- a/.agents/skills/bump-version/SKILL.md +++ b/.agents/skills/bump-version/SKILL.md @@ -1,14 +1,14 @@ --- name: bump-version description: > - Set the laud version across every package.json and promote the CHANGES.md + Set the ailoud version across every package.json and promote the CHANGES.md Development section to a released Version section, then create the release commit. Does not tag or push. --- # bump-version -Bump the laud version and prepare the release commit. Tagging and pushing +Bump the ailoud version and prepare the release commit. Tagging and pushing remain separate, deliberate steps. ## Steps @@ -44,7 +44,7 @@ Run `git diff` and confirm the version bump touched exactly these files: `## Development`). No other files changed. There is no `Cargo.toml` and no -`tauri.conf.json` in this project -- laud is a pure TypeScript workspace. +`tauri.conf.json` in this project -- ailoud is a pure TypeScript workspace. ### 4. Create the release commit diff --git a/.agents/skills/check-dependencies/SKILL.md b/.agents/skills/check-dependencies/SKILL.md new file mode 100644 index 0000000..cdaf2b2 --- /dev/null +++ b/.agents/skills/check-dependencies/SKILL.md @@ -0,0 +1,141 @@ +--- +name: check-dependencies +description: > + Audit npm dependencies for known vulnerabilities, report who asks for + funding, and update what is behind -- refusing any version published less + than 14 days ago unless it fixes a critical advisory. Run before every + release and after any dependency change. +--- + +# check-dependencies + +Three questions, in this order: is anything we depend on known to be +vulnerable, who is asking to be paid for it, and what should be updated. +Updating comes last because the first two decide what an update is for. + +## Steps + +1. **Look for known vulnerabilities.** + + ```bash + pnpm audit # everything, including the dev toolchain + pnpm audit --prod # only what the three published packages ship + ``` + + Read them separately. A high-severity advisory in a test runner is a + different problem from one in `commander`: the first cannot reach a user, + the second is installed on their machine. Fix the `--prod` findings first + and say which list a finding came from. + +2. **Report funding.** + + ```bash + npm fund + ``` + + `pnpm` has no `fund` command; `npm fund` reads the installed tree and works + in this workspace. The list is long and almost all of it is the dev + toolchain. What is worth surfacing is the intersection with what actually + ships: + + ```bash + npm fund --json | node -e ' + let s = ""; + process.stdin.on("data", (d) => (s += d)).on("end", () => { + const names = []; + const rec = (n) => { + for (const [name, child] of Object.entries(n.dependencies ?? {})) { + names.push(name); + rec(child); + } + }; + rec(JSON.parse(s)); + const runtime = new Set(); + for (const p of ["packages/core", "packages/providers", "apps/cli"]) { + Object.keys(require(`./${p}/package.json`).dependencies ?? {}).forEach((k) => + runtime.add(k), + ); + } + console.log(names.filter((n) => runtime.has(n)).join(", ") || "(none)"); + })' + ``` + + Report the count and that intersection. Do not open funding pages or + install anything. + +3. **See what is behind.** + + ```bash + pnpm outdated -r + ``` + + Nothing here is urgent by default. A version bump is worth taking when it + fixes something this project hits, closes an advisory from step 1, or keeps + a major version from drifting far enough to become a project of its own. + "It is newer" is not a reason. + +4. **Apply the 14-day rule BEFORE updating anything.** + + ```bash + node scripts/check-dependency-age.mjs + ``` + + A compromised release is discovered by other people, and that takes days -- + every npm supply-chain incident in recent memory was caught within a week + or two of publication, after everyone who upgraded immediately had already + installed it. Waiting costs nothing: there is no urgency in a patch that + has been out for two weeks that was not there on day one. + + So a candidate version younger than 14 days is not taken. Pin the previous + one and come back to it. + + **Unless it fixes a critical or high advisory.** Then waiting is the worse + risk -- a known exploit beats an unaudited release. Record the decision in + `scripts/dependency-age-exceptions.json` with the advisory ID: + + ```json + { + "some-package@2.0.1": "fixes GHSA-xxxx-yyyy-zzzz (critical), 2026-09-05" + } + ``` + + The exception is a statement about one moment, so the check reports entries + that have aged out and should be deleted. Never add one to silence the + check for convenience; if there is no advisory ID, there is no exception. + +5. **Update carefully.** + + One concern per commit, so a regression is bisectable: + + ```bash + pnpm update --latest # or edit the manifest and pnpm install + ``` + + - Keep versions **exact**. This project pins them; a range hands the + decision to whatever resolved last, and the age check cannot judge it. + - Run the `check-licenses` skill if any `package.json` changed -- a new + version can change its licence. + - Run the `run-tests-and-linters` skill. A dependency update that passes + nothing is not an update. + - A user-visible consequence goes in CHANGES.md; a toolchain bump does not. + +6. **Verify the result.** + + ```bash + pnpm audit --prod + node scripts/check-dependency-age.mjs + ``` + + Both must pass. Report what changed, what was deliberately left behind and + why, and any advisory that remains open with the reason it cannot be closed + yet. + +## Do not + +- Do not run `pnpm audit --fix` or `npm audit fix`. They resolve to whatever + is newest, which is the version the 14-day rule exists to refuse. +- Do not update a major version as part of a release. It is its own change, + with its own testing. +- Do not treat a clean `pnpm audit` as proof of anything beyond "no advisory + has been published". Most of what the age rule protects against has no + advisory yet. diff --git a/.agents/skills/check-docs/SKILL.md b/.agents/skills/check-docs/SKILL.md index 2cc06fb..63c1310 100644 --- a/.agents/skills/check-docs/SKILL.md +++ b/.agents/skills/check-docs/SKILL.md @@ -22,11 +22,17 @@ release, or right after adding or changing a CLI command or option. - Confirm every `pnpm` command shown in README.md exists as a script in the root `package.json` (`pnpm lint`, `pnpm typecheck`, `pnpm test:cov`, `pnpm build`, `pnpm format:check`, `pnpm test:e2e`, etc.). - - Confirm every `laud` CLI command shown (`import`, `transcribe`, `ls`, - `show`, `doctor`) is a command M1 actually ships, per the "Project - Overview" section of AGENTS.md. `laud` has no `search`, `collection`, - `tag`, `summarize`, `export`, or `config` command yet; flag any of - those names if they appear in README.md or AGENTS.md. + - Confirm every `ailoud` command shown really exists, by asking the + BINARY rather than any prose: + `node apps/cli/dist/bin/ailoud.js --help` after `pnpm build`, then + `--help` on each group and verb. AGENTS.md says outright not to trust a + list of commands in prose over the binary, and this skill used to name a + fixed set of commands and assert that `search` and `summarize` did not + exist -- both had shipped long before anyone noticed the skill was + telling agents otherwise. + Note that zsh does NOT word-split an unquoted variable, so `$cmd --help` + with `cmd="audio ls"` passes one argument and commander prints the + TOP-LEVEL help, making every subcommand look like it has no flags. - Confirm every relative link in README.md and AGENTS.md resolves to a file that exists in the repository. Links into `.superpowers/` are a defect: that directory is git-ignored and absent from a fresh clone. @@ -45,7 +51,13 @@ release, or right after adding or changing a CLI command or option. 5. **Check version references.** Search `README.md`, `AGENTS.md` and `CONTRIBUTING.md` for version strings. Any hardcoded version must match the version in root - `package.json`. This project ships no documentation site. + `package.json`. + + This project DOES ship a documentation site: `docs/` built with mkdocs and + published per release to `gh-pages`. Build it with `pnpm docs:build` + (strict) and check the rendering with `node scripts/check-docs-render.mjs` + -- the strict build cannot see an admonition whose content fell outside its + box, which has shipped here before. 6. **Check CHANGES.md structure.** Confirm the file starts with a `## Development` section and that diff --git a/.agents/skills/check-fixtures/SKILL.md b/.agents/skills/check-fixtures/SKILL.md index 48451bd..6d27d26 100644 --- a/.agents/skills/check-fixtures/SKILL.md +++ b/.agents/skills/check-fixtures/SKILL.md @@ -1,7 +1,7 @@ --- name: check-fixtures description: > - Drive the built laud binary against fixtures/ end to end -- import, + Drive the built ailoud binary against fixtures/ end to end -- import, transcribe, ls, show, and doctor -- in a throwaway HOME, XDG_CONFIG_HOME, and XDG_DATA_HOME, and confirm the working tree stays clean afterward. --- @@ -10,7 +10,7 @@ description: > The unit tests cover the domain core against fakes (`MemFs`, `FakeClock`, `FakeIds`, `FakeStt`). This skill covers the layer they cannot: the real -`laud` binary, against real audio, writing to a real filesystem (inside a +`ailoud` binary, against real audio, writing to a real filesystem (inside a sandbox). It is the only check that would catch a regression living in the wiring between the CLI, the providers, and the filesystem -- for example a provider writing to the wrong data directory, or a pipeline that behaves @@ -28,14 +28,14 @@ right lens" below. Three short fixtures, each with a reference transcript: an English clip, a Russian clip, and a clip that mixes both languages. The suite drives: -- `laud doctor` against a sandbox with no config, and again after +- `ailoud doctor` against a sandbox with no config, and again after configuring the model. -- `laud import` against a fixture file, including the "already present" +- `ailoud import` against a fixture file, including the "already present" path on a repeat import. -- `laud transcribe`, checked by word error rate against the reference +- `ailoud transcribe`, checked by word error rate against the reference transcript rather than exact string equality -- a model or quantization change shifts wording by a word or two without being a regression. -- `laud show` in both `srt` and `json` formats, plus its error paths (a +- `ailoud show` in both `srt` and `json` formats, plus its error paths (a missing id, an unsupported `--format`). ## Isolation @@ -44,7 +44,7 @@ The suite must never touch the developer's machine state. Every invocation of the built binary sets all three of: - `XDG_CONFIG_HOME`, which relocates `config.yaml`. -- `XDG_DATA_HOME`, which relocates `laud.db` and the `media/` tree. +- `XDG_DATA_HOME`, which relocates `ailoud.db` and the `media/` tree. - `HOME`, so nothing the process resolves relative to the real home directory (for example a fallback default when an XDG variable is unset) can reach outside the sandbox. diff --git a/.agents/skills/check-licenses/SKILL.md b/.agents/skills/check-licenses/SKILL.md index e6bfe48..741aac9 100644 --- a/.agents/skills/check-licenses/SKILL.md +++ b/.agents/skills/check-licenses/SKILL.md @@ -11,7 +11,7 @@ description: > Verify that every direct npm dependency is license-compatible with Apache-2.0 and keep the Third-Party Notices section of `LICENSE` up to -date. `laud` is a pure TypeScript pnpm workspace -- there is no cargo +date. `ailoud` is a pure TypeScript pnpm workspace -- there is no cargo workspace to check, unlike the source repository this project inherits its conventions from. @@ -71,7 +71,7 @@ dependencies` subsection's table -- replace from its `| Package |` | | npm | | | | ``` - Every row's Ecosystem column is `npm` -- laud has no other dependency + Every row's Ecosystem column is `npm` -- ailoud has no other dependency ecosystem. List rows alphabetically by package name. Preserve everything above the table verbatim: the Apache 2.0 license text, the `## Third-Party Notices` heading, and the intro paragraph above the diff --git a/.agents/skills/check-migrations/SKILL.md b/.agents/skills/check-migrations/SKILL.md new file mode 100644 index 0000000..85496fe --- /dev/null +++ b/.agents/skills/check-migrations/SKILL.md @@ -0,0 +1,130 @@ +--- +name: check-migrations +description: > + After touching packages/core/src/db/schema.ts -- verify a new migration + carries a lock entry and an up-to-date snapshot, and judge the one thing + the guard tests cannot see: whether an edited migration has already + shipped. +--- + +# check-migrations + +`MIGRATIONS` in `packages/core/src/db/schema.ts` is append-only: every entry +that has reached a released version is a promise to every database that has +already run it. Two tests in +`packages/providers/src/store/schemaGuard.test.ts` enforce most of that +promise mechanically. This skill runs them, reads what they show you, and +judges the part they structurally cannot: whether a migration you are +looking at has shipped. + +## When to use it + +After any change to `packages/core/src/db/schema.ts` -- a new migration, or +an edit to an existing one -- before it goes into a commit. + +## What the guard cannot see + +A test can hash a migration and compare it to a database dump. Neither can +tell you when the migration was written. That is the whole reason this is a +skill and not a third test: + +- **Editing a migration that has not shipped yet is fine.** If it was added + in this branch and nothing has run it, changing it is just editing a + patch before it lands -- update the lock and the snapshot together with + the edit, in the same commit. +- **Editing a migration that has already shipped is not fine**, no matter + how small the change. A database that ran it at an older version will + never run it again, so an edited version means two machines at the same + `user_version` with different schemas -- and neither the fingerprint test + nor the snapshot test can tell you this from the file alone, because both + only ever see the current content of `schema.ts`. +- **Deleting a shipped migration is the same hazard, not a different one.** + A database that already ran it does not forget; removing its entry from + `MIGRATIONS` just means the promise that entry made is no longer written + down anywhere for the next person to honor. Treat "remove this migration" + the same as "edit this migration": fine if it never shipped, never fine if + it did. +- **Reordering the `MIGRATIONS` array is the same hazard again**, even with + every `version` field left untouched -- a migration's position in the + array is not what makes it safe to change, its `version` having already + run somewhere is. (The lock-completeness test happens to catch a reorder + today, because it compares sorted lock keys against `MIGRATIONS` in + array order, but that is not the same as the skill telling you to look + for it.) +- **Only git history tells the two apart.** Before touching an existing + migration entry, run: + + ```bash + git log --oneline -- packages/core/src/db/schema.ts + ``` + + and check whether the commit that introduced this migration's `version` + has been released (tagged, merged to `main` and shipped) or is still + local to the current branch. If it has shipped, the change belongs in a + **new** migration with the next version number, never as an edit to the + old one -- even to fix a typo in a column name. + +## Steps + +1. **Run the two tests.** + + ```bash + pnpm build + NODE_OPTIONS=--disable-warning=ExperimentalWarning pnpm vitest run packages/providers/src/store/schemaGuard.test.ts + ``` + +2. **If "has a lock entry for every migration, and no extras" fails:** + A migration was added or removed without regenerating the lock. Run: + + ```bash + node scripts/write-schema-snapshot.mjs + ``` + + then re-run the test. This is safe whenever the mismatch is a missing or + extra entry -- it is never safe to run this just because the fingerprint + test below is failing on a migration that has already shipped. + +3. **If "still matches the fingerprint of every shipped migration" fails:** + Read which migration's fingerprint changed and check git history per the + section above. + - Unshipped: run `node scripts/write-schema-snapshot.mjs` to update the + lock, and continue. + - Shipped: do not regenerate anything. Revert the edit to that migration + and put the intended change in a new migration instead. + +4. **If "produces exactly the snapshotted schema" fails:** read the diff + Vitest prints between the snapshotted schema and the one the migrations + now produce. Confirm the diff is exactly the change you intended -- a new + table, an added column, an added trigger -- and nothing incidental (for + example a different SQLite version rewriting an unrelated table's + `CREATE` text). Once confirmed, run: + + ```bash + node scripts/write-schema-snapshot.mjs + ``` + + and re-run the test. + +5. **For a genuinely new migration, confirm the comment.** Every existing + migration explains, in a comment above its statements, why the table or + column exists rather than merely what it is. Read the new migration and + confirm it carries one too -- the guard tests cannot check this, because + the comment lives outside the hashed statement strings by design (see + the comment on `fingerprint` in `schemaGuard.test.ts`). + +6. **Re-run the whole gate.** `schemaGuard.test.ts` is one file among many; + finish with the `run-tests-and-linters` skill before calling the change + done. + +## Report + +``` +schemaGuard.test.ts: PASS / FAIL +lock regenerated: yes / no +snapshot regenerated: yes / no +new migration comment: present / missing +shipped migration edited: yes (blocking) / no +``` + +If a shipped migration was edited, say so plainly and do not propose +regenerating the lock as the fix -- the fix is a new migration. diff --git a/.agents/skills/dev-tag/SKILL.md b/.agents/skills/dev-tag/SKILL.md new file mode 100644 index 0000000..f0d248a --- /dev/null +++ b/.agents/skills/dev-tag/SKILL.md @@ -0,0 +1,103 @@ +--- +name: dev-tag +description: > + Cut a throwaway development tag (`v-dev.`) to publish a snapshot + to npm under the `dev` dist-tag and exercise the release pipeline. Publishes + no documentation and never moves `latest`. +--- + +# dev-tag + +A development tag publishes a snapshot nobody will get by accident, so a real +install of real code can be tried before a release is promised. + +## The three kinds of tag + +| Tag | Cut from | npm dist-tag | Docs published | +| -------------- | ---------- | ------------ | -------------- | +| `v1.2.3-dev.1` | any branch | `dev` | no | +| `v1.2.3-rc.1` | `develop` | `next` | no | +| `v1.2.3` | `main` | `latest` | yes | + +Only a final tag moves `latest` and publishes documentation. `npm install +ailoud` therefore never picks up a dev tag, and the site never describes a +version nobody can install. + +Except once per package: npm sets `latest` on a FIRST publish whatever `--tag` +says, and `latest` cannot be removed afterwards, only moved. If a dev tag is +what introduces a package to the registry -- as v1.0.0-dev.1 was -- then +`latest` points at that snapshot until a final version is published. + +## Cutting one + +1. The working tree must be clean and the gate green: + + ``` + pnpm build && pnpm format:check && pnpm lint && pnpm typecheck && pnpm test:cov + ``` + +2. Set the version across the manifests. `-dev.` is a pre-release, so it + sorts BELOW the release it precedes: + + ``` + node scripts/bump-version.mjs 1.2.3-dev.1 + ``` + + `bump-version` promotes the CHANGES.md `## Development` section, which a dev + tag does not want -- so for a dev tag, set the versions by hand and leave + the changelog alone: + + ``` + node -e "for (const p of ['package.json','packages/core/package.json','packages/providers/package.json','apps/cli/package.json']) { const fs=require('fs'); const d=JSON.parse(fs.readFileSync(p,'utf8')); d.version='1.2.3-dev.1'; fs.writeFileSync(p, JSON.stringify(d,null,2)+'\n'); }" + ``` + +3. Commit, tag and push. The publish workflow checks that every manifest + agrees with the tag, so these cannot drift: + + ``` + git commit -am "chore: 1.2.3-dev.1" + git tag -s v1.2.3-dev.1 -m "v1.2.3-dev.1" + git push origin HEAD --tags + ``` + +4. Try it from the registry, which is the whole point: + + ``` + npm install -g ailoud@dev + ailoud --version + ``` + +## What it does not do + +- It does not publish documentation. `docs.yml` starts only on a final tag and + refuses a pre-release twice over. +- It does not move `latest`. +- It is not a release candidate. An `-rc.` tag says "this is what the release + will be"; a `-dev.` tag says "this is a snapshot to try". + +## Cleaning up + +A dev version cannot be unpublished after 72 hours and its number can never be +reused, so use a fresh `-dev.` each time rather than retrying one. + +Once the final release is out, retire every snapshot it supersedes in one step. +**By hand, and nothing will remind you** -- the release workflow cannot do it, +and a release looks finished without it: + +``` +pnpm retire 1.2.3 # prints the plan +NPM_TOKEN=npm_... pnpm retire 1.2.3 --yes # carries it out +``` + +`NPM_TOKEN` is a granular access token with read-and-write on the three +packages, and it is required for `--yes`: without one npm asks for a 2FA code +on each of the twelve writes, or waits on an interactive login. The plan needs +no credential. + +It deprecates the versions rather than unpublishing them, moves the `dev` +dist-tag onto the release rather than removing it, and deletes the tags -- but +only those whose commit is reachable from `main`. The provenance of a published package names both the commit and the tag +it was built from: delete the tag and the name stops resolving, which costs +convenience; delete a tag holding the only reference to its commit and the +commit itself can be collected, which costs the attestation its subject. Tags +in the second case are reported and left alone. diff --git a/.agents/skills/pre-release-check/SKILL.md b/.agents/skills/pre-release-check/SKILL.md index d6dcaad..7cfc65e 100644 --- a/.agents/skills/pre-release-check/SKILL.md +++ b/.agents/skills/pre-release-check/SKILL.md @@ -1,10 +1,10 @@ --- name: pre-release-check description: > - Gate a release by running check-licenses, run-tests-and-linters, - check-fixtures, check-docs, and check-changes, plus verifying the version - bump and that all commits since the last release follow the - conventional-commits format. + Gate a release by running check-licenses, check-dependencies, + run-tests-and-linters, check-fixtures, check-docs, and check-changes, plus + verifying the version bump and that all commits since the last release + follow the conventional-commits format. --- # pre-release-check @@ -21,8 +21,11 @@ details): 1. **check-licenses** -- must run first because a license failure is the most fundamental blocker. -2. **run-tests-and-linters** -- lint, typecheck, and coverage at 90%. -3. **check-fixtures** -- drive the built binary against `fixtures/` end to +2. **check-dependencies** -- advisories, funding, and the 14-day rule on any + version taken. Runs early: an update it recommends changes what everything + below is testing, so taking one afterwards invalidates the whole gate. +3. **run-tests-and-linters** -- lint, typecheck, and coverage at 90%. +4. **check-fixtures** -- drive the built binary against `fixtures/` end to end. This is the only check that exercises the real binary against real files, so it catches wiring regressions the unit tests (which run against in-memory fakes) cannot see. Six of the twelve specs need a real @@ -31,12 +34,14 @@ details): not a release blocker on its own -- attribute every failure (missing whisper-cli, fixture drift, product change, harness defect) before deciding whether it blocks. -4. **check-docs** -- README.md, cross-references, command accuracy, +5. **check-docs** -- README.md, cross-references, command accuracy, version references. -5. **check-changes** -- CHANGES.md Development section vs. commit history. +6. **check-changes** -- CHANGES.md Development section vs. commit history. -If check-licenses or run-tests-and-linters fails, report the failure and -stop. If check-fixtures fails for a reason other than the missing +If check-licenses, check-dependencies or run-tests-and-linters fails, report +the failure and stop. A `--prod` advisory or a version younger than 14 days +blocks a release: the first ships a known vulnerability, and the second ships +a version nobody has had time to find one in. If check-fixtures fails for a reason other than the missing `whisper-cli` binary (fixture drift, a product change, or a harness defect), treat it the same way -- a release must not ship while tests, licenses, or the end-to-end run are failing for a reason within this @@ -93,6 +98,7 @@ Produce a release-readiness summary: ``` check-licenses: PASS / FAIL +check-dependencies: PASS / FAIL (advisories: prod / dev only; ages: ok / N too young) run-tests-and-linters: PASS / FAIL check-fixtures: PASS / FAIL (attribute: missing whisper-cli / fixture drift / product change / harness defect) check-docs: PASS / FAIL @@ -108,3 +114,18 @@ If the gate is PASS, confirm: "All checks passed. The release may proceed." If the gate is FAIL, list every blocking issue with enough detail for the developer to act immediately. Do not tag or publish a release while any required check is failing. + +## After the release lands + +The gate ends at the tag. One step remains that no workflow performs, and a +release looks complete without it -- so report it as outstanding whenever this +skill runs for a final version: + +``` +pnpm retire # prints the plan +NPM_TOKEN=npm_... pnpm retire --yes # carries it out +``` + +It deprecates the `-dev.*` and `-rc.*` snapshots on npm, moves the `dev` dist-tag +onto the release, and deletes the superseded tags. Until it runs, someone +installing `ailoud@dev` gets an older build than `ailoud`. diff --git a/.agents/skills/run-tests-and-linters/SKILL.md b/.agents/skills/run-tests-and-linters/SKILL.md index 3b42bdb..685906d 100644 --- a/.agents/skills/run-tests-and-linters/SKILL.md +++ b/.agents/skills/run-tests-and-linters/SKILL.md @@ -9,7 +9,7 @@ description: > # run-tests-and-linters -Run the full quality gate for laud before marking any task done. `laud` is +Run the full quality gate for ailoud before marking any task done. `ailoud` is a pure TypeScript pnpm workspace, so the gate has no Rust or native half to run alongside it. diff --git a/.gitattributes b/.gitattributes index 4ac1b58..f92e46c 100644 --- a/.gitattributes +++ b/.gitattributes @@ -5,4 +5,13 @@ *.mp3 filter=lfs diff=lfs merge=lfs -text *.m4a filter=lfs diff=lfs merge=lfs -text +# Video fixtures: `ailoud audio import` accepts video because a meeting +# recording usually is one, and the four containers it maps to a MIME type each +# get one. They are tiny (under 15 kB) but binary, so LFS keeps them out of the +# packfile like the audio. +*.mp4 filter=lfs diff=lfs merge=lfs -text +*.mov filter=lfs diff=lfs merge=lfs -text +*.mkv filter=lfs diff=lfs merge=lfs -text +*.webm filter=lfs diff=lfs merge=lfs -text + pnpm-lock.yaml linguist-generated=true diff --git a/.github/actions/setup-node-pnpm/action.yml b/.github/actions/setup-node-pnpm/action.yml new file mode 100644 index 0000000..3ac6725 --- /dev/null +++ b/.github/actions/setup-node-pnpm/action.yml @@ -0,0 +1,38 @@ +# The Node and pnpm setup every job shares, in one place because it is four +# jobs across two workflows and the corepack part needs care. +# +# `corepack enable` only writes shims; the pinned pnpm is downloaded lazily by +# whatever invokes it first, which used to be setup-node's cache probe. That +# download has failed there -- an undici assertion inside Node 24.20.0 when the +# connection ends mid-response -- and a failure inside another action's step +# cannot be retried. Downloading it here instead puts the network call in a +# step of our own, so a flake costs five seconds rather than the run. + +name: Set up Node.js and pnpm +description: Activate the pinned pnpm, then Node 24 with the pnpm store cached. + +runs: + using: composite + steps: + - name: Activate the pnpm pinned in package.json + shell: bash + env: + # Without this corepack prints a download notice that GitHub renders as + # an error annotation on a run that is fine. + COREPACK_ENABLE_DOWNLOAD_PROMPT: '0' + run: | + corepack enable + for attempt in 1 2 3; do + if corepack install; then break; fi + echo "corepack install failed (attempt ${attempt} of 3); retrying in 5s" + sleep 5 + done + # Fails the step if all three attempts did, rather than leaving the + # next job step to fail with something less specific. + pnpm --version + + - name: Set up Node.js 24 with pnpm cache + uses: actions/setup-node@v5 + with: + node-version: '24' + cache: pnpm diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml new file mode 100644 index 0000000..49afe3e --- /dev/null +++ b/.github/codeql/codeql-config.yml @@ -0,0 +1,32 @@ +# CodeQL configuration for AILoud. +# +# Queries are named here rather than in the workflow's `queries:` input so the +# reason for each exclusion can sit next to it. + +name: AILoud + +queries: + - uses: security-and-quality + +query-filters: + # js/file-system-race, three times over, always on apps/cli/src/setupLock.ts. + # + # That file's whole job is to be the thing a file-system race goes through. + # It acquires with `open(path, 'wx')` -- one atomic syscall -- and takes over + # a stale lock under a SECOND lock file (`provisioning.lock.steal`, also + # `wx`), which is what makes the takeover exclusive. The analysis sees a read + # of one path followed by a write to it and reports a race; it cannot see + # that the exclusion is held by a different file. + # + # Two of the three reports were nonetheless real, and both are fixed: the + # first version deleted a live lock, the second let two runs each read back + # their own pid and proceed. What stands in for this query now is a + # concurrency test (`setupLock.test.ts`, "under contention") that runs two + # processes against the compiled lock and fails on either old implementation + # -- an actual measurement rather than a pattern match. + # + # Excluded by id, so the rule is off everywhere. It has produced nothing but + # this file and test code in this repository; if that changes, the honest fix + # is to narrow this with a `paths` filter rather than keep the noise. + - exclude: + id: js/file-system-race diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..480e7d6 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,59 @@ +# Dependabot, held to the same rule as a human: nothing younger than 14 days. +# +# `cooldown` is what makes that true. Without it Dependabot opens a pull +# request the moment a version appears, which is precisely the window +# `scripts/check-dependency-age.mjs` exists to refuse -- the check would then +# fail on Dependabot's own branch and the two would be arguing. +# +# Security updates deliberately ignore cooldown, and that is the behaviour we +# want: a known advisory beats an unaudited release, which is the same +# exception `scripts/dependency-age-exceptions.json` records for a human. +# +# See .agents/skills/check-dependencies/SKILL.md for the reasoning, and +# AGENTS.md for how a dependency change reaches a release. + +version: 2 + +updates: + - package-ecosystem: npm + # The workspace root: one pnpm-lock.yaml covers all four manifests. + directory: / + schedule: + interval: weekly + day: monday + cooldown: + default-days: 14 + # Exact pins are this project's convention, and `increase` preserves them. + # `widen` would turn a pin into a range and hand the choice of version to + # whatever resolved last, which no age check can judge. + versioning-strategy: increase + open-pull-requests-limit: 5 + groups: + # The dev toolchain moves constantly and cannot reach a user, so it + # arrives as one reviewable pull request. Anything that ships gets its + # own, because a runtime dependency deserves its own reading. + dev-toolchain: + dependency-type: development + ignore: + # typescript-eslint 8.67.0 refuses to load against TS 7.0 outright + # ("typescript-eslint does not support TS 7.0"), so the bump Dependabot + # opened could not lint -- it is blocked by a peer, not by us. Tracked as + # typescript-eslint#10940; drop this entry when it supports TS >= 7.1. + - dependency-name: typescript + update-types: ['version-update:semver-major'] + labels: + - dependencies + + - package-ecosystem: github-actions + directory: / + schedule: + interval: weekly + day: monday + cooldown: + default-days: 14 + groups: + actions: + patterns: + - '*' + labels: + - dependencies diff --git a/.github/workflows/backmerge.yml b/.github/workflows/backmerge.yml index 36a3d5d..e47c8f7 100644 --- a/.github/workflows/backmerge.yml +++ b/.github/workflows/backmerge.yml @@ -27,13 +27,21 @@ jobs: open-backmerge-pr: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v5 + - uses: actions/checkout@v7 with: fetch-depth: 0 - name: Skip if develop already contains main id: check run: | + set -uo pipefail + # Said plainly rather than dying on `fatal: Not a valid object name`, + # which is what the first run did before develop existed. + if ! git rev-parse --verify --quiet origin/develop >/dev/null; then + echo "needs_pr=false" >> "$GITHUB_OUTPUT" + echo "::notice::there is no develop branch, so there is nothing to back-merge into." + exit 0 + fi if git merge-base --is-ancestor origin/main origin/develop; then echo "needs_pr=false" >> "$GITHUB_OUTPUT" echo "develop already contains main; nothing to do." diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8e8bfca..0efcdb4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -46,18 +46,12 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v7 with: lfs: true - - name: Enable pnpm via corepack - run: corepack enable - - - name: Set up Node.js 24 with pnpm cache - uses: actions/setup-node@v5 - with: - node-version: '24' - cache: pnpm + - name: Set up Node.js 24 and pnpm + uses: ./.github/actions/setup-node-pnpm - name: Install ffmpeg # packages/providers/src/audio/ffmpeg.test.ts spawns the real binary to @@ -96,18 +90,12 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v7 with: lfs: true - - name: Enable pnpm via corepack - run: corepack enable - - - name: Set up Node.js 24 with pnpm cache - uses: actions/setup-node@v5 - with: - node-version: '24' - cache: pnpm + - name: Set up Node.js 24 and pnpm + uses: ./.github/actions/setup-node-pnpm - name: Install dependencies (frozen lockfile) run: pnpm install --frozen-lockfile @@ -128,20 +116,14 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v7 with: # The audio fixtures are LFS-tracked; without this the suite # transcribes pointer files. lfs: true - - name: Enable pnpm via corepack - run: corepack enable - - - name: Set up Node.js 24 with pnpm cache - uses: actions/setup-node@v5 - with: - node-version: '24' - cache: pnpm + - name: Set up Node.js 24 and pnpm + uses: ./.github/actions/setup-node-pnpm - name: Install dependencies (frozen lockfile) run: pnpm install --frozen-lockfile @@ -149,20 +131,79 @@ jobs: - name: Cache the whisper models # Keyed on the catalogue, which holds the pinned file names, sizes and # URLs: a model change invalidates the cache, and nothing else does. - uses: actions/cache@v4 + uses: actions/cache@v6 with: path: ~/.local/share/ailoud/models key: ailoud-models-${{ runner.os }}-${{ hashFiles('packages/core/src/provision/catalogue.ts') }} + - name: Dependency age (advisory) + # Reports, never blocks. Dependabot's version updates already respect + # the same 14 days through `cooldown` in .github/dependabot.yml, so the + # two do not argue -- but a Dependabot SECURITY update deliberately + # ignores cooldown, and a fresh version that closes an advisory is the + # one case where taking it beats waiting. Failing the build there would + # make the rule an obstacle to the thing it exists to protect. + # + # Not run on tags: ci.yml does not trigger on them, and the release + # path asks the question through the check-dependencies skill instead. + run: | + if ! node scripts/check-dependency-age.mjs; then + echo "::warning::A direct dependency is younger than 14 days, or its age could not be" + echo "::warning::checked. If it closes a critical advisory, record it in" + echo "::warning::scripts/dependency-age-exceptions.json with the advisory ID; otherwise" + echo "::warning::pin the previous version. See .agents/skills/check-dependencies." + fi + - name: Build run: pnpm build - - name: Provision ffmpeg, whisper.cpp and the models - # The same command a user runs, which makes this a test of `setup` as - # well as a prerequisite for the suite. --llm skip because no spec here - # summarises, and it would otherwise download another 2.1 GB. + - name: Install ffmpeg + # Installed with apt rather than left to `setup`, which correctly + # refuses to run `sudo apt-get` with no terminal to answer a password + # prompt on -- it reports the exact command instead of hanging. There + # is no terminal on a runner, so the sudo step is ours to do. + run: sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg + + - name: Provision whisper.cpp and the models + # Everything that needs no sudo: the whisper release and the model + # files. Still the command a user runs, so this remains a test of + # `setup` and not only a prerequisite for the suite. --llm skip because + # no spec here summarises, and it would otherwise fetch another 2.1 GB. run: node apps/cli/dist/bin/ailoud.js setup --yes --llm skip + - name: Put the provisioned binaries on PATH + # The e2e sandbox writes its own config naming only the MODEL, and + # leaves `binary` at its default -- which means whisper-cli has to be + # found on PATH. `setup` installs it under the data directory and + # records the absolute path in the user's config instead, so doctor was + # entirely green while every transcribe spec exited 3. + # + # The directories are read back out of that config rather than spelled + # here, so bumping a pinned release does not silently break this. + run: | + set -euo pipefail + read_dir() { + node -e " + const { parseConfig } = require('./apps/cli/dist/config.js'); + const { readFileSync } = require('node:fs'); + const { homedir } = require('node:os'); + const { dirname } = require('node:path'); + const config = parseConfig( + readFileSync(homedir() + '/.config/ailoud/config.yaml', 'utf8'), + ); + const value = $1; + if (value && value.includes('/')) console.log(dirname(value)); + " + } + for dir in \ + "$(read_dir 'config.stt.whisperCpp.binary')" \ + "$(read_dir 'config.stt.diarization.binary')"; do + if [ -n "$dir" ]; then + echo "adding $dir to PATH" + echo "$dir" >> "$GITHUB_PATH" + fi + done + - name: Report what the machine now has # Printed whether or not the suite passes: a red e2e run is far quicker # to read next to doctor's account of what was actually installed. @@ -178,7 +219,7 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v7 - name: Install uv uses: astral-sh/setup-uv@v7 diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000..c14bbe5 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,51 @@ +# CodeQL, GitHub's static analysis, over the TypeScript in this workspace. +# +# Committed rather than enabled through the repository's default setup, for the +# same reason every other check here is a file: what runs, when, and over what +# should be reviewable in a diff. +# +# The weekly run matters as much as the per-push one -- most findings arrive +# because the queries improved, not because the code changed. + +name: CodeQL + +on: + push: + branches: [main, develop] + pull_request: + schedule: + # Monday morning, after Dependabot has opened whatever it is going to. + - cron: '30 6 * * 1' + +concurrency: + group: codeql-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +jobs: + analyze: + name: Analyze TypeScript + runs-on: ubuntu-latest + timeout-minutes: 30 + + permissions: + # security-events: write is how the results reach the Security tab; a run + # without it analyses the code and then throws the answer away. + security-events: write + contents: read + + steps: + - name: Checkout repository + uses: actions/checkout@v7 + + - name: Initialize CodeQL + uses: github/codeql-action/init@v4 + with: + languages: javascript-typescript + # The query set and the one rule this project excludes live in the + # config file, where each can carry its reason. security-and-quality + # over the default security-extended: this is a small codebase where + # the quality queries are worth reading rather than noise to filter. + config-file: .github/codeql/codeql-config.yml + + - name: Analyze + uses: github/codeql-action/analyze@v4 diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 2b17b60..03d7856 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -29,12 +29,13 @@ name: Docs on: - push: - tags: - # Final releases only: the negative pattern excludes any pre-release tag - # (a `-` qualifier), so an RC never starts this workflow at all. - - 'v*' - - '!v*-*' + # After a successful publish, not beside it. On the tag push both workflows + # started at once, so a publish that then refused -- or failed halfway + # through its three packages -- left the site advertising a version npm did + # not have. `workflow_run` waits for the answer. + workflow_run: + workflows: [Publish] + types: [completed] workflow_dispatch: # mike pushes commits to the gh-pages branch. @@ -50,11 +51,22 @@ jobs: deploy: name: Build and deploy versioned docs runs-on: ubuntu-latest + timeout-minutes: 20 + # A publish that failed publishes no documentation. Whether the commit is a + # final release is decided in the job, from the manifest in the commit + # itself -- `head_branch` on a workflow_run is documented as a branch, and + # a release must not rest on what it happens to hold for a tag. + if: >- + ${{ github.event_name == 'workflow_dispatch' || + github.event.workflow_run.conclusion == 'success' }} steps: - - name: Checkout repository - uses: actions/checkout@v5 + - name: Checkout the commit that was published + uses: actions/checkout@v7 with: + # The SHA, which is unambiguous. Without a ref this would check out + # the default branch, which is not necessarily what was published. + ref: ${{ github.event.workflow_run.head_sha || github.ref }} # mike needs full history plus the gh-pages branch it maintains. fetch-depth: 0 @@ -73,15 +85,13 @@ jobs: run: | set -euo pipefail - if [ "${GITHUB_REF_TYPE}" != "tag" ]; then - echo "::error::Docs publish only from a release tag; ${GITHUB_REF_NAME} is not one." - exit 1 - fi - - version="${GITHUB_REF_NAME#v}" - # The tag filter already keeps pre-releases out, but - # workflow_dispatch can be pointed at any ref -- so refuse here too - # rather than trusting the trigger alone. + # The version of the thing that was published, from the commit that + # was published. publish.yml refuses a tag that disagrees with any + # manifest, so this and the tag are the same number by then. + version="$(node -p "require('./package.json').version")" + # publish.yml runs for pre-release tags too, so this workflow starts + # for them and stops here. A pre-release must not move the site: the + # docs describe what `npm install ailoud` gives you. case "${version}" in *-*) echo "${version} is a pre-release; skipping docs publication." diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 226ccf9..354160f 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -43,8 +43,12 @@ on: # id-token is what makes trusted publishing work; without it npm has no OIDC # token to exchange and falls back to asking for a credential that is not there. permissions: - contents: read + # contents: write is for creating the GitHub release. + contents: write id-token: write + # Reading code scanning results, to refuse a release that ships a finding + # nobody has looked at. + security-events: read concurrency: group: npm-publish @@ -56,22 +60,29 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 30 + env: + # The tag being released, resolved once. `github.ref_name` is the tag on + # a tag push but the BRANCH on a workflow_dispatch, so a step that let it + # default -- the changelog check did -- looked for a "## Version main" + # section and failed a re-run of a perfectly good tag. + TAG: ${{ github.event.inputs.tag || github.ref_name }} + steps: - name: Checkout repository - uses: actions/checkout@v5 + uses: actions/checkout@v7 with: ref: ${{ github.event.inputs.tag || github.ref }} lfs: true - - name: Enable pnpm via corepack - run: corepack enable - - - name: Set up Node.js 24 with pnpm cache - uses: actions/setup-node@v5 - with: - node-version: '24' - cache: pnpm - registry-url: 'https://registry.npmjs.org' + - name: Set up Node.js 24 and pnpm + # Deliberately WITHOUT setup-node's registry-url. Given one, it writes + # an .npmrc containing `_authToken=${NODE_AUTH_TOKEN}`; with no token + # secret to fill it, npm still saw credentials, sent an empty one, and + # never attempted the OIDC exchange -- the registry answered 404 on the + # PUT and the log said nothing about trusted publishing at all. With no + # .npmrc, npm finds it has no credentials and does the exchange. The + # default registry is registry.npmjs.org either way. + uses: ./.github/actions/setup-node-pnpm - name: Require an npm that can do trusted publishing # Trusted publishing landed in npm 11.5.1. An older npm fails by asking @@ -87,15 +98,103 @@ jobs: fi echo "npm $have" + - name: Refuse a final release while NPM_TOKEN is set + # First, not last. This used to live inside the publish step, after + # pnpm install, three checks, apt-get, the whole gate and the + # end-to-end suite -- around twenty minutes to learn a fact known at + # second zero. Nothing was ever published by it, so the cost was only + # time, but the documented order said otherwise and this is the most + # likely way a first production release fails. + # + # The bootstrap is over the moment the packages exist, and a token that + # keeps working is a token nobody gets round to removing. + env: + NPM_TOKEN: ${{ secrets.NPM_TOKEN }} + run: | + set -euo pipefail + version="${TAG#v}" + if [ -z "${NPM_TOKEN:-}" ]; then + echo "no NPM_TOKEN secret; this run will use trusted publishing" + exit 0 + fi + case "$version" in + *-*) + echo "::warning::publishing $version with the NPM_TOKEN secret; attach the trusted publisher and delete the secret" + ;; + *) + echo "::error::$version is a final release and NPM_TOKEN is set." + echo "::error::Attach the trusted publisher on all three package pages" + echo "::error::(organization lorem-dev, repository ailoud, workflow publish.yml," + echo "::error::environment empty), then delete the NPM_TOKEN secret and re-run." + exit 1 + ;; + esac + - name: Install dependencies (frozen lockfile) run: pnpm install --frozen-lockfile + - name: Check the changelog is fit to release + # First, and before the gate: a version number can never be reused and + # the unpublish window is 72 hours, so every reason to refuse should be + # found while refusing is still free. This checks the section exists and + # has entries, that it is inside the hard limit, that nothing is + # stranded under Development, and -- for a final tag -- that its + # pre-release sections were folded. + # + # The scripts' own tests run in ci.yml, on branches and pull requests + # rather than on tags, so a broken script is caught before a release + # depends on it. + run: node scripts/check-changelog.mjs "$TAG" + + - name: Refuse a release carrying unreviewed high-severity findings + # The dependency side of this is already enforced -- a --prod advisory + # or a version younger than 14 days blocks a release -- while what + # CodeQL finds in our own code did not reach here at all. This closes + # that asymmetry. + # + # Only `state=open` counts. A finding that was reviewed and explained + # is `dismissed` with its reason attached, and does not block; one + # nobody has looked at does. Fixing it and dismissing it are both + # answers -- ignoring it is not. + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + # --paginate, because an alert past the first hundred blocked nothing. + # jq -s 'add' folds the pages into one array. + if ! alerts=$(gh api --paginate \ + "repos/${{ github.repository }}/code-scanning/alerts?state=open&per_page=100" \ + | jq -s 'add // [] | [.[] | select(.rule.security_severity_level == "high" or .rule.security_severity_level == "critical")]'); then + echo "::error::could not read code scanning results. A release that cannot check" + echo "::error::whether it ships a known finding is not one to make blind; fix the" + echo "::error::access or the analysis, then re-run." + exit 1 + fi + count=$(jq 'length' <<< "$alerts" || true) + # An empty or malformed answer used to pass: `jq length` printed + # nothing, `[ "" -gt 0 ]` errored, and a failing `if` condition is not + # caught by `set -e`, so the else branch reported "no alerts". + case "$count" in + '' | *[!0-9]*) + echo "::error::could not count the code scanning alerts (got '${count}')." + echo "::error::Refusing rather than assuming there are none." + exit 1 + ;; + esac + if [ "$count" -gt 0 ]; then + jq -r '.[] | "::error::\(.rule.security_severity_level) \(.rule.id) at \(.most_recent_instance.location.path):\(.most_recent_instance.location.start_line)"' <<< "$alerts" + echo "::error::${count} open high or critical code scanning alert(s). Fix them, or" + echo "::error::dismiss each with a reason, then re-run this workflow on the tag." + exit 1 + fi + echo "no open high or critical code scanning alerts" + - name: Check that the tag matches the version in the manifests # A tag that disagrees with package.json publishes a version nobody # asked for, under a tag that points at different code. run: | set -euo pipefail - tag="${{ github.event.inputs.tag || github.ref_name }}" + tag="$TAG" want="${tag#v}" for manifest in packages/core packages/providers apps/cli; do have=$(node -p "require('./$manifest/package.json').version") @@ -106,6 +205,13 @@ jobs: done echo "all three manifests are at $want" + - name: Install ffmpeg + # The gate includes packages/providers/src/audio/ffmpeg.test.ts, which + # spawns the real binary. CI installs it for the same reason; this job + # runs the same gate, so it needs the same tool. Without it the release + # failed here rather than on anything about the release. + run: sudo apt-get update -qq && sudo apt-get install -y -qq ffmpeg + - name: Run the gate # Nothing is published that has not built, linted, typechecked and # passed its tests on this runner. An unpublish window is 72 hours and @@ -130,28 +236,134 @@ jobs: done ls -l dist-npm + - name: Check each tarball carries the licence and no source + # pnpm copies the repository's LICENSE into every workspace tarball, + # which is why no package directory holds its own copy -- three copies + # of a 224-line file would drift. That makes it an invariant worth + # checking rather than assuming: if pnpm ever stops, this fails the + # release instead of publishing an unlicensed package. + # + # The same step guards the other packing mistake: shipping `src` and + # the tests. Before `files` was set, the CLI tarball carried 82 source + # files and 32 test files. + run: | + set -euo pipefail + for tarball in dist-npm/*.tgz; do + echo "--- $tarball" + contents=$(tar -tzf "$tarball") + if ! grep -qiE '^package/LICEN[SC]E' <<< "$contents"; then + echo "::error::$tarball has no LICENSE." + exit 1 + fi + if grep -qE '^package/src/' <<< "$contents"; then + echo "::error::$tarball ships src/; check the \`files\` field." + exit 1 + fi + if grep -qE '\.test\.' <<< "$contents"; then + echo "::error::$tarball ships tests; check the \`files\` field." + exit 1 + fi + # 1.0.0-dev.1 went out with no README, so all three npm pages read + # "This package does not have a README" -- the first thing anyone + # arriving from a search sees. The libraries hold their own; the + # CLI's is the repository's, copied in by its prepack script, which + # is the part that could silently stop working. + if ! grep -qiE '^package/README' <<< "$contents"; then + echo "::error::$tarball has no README." + exit 1 + fi + echo "licence, readme, no source, no tests" + done + - name: Publish, dependencies first # Order matters: the CLI's manifest names exact versions of the two # libraries, so they have to exist first. --provenance records which # workflow and which commit produced each tarball, verifiable from the - # package page. + # package page -- it works under either credential below. + env: + NPM_TOKEN: ${{ secrets.NPM_TOKEN }} run: | set -euo pipefail - tag="${{ github.event.inputs.tag || github.ref_name }}" + tag="$TAG" version="${tag#v}" - # A pre-release goes out under `next`, so `npm install ailoud` keeps - # returning the last stable version until a final tag moves `latest`. + # BOOTSTRAP ONLY. Trusted publishing is configured per package on + # npmjs.com, which cannot be done for a package that does not exist + # yet -- so the first version of each of the three has to go out on a + # token, and npm answers ENEEDAUTH without one however complete the + # OIDC setup is. Once the trusted publisher is attached to all three + # package pages, delete the NPM_TOKEN secret: with it gone this + # branch is skipped and npm does the OIDC exchange instead, which is + # the arrangement with nothing to expire. + # Whether a token is allowed here at all was settled in the first + # step of this job; this only applies it. + if [ -n "${NPM_TOKEN:-}" ]; then + npm config set //registry.npmjs.org/:_authToken "$NPM_TOKEN" + fi + # Three kinds of tag, three destinations. See the dev-tag skill. + # -dev.N a snapshot to try -> dev + # -rc.N what the release will be -> next + # final the release -> latest + # Only the last moves `latest`, so `npm install ailoud` never picks up + # a pre-release by accident -- with one exception nothing here can + # prevent: npm sets `latest` on a package's first publish whatever + # --tag says, and `latest` can be moved but never removed. case "$version" in - *-*) dist_tag=next ;; - *) dist_tag=latest ;; + *-dev.*) dist_tag=dev ;; + *-*) dist_tag=next ;; + *) dist_tag=latest ;; esac + # Everything checkable, checked before the first publish. The loop + # goes library, library, CLI, and a version number npm has seen can + # never be reused -- so a fault found on the third iteration leaves + # two packages published at a version the release can no longer use. + for name in ailoud-core ailoud-providers ailoud; do + # An ABSOLUTE path. Given `dist-npm/ailoud-core-1.0.0-dev.1.tgz`, + # npm read the slash as the `owner/repo` GitHub shorthand and tried + # to `git ls-remote ssh://git@github.com/dist-npm/...git`, which + # failed on a missing public key rather than on anything to do with + # publishing. + if [ ! -f "$PWD/dist-npm/${name}-${version}.tgz" ]; then + echo "::error::dist-npm/${name}-${version}.tgz was not packed." + exit 1 + fi + done + node scripts/preflight-npm-auth.mjs + echo "publishing $version under dist-tag $dist_tag" for name in ailoud-core ailoud-providers ailoud; do - tarball=$(ls dist-npm/${name}-${version}.tgz) + tarball="$PWD/dist-npm/${name}-${version}.tgz" echo "--- $tarball" npm publish "$tarball" --provenance --access public --tag "$dist_tag" done + - name: Create the GitHub release from CHANGES.md + # After the publish, so a release object never advertises a version the + # registry refused. `gh release create` rather than a third-party + # action: adding one to the release path would sit oddly beside a + # 14-day rule for everything else we depend on. + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + version="${TAG#v}" + # Writes RELEASE_NOTES.md from the `## Version ` section, + # and refuses if that section is missing, empty or over the limit. + node scripts/release-notes.mjs "$TAG" + prerelease="" + case "$version" in + *-*) prerelease="--prerelease" ;; + esac + if gh release view "$TAG" >/dev/null 2>&1; then + echo "release $TAG already exists; updating its notes" + gh release edit "$TAG" --notes-file RELEASE_NOTES.md + else + gh release create "$TAG" \ + --title "$TAG" \ + --notes-file RELEASE_NOTES.md \ + --verify-tag \ + $prerelease + fi + - name: Report what was published if: always() run: ls -l dist-npm || true diff --git a/.gitignore b/.gitignore index 6c04168..7352d59 100644 --- a/.gitignore +++ b/.gitignore @@ -26,3 +26,7 @@ docs/superpowers/**/scratch/ RELEASE_NOTES.md # npm tarballs built by the publish workflow dist-npm/ + +# Copied from the repository root by apps/cli prepack, so the npm page for +# `ailoud` shows the real README without a second copy in git. +apps/cli/README.md diff --git a/AGENTS.md b/AGENTS.md index 9652edd..97f6f1e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,12 +18,26 @@ resulting text. No microphone or system-audio capture; import only. The CLI is the only front end -- there is no GUI, and the interface is English-only. The multilingual part of this project is the audio, not the UI. -M1, the current milestone, ships `import`, `transcribe`, `ls`, `show`, and -`doctor`. The rest of the CLI surface (`search`, `collection`, `tag`, -`summarize`, `export`, `config`) belongs to later milestones and does not -exist yet; do not document or assume commands beyond what M1 lists. The full -design lives in the maintainer's planning notes under `.superpowers/`, which -is not tracked in git, so treat this file as the authority on what exists. +The commands, grouped by the noun they act on: + +| Command | Does | +| ---------------------------------------------------------- | ----------------------------------------- | +| `audio import\|transcribe\|annotate\|search\|ls\|show\|rm` | the library and everything over it | +| `audio summarize` | writes a summary and saves it as a report | +| `report ls\|show\|rm` | saved reports | +| `template ls\|new` | what shape a summary of a kind takes | +| `mcp` and `mcp install\|uninstall\|update` | serve the library to an agent | +| `doctor`, `setup` | check and provision the machine | +| `self check\|update\|sync` | this installation of ailoud | + +The verbs also answer at the top level (`ailoud search`, `ailoud transcribe`), +and each second-level verb has a one-letter alias. + +Do not trust a list of commands in prose over the binary. This section was +wrong for a while -- it described an early milestone and said `search` and +`summarize` "do not exist yet" long after both shipped -- so check with +`node apps/cli/dist/bin/ailoud.js --help` after a build, which cannot be +stale. --- @@ -224,13 +238,13 @@ Python tooling here is driven by `uv`, never `pip` or a hand-rolled venv. ### Structure -Four sections, and new pages belong in one of them: +Four sections, in nav order, and new pages belong in one of them: | Section | Holds | | --------------- | ------------------------------------------------ | | Getting Started | install, set up, first transcript, first summary | -| Usage | one page per thing you do with the CLI | | MCP | configuring and using the MCP server | +| Usage | one page per thing you do with the CLI | | Development | architecture, the gate, releasing | Every page added to `docs/` must appear in `nav:` in `mkdocs.yml`, or the @@ -257,6 +271,203 @@ Run the `check-docs` skill after changing any command or option. --- +## Branches and Tags + +| Branch | Is | +| --------- | --------------------------------------- | +| `main` | the release branch | +| `develop` | integration; feature branches land here | + +**While the project is pre-release, work happens directly on `main`** and +`develop` follows it: a push to `main` opens a back-merge PR +(`.github/workflows/backmerge.yml`). Once releases start that inverts -- +features land on `develop`, only release commits reach `main` -- and the same +workflow keeps `develop` from falling behind either way. Both branches are +protected against deletion and force-push with CI required; an admin can still +push directly, which is what makes the pre-release flow possible. + +The three required checks are the ones that run on a pull request. The +provisioned end-to-end job is deliberately NOT required: it runs only on push, +so requiring it would leave every PR waiting for a check that never arrives. + +| Tag | Cut from | npm dist-tag | Docs | Retires snapshots | +| -------------- | ---------- | ------------ | ---- | ----------------- | +| `v1.2.3-dev.1` | any branch | `dev` | no | no | +| `v1.2.3-rc.1` | `develop` | `next` | no | no | +| `v1.2.3` | `main` | `latest` | yes | yes | + +Only a final tag moves `latest`, publishes the site and retires what it +supersedes. `publish.yml` refuses a tag that disagrees with any manifest +version. + +Use the `dev-tag` skill for a snapshot; `bump-version` then `pre-release-check` +for a release. + +Cutting a final tag folds its pre-release sections back into one. The order +matters, and it is the opposite of what reads naturally: + +``` +node scripts/bump-version.mjs 1.0.0 # FIRST: promotes Development +node scripts/fold-prereleases.mjs 1.0.0 # then merges the 1.0.0-dev.* sections in +node scripts/check-changelog.mjs v1.0.0 # refuses if anything is left over +``` + +`bump-version` promotes `## Development` into a `## Version ` section, so +folding first leaves it a second, empty one to create -- two `## Version 1.0.0` +headings, and `check-changelog` then fails on the empty one with "has no +entries". Tested both ways in a sandbox before the first release. + +`publish.yml` runs that check on every tag before it builds anything. The +limits live in `scripts/lib/changelog.mjs` and are quoted, not restated, +everywhere else -- a limit that differs between the script that warns and the +script that refuses is worse than no limit. The scripts have tests +(`scripts/**/*.test.mjs`), which run in CI on branches and pull requests but +not on tags. + +--- + +## Publishing + +Nothing long-lived is stored. Publishing authenticates by exchanging the CI +job's OIDC identity for a short-lived, per-package npm token. Retiring the +snapshots a release supersedes does NOT -- see below; it is a manual step. + +### How the exchange works + +`npm publish` does this for itself, and `scripts/lib/npmOidc.mjs` makes the +same two calls (read out of npm's `lib/utils/oidc.js`) so that +`preflight-npm-auth.mjs` can establish before publishing that npm will accept +every package: + +``` +GET $ACTIONS_ID_TOKEN_REQUEST_URL&audience=npm:registry.npmjs.org + Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN -> { value } +POST /-/npm/v1/oidc/token/exchange/package/ + Authorization: Bearer -> { token } +``` + +Three consequences worth knowing before changing any of this: + +- **`id-token: write` is required**, or GitHub mints no identity and the first + call is skipped entirely. +- **The token it returns authenticates `npm publish` and nothing else.** It is + publish-scoped and spent: `npm deprecate` with it answered `E404 ... or you +do not have permission`, then `E401 ... token is invalid` on every call + after. Measured on the 1.0.0 release. So the exchange can answer "will npm + accept us", and cannot be used to do anything but publish. +- **A trusted publisher is bound to a workflow FILE.** Entering through a + reusable workflow leaves `workflow_ref` pointing at the caller and + `job_workflow_ref` at the callee; npm matches the caller, which the 1.0.0 + release also confirmed. +- **The token never reaches a command line or a log.** It goes into a + temporary 0600 `.npmrc` removed in a `finally`; `withNpmToken` is tested + both for the normal path and for a body that throws. The id token's claims + ARE logged -- `sub`, `workflow_ref`, `job_workflow_ref` -- because a + rejection is otherwise indistinguishable from a missing package. + +### The bootstrap exception + +A trusted publisher is attached to a package that already exists, so the FIRST +version of each package cannot use one: npm answers `ENEEDAUTH` however +complete the setup is. A token in the `NPM_TOKEN` secret introduces the +packages to the registry, and deleting the secret is the whole of the switch -- +`publish.yml` uses it when present and OIDC when not. + +The rule is enforced rather than remembered, because a token that keeps working +is a token nobody removes: + +| Tag | `NPM_TOKEN` set | Result | +| ----------- | --------------- | -------------------------- | +| pre-release | yes | publishes, logs a warning | +| final | yes | **refuses**, names the fix | +| any | no | OIDC exchange | + +### What blocks a release + +Enforced by `publish.yml`, all of it before anything is built or published: + +| Refuses when | Why | +| ---------------------------------------------- | ------------------------------------------- | +| the changelog is unfit (`check-changelog.mjs`) | a release nobody can read the notes for | +| an open high or critical code scanning alert | a finding nobody has looked at, shipped | +| a manifest disagrees with the tag | publishes a version nobody asked for | +| `NPM_TOKEN` is set on a final tag | a stored credential outliving its bootstrap | +| the gate or the end-to-end suite fails | the ordinary reasons | +| a tarball lacks its LICENCE or README | what 1.0.0-dev.1 shipped | + +A code scanning finding counts only while it is `open`. One reviewed and +explained is `dismissed` with its reason attached and does not block; fixing +and dismissing are both answers, ignoring is not. The dependency side of the +same question is the `check-dependencies` skill, which a human runs before +tagging. + +### Retiring superseded snapshots + +Only a production release retires anything, and it happens after the publish +succeeds -- deprecating first would, if the publish then failed, leave every +`-dev.N` pointing at a release that does not exist while `dev` is the only +installable thing. + +``` +pnpm retire 1.0.0 # prints the plan, changes nothing +NPM_TOKEN=npm_... pnpm retire 1.0.0 --yes # carries it out +``` + +**Do this after every final release, by hand.** Nothing prompts for it: the +release workflow cannot, and a release that is otherwise complete looks +finished. Until it runs, the superseded snapshots stay undeprecated and `@dev` +still resolves to the last one -- someone installing `ailoud@dev` gets an older +build than `ailoud`. + +`NPM_TOKEN` is **required** for `--yes`, not merely preferred. Falling back on +the ambient `npm login` sounds accommodating and is not: with 2FA on writes -- +the default -- npm asks for a one-time code on every write, and this makes +twelve of them, and without a usable credential it drops into an interactive +web login and waits, so the script looks hung. Refusing up front is the +difference between one authentication and twelve prompts. + +Use a granular access token with read-and-write on the three packages. It goes +into a temporary 0600 npmrc and never onto a command line. The plan (no +`--yes`) needs no credential at all. + +This is a manual step, run under `npm login`. Automating it was tried and +removed: the OIDC token cannot deprecate (above). If the npm side does not +complete, the script leaves the tags alone and exits non-zero -- the tags are +what name which pre-releases to retire, so deleting them after a failed +deprecation would destroy the only record of what was missed. + +It deprecates every `1.0.0-dev.*` of all three packages, moves the `dev` +dist-tag onto the release, and deletes the tags. Which versions exist comes +from the registry, not from the git tags: the tags used to be the list, which +made them the only record of what still needed retiring, and a tag is a thing +that gets deleted. Three things it deliberately does not do: + +- **Unpublish.** Allowed for 72 hours only, the version number can never be + reused after, and anyone who pinned it has their install broken. A deprecated + version keeps working and prints a notice. +- **Delete a tag whose commit is not on `origin/main`.** The published + provenance attests that commit; unreachable, it can be collected, leaving the + attestation pointing at nothing. Those tags are reported and kept. +- **Remove the `dev` dist-tag.** `npm dist-tag rm` makes `install @dev` + fail outright for anyone who uses it. It is pointed at the release instead, + so `@dev` keeps working and never hands out something older than `latest`. + +### npm facts that constrain all of the above + +None of these can be worked around, so design around them: + +- **A version number is used up forever.** `1.0.0-dev.1` cannot be republished + even after an unpublish. A botched snapshot needs a new number, not a retry. +- **`latest` is set on a package's first publish** whatever `--tag` says, and + `latest` can be moved but never removed. A package introduced by a + pre-release answers `npm install ` with it until a final version + exists. +- **`npm publish` is the only thing trusted publishing authenticates.** + `deprecate` and `dist-tag` need a real credential, which in practice means a + human at a terminal. + +--- + ## The Changelog `CHANGES.md` is for someone deciding whether to upgrade. It is not a commit @@ -319,7 +530,7 @@ about to add an entry will actually see them. ## Local Development Skills -Seven skills live under `.agents/skills/`. Invoke them when the situation +Ten skills live under `.agents/skills/`. Invoke them when the situation calls for it: | Skill | When to use | @@ -330,7 +541,10 @@ calls for it: | `check-licenses` | After editing any `package.json` -- verify all npm dependencies are license-compliant and update LICENSE. | | `run-tests-and-linters` | Before marking any task done -- run the full gate (build, format check, lint, typecheck, test:cov at 90%). | | `check-fixtures` | After touching import, transcribe, or the audio/STT providers -- drive the built binary against `fixtures/` end to end, in a throwaway `HOME`, `XDG_CONFIG_HOME`, and `XDG_DATA_HOME`, and confirm the working tree stays clean. | +| `check-dependencies` | Before every release and after any dependency change -- audit for advisories, report funding, and update what is behind, refusing any version published less than 14 days ago unless it fixes a critical advisory. | | `pre-release-check` | Before cutting a release -- runs the `check-*` and `run-tests-and-linters` skills above (not `bump-version`) plus version-bump and commit-format checks. | +| `dev-tag` | To publish a snapshot to npm under the `dev` dist-tag without promising a release -- cuts a `v-dev.` tag. | +| `check-migrations` | After touching `packages/core/src/db/schema.ts` -- run the migration lock and snapshot tests, and judge from git history whether an edited migration has already shipped (which the tests cannot see). | --- diff --git a/CHANGES.md b/CHANGES.md index b43f25e..88734d6 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -40,6 +40,33 @@ ## Development +## Version 1.1.0 + +### Added + +- `ailoud self update` installs a newer version, then refreshes the agent + rules block in every project ailoud has been used in. +- `ailoud self check` reports whether a newer version exists without + installing it, and takes `--json`. +- `ailoud self sync` refreshes the agent rules block in every registered + project without updating anything. +- Other commands mention a newer version at most once a day. Turn it off with + `AILOUD_NO_UPDATE_CHECK=1` or `update.check: false`. + +### Changed + +- `ailoud report ls` exits 0 when there are no reports, matching `ls` on an + empty library and its own `--json` form. +- Command output now stays inside the terminal frame instead of printing + around it. + +### Fixed + +- `ailoud mcp install` and `mcp update` no longer empty a rules file when the + write fails part-way, such as on a full disk. + +## Version 1.0.0 + ### Added - `ailoud audio import` adds audio and video files, or whole directories, to a @@ -55,10 +82,10 @@ as a report. `--template` shapes it for the kind of conversation -- a 1:1, a performance review, an architecture discussion, a decision between solutions -- and `--context` supplies what the transcript does not say. Templates are - editable YAML files under the config directory. -- Four summarisation engines behind one setting: a local GGUF model through - llama.cpp, Claude by subscription through the Claude Code CLI, Claude by API, - and any OpenAI-compatible endpoint including Ollama and LM Studio. + editable YAML files under the config directory. Four engines sit behind one + setting: a local GGUF model through llama.cpp, Claude by subscription through + the Claude Code CLI, Claude by API, and any OpenAI-compatible endpoint, + including Ollama and LM Studio. - `ailoud report ls|show|rm` lists, prints and deletes saved reports. - `ailoud mcp` serves the library to an AI agent over MCP: sixteen tools, three prompts, and transcripts as addressable resources. Deleting takes two calls, diff --git a/CLAUDE.md b/CLAUDE.md index 305d1be..999534e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,10 +1,10 @@ # CLAUDE.md -- ailoud `ailoud` is a command-line tool that transcribes audio and video into a local -recording library, and, in a later milestone, will summarize and answer -questions over it through a large language model. Speech-to-text and the -LLM are separate engine layers behind stable ports. There is no GUI; the -CLI is the only front end, and the interface is English-only. +recording library, searches it, and summarizes and answers questions over it +through a large language model. Speech-to-text and the LLM are separate engine +layers behind stable ports. There is no GUI; the CLI is the only front end, and +the interface is English-only. **Must read before touching code:** diff --git a/README.md b/README.md index 1e9c411..0da742a 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,9 @@

AILoud

+ npm Documentation - License + License Coverage CI

@@ -14,21 +15,6 @@ summarises it. Speech-to-text runs on your machine; summaries can too. --- -## Overview - -One CLI (`ailoud`) over a local library: - -- **Transcribe** audio and video with whisper.cpp, including recordings that - switch between languages, and attribute lines to speakers. -- **Search** the whole library full-text and get back the matching lines with - timestamps, not whole transcripts. -- **Summarise** one recording or a tagged group into a saved report, shaped by - a template for the kind of conversation it was. -- **Serve** the library to an AI agent over - [MCP](https://lorem-dev.github.io/ailoud/latest/mcp/). - -Nothing leaves your machine unless you choose a hosted model for summaries. - ## Install Needs [Node.js](https://nodejs.org/) 24 or newer. @@ -37,82 +23,99 @@ Needs [Node.js](https://nodejs.org/) 24 or newer. npm install -g ailoud ``` -Then install the tools it drives -- ffmpeg, whisper.cpp, the models: +`setup` installs the tools it drives -- ffmpeg, whisper.cpp, the models -- +and `doctor` checks them: ```shell ailoud setup ailoud doctor ``` -See [Getting Started](https://lorem-dev.github.io/ailoud/latest/getting-started/). - ---- - -## CLI quick start +Nothing leaves your machine unless you choose a hosted model for summaries. -Import, transcribe, read: +## Update ```shell -ailoud audio import ~/Recordings --tag standup -ailoud audio transcribe -ailoud audio ls -ailoud audio show 01M1B2 +ailoud self update ``` -Find where something was said, without reading a transcript: +It checks the registry first, so there is nothing to run before it. A snapshot +moves only to a newer snapshot of the same version, or to a release. -```shell -ailoud audio search rollback -ailoud audio f "before sunrise" --tag standup -``` +--- -Summarise, with a shape and the context the transcript does not carry: +## Use it with an agent ```shell -ailoud audio summarize 01M1B2 --template one-on-one \ - --context "Ann is Ben's manager; this is their fortnightly." -ailoud report ls +ailoud mcp install ``` -Every verb has a one-letter alias, and the letter means the same in every -group -- `l` list, `v` view, `r` remove, `f` find: +It configures one or more agents, at project or global scope: + +| Agent | Scopes | +| ---------- | --------------- | +| `claude` | project, global | +| `codex` | project, global | +| `opencode` | project, global | +| `gemini` | project, global | +| `hermes` | global only | +| `copilot` | global only | + +It writes the MCP registration and a rules block the agent reads before its +first call. The agent then gets these tools: ```shell -ailoud audio l -ailoud report l +echo '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | ailoud mcp | jq -r '.result.tools[].name' ``` -Run `ailoud --help` or ` --help` for the full set, also in the -[CLI Reference](https://lorem-dev.github.io/ailoud/latest/usage/cli/). +``` +list_recordings +list_untagged +list_tags +search_transcripts +get_transcript +list_speakers +list_reports +get_report +list_templates +annotate +import_recording +transcribe +summarize +create_template +delete_recording +delete_report +``` ---- +Reading tools return matches and file paths, never a whole transcript in one +call; deleting needs a second call carrying a confirmation token. See +[MCP](https://lorem-dev.github.io/ailoud/latest/mcp/). -## Templates +--- -A template decides a summary's headings, because different conversations -divide differently: `one-on-one`, `performance-review`, -`architecture-planning`, `solution-decision`, `offsite`, `meeting`. +## The CLI -```shell -ailoud template ls -ailoud template new sprint-retro --from one-on-one -``` +| Command | Does | +| ---------------------------------------------------------- | ----------------------------------------- | +| `audio import\|transcribe\|annotate\|search\|ls\|show\|rm` | the library and everything over it | +| `audio summarize` | writes a summary and saves it as a report | +| `report ls\|show\|rm` | saved reports | +| `template ls\|new` | what shape a summary of a kind takes | +| `mcp` and `mcp install\|uninstall\|update` | serve the library to an agent | +| `doctor`, `setup` | check and provision the machine | +| `self check\|update\|sync` | this installation of ailoud | -They are YAML files in `~/.config/ailoud/templates/`. Edit one and the change -takes effect; AILoud never overwrites a file you have edited. See -[Templates](https://lorem-dev.github.io/ailoud/latest/usage/templates/). +Every verb also works at the top level, and has a one-letter alias. Full +reference: [CLI Reference](https://lorem-dev.github.io/ailoud/latest/usage/cli/). --- -## MCP - -```json -{ "mcpServers": { "ailoud": { "command": "ailoud", "args": ["mcp"] } } } -``` +## Documentation -Sixteen tools over the same library the CLI uses. Deleting takes two calls: the -first describes what would go and returns a confirmation token, the second -carries it out. See [MCP](https://lorem-dev.github.io/ailoud/latest/mcp/). +- [Getting Started](https://lorem-dev.github.io/ailoud/latest/getting-started/) +- [Usage](https://lorem-dev.github.io/ailoud/latest/usage/recordings/) +- [MCP](https://lorem-dev.github.io/ailoud/latest/mcp/) +- [Development](https://lorem-dev.github.io/ailoud/latest/development/development/) --- @@ -125,14 +128,11 @@ A pnpm workspace: `packages/core` (domain and ports, no I/O), pnpm build && pnpm format:check && pnpm lint && pnpm typecheck && pnpm test:cov ``` -See the -[Development guide](https://lorem-dev.github.io/ailoud/latest/development/development/) -and [Architecture](https://lorem-dev.github.io/ailoud/latest/development/architecture/). Commit rules and the dependency licence policy are in -[CONTRIBUTING.md](./CONTRIBUTING.md). +[CONTRIBUTING.md](https://github.com/lorem-dev/ailoud/blob/main/CONTRIBUTING.md). --- ## License -Apache-2.0. See [LICENSE](./LICENSE). +Apache-2.0. See [LICENSE](https://github.com/lorem-dev/ailoud/blob/main/LICENSE). diff --git a/apps/cli/package.json b/apps/cli/package.json index ef55947..f40caad 100644 --- a/apps/cli/package.json +++ b/apps/cli/package.json @@ -1,6 +1,6 @@ { "name": "ailoud", - "version": "0.0.0", + "version": "1.1.0", "type": "module", "bin": { "ailoud": "./dist/bin/ailoud.js" @@ -18,10 +18,12 @@ "zod": "4.4.3" }, "scripts": { + "prepack": "node -e \"require('node:fs').copyFileSync('../../README.md','README.md')\"", "build": "tsc -b tsconfig.build.json && chmod +x dist/bin/ailoud.js", "typecheck": "tsc -p tsconfig.json --noEmit" }, "description": "Transcribe audio and video into a local library, then search and summarise it", + "author": "Lorem Dev ", "license": "Apache-2.0", "homepage": "https://lorem-dev.github.io/ailoud/", "bugs": { diff --git a/apps/cli/src/bin/ailoud.ts b/apps/cli/src/bin/ailoud.ts index ce2adff..81eaa1b 100644 --- a/apps/cli/src/bin/ailoud.ts +++ b/apps/cli/src/bin/ailoud.ts @@ -1,32 +1,80 @@ #!/usr/bin/env -S node --disable-warning=ExperimentalWarning +import { styleText } from 'node:util'; import { buildProgram, exitCodeFor, isCommanderError } from '../program.js'; import { createContext } from '../wiring.js'; - -async function main(): Promise { - const context = await createContext(process.env); - try { - await buildProgram(context).parseAsync(process.argv); - return 0; - } finally { - context.store.close(); - } -} +import { startUpdateCheck } from '../updateNotice.js'; +import type { UpdateCheck } from '../updateNotice.js'; +import { VERSION } from '../version.js'; function messageFor(error: unknown): string { return error instanceof Error ? error.message : String(error); } -main().then( - (code) => { - process.exitCode = code; - }, - (error: unknown) => { - const code = exitCodeFor(error); +async function main(): Promise { + // Started as early as possible -- right after the context exists -- and + // read only once the command has finished, below. Never awaited here: + // waiting on the network before the command's own work even starts would + // add up to `updateTimeoutMs` to every single run of, say, `ailoud ls`. + let notice: UpdateCheck | null = null; + let code: number; + try { + const context = await createContext(process.env); + notice = startUpdateCheck({ + fs: context.fs, + clock: context.clock, + userDataDir: context.paths.userDataDir, + currentVersion: VERSION, + argv: process.argv.slice(2), + env: process.env, + stderrIsTTY: process.stderr.isTTY === true, + checkEnabled: context.config.update.check, + // The port, not a second client. `VersionSource.published` takes the + // cancellation signal precisely so this caller can share the provider's + // implementation -- guards, escaping, empty-list refusal and all -- + // instead of the hand-rolled copy that used to live here and had + // drifted out of step with it. + published: (signal) => context.versionSource.published('ailoud', signal), + }); + try { + await buildProgram(context).parseAsync(process.argv); + code = 0; + } finally { + context.store.close(); + } + } catch (error) { + code = exitCodeFor(error); // commander already printed its own message for a usage failure, or // its help/version text, through configureOutput above. if (!isCommanderError(error)) { process.stderr.write(`ailoud: ${messageFor(error)}\n`); } - process.exitCode = code; - }, -); + } + + // Same place the process decides its exit code: printed only when the + // check has already settled (see `UpdateCheck.finish`'s own doc comment). + // Never printed when the context itself failed to build -- there is no + // `notice` at all in that case, so nothing to abort or to await. + if (notice !== null) { + const target = await notice.finish(); + if (target !== null) { + // Yellow: this is the one line ailoud prints that the user did not ask + // for, so it has to be distinguishable at a glance from the output they + // did. `styleText` is given the stream, so it emits nothing when stderr + // cannot take colour and it honours NO_COLOR -- belt and braces, since + // the notice is already suppressed when stderr is not a terminal. + process.stderr.write( + styleText( + 'yellow', + `ailoud: a newer version is available (${VERSION} -> ${target}). Run "ailoud self update" to install it.`, + { stream: process.stderr }, + ) + '\n', + ); + } + } + + return code; +} + +main().then((code) => { + process.exitCode = code; +}); diff --git a/apps/cli/src/commands/annotate.ts b/apps/cli/src/commands/annotate.ts index 1a54522..3ee2fa1 100644 --- a/apps/cli/src/commands/annotate.ts +++ b/apps/cli/src/commands/annotate.ts @@ -121,7 +121,7 @@ export function registerAnnotate(program: Command, context: CliContext): void { ? [] : [`${assignments.length} speaker name${assignments.length === 1 ? '' : 's'}`]), ]; - context.write(`${recording.id} set ${parts.join(', ')}`); + context.ui.success(`${recording.id} set ${parts.join(', ')}`); }); }); } diff --git a/apps/cli/src/commands/commands.test.ts b/apps/cli/src/commands/commands.test.ts index 6e93353..3fb4304 100644 --- a/apps/cli/src/commands/commands.test.ts +++ b/apps/cli/src/commands/commands.test.ts @@ -1,8 +1,25 @@ import { describe, expect, it } from 'vitest'; +import { Command } from 'commander'; import { FailureError } from '@ailoud/core'; import { buildProgram } from '../program.js'; import { context } from './testContext.js'; import { parseLanguages } from './transcribe.js'; +import { group } from './groups.js'; + +describe('group', () => { + it('gives a noun without a plural exactly one name', () => { + const program = new Command(); + group(program, 'self', undefined, 'manage this installation'); + const self = program.commands.find((c) => c.name() === 'self')!; + expect(self.aliases()).toEqual([]); + }); + + it('still aliases a noun that has a plural', () => { + const program = new Command(); + group(program, 'report', 'reports', 'saved reports'); + expect(program.commands.find((c) => c.name() === 'report')!.aliases()).toEqual(['reports']); + }); +}); describe('ailoud import', () => { it('prints the id of an imported recording', async () => { diff --git a/apps/cli/src/commands/doctor.test.ts b/apps/cli/src/commands/doctor.test.ts index 517fd45..161a187 100644 --- a/apps/cli/src/commands/doctor.test.ts +++ b/apps/cli/src/commands/doctor.test.ts @@ -550,6 +550,7 @@ describe('doctor --fix scope: remedies come only from failing checks', () => { dbFile: join(scopedDir, 'ailoud.db'), mediaRoot: join(scopedDir, 'media'), isProjectLibrary: false, + userDataDir: scopedDir, }, config: { stt: { @@ -580,6 +581,7 @@ describe('doctor --fix scope: remedies come only from failing checks', () => { model: join(scopedDir, 'llm-model.gguf'), }, }, + update: parseConfig(null).update, }, }; } @@ -691,6 +693,7 @@ describe('doctor: an unconfigured optional feature does not mean "not ready"', ( dbFile: join(dataDir, 'ailoud.db'), mediaRoot: join(dataDir, 'media'), isProjectLibrary: false, + userDataDir: dataDir, }, config: { stt: { @@ -710,6 +713,7 @@ describe('doctor: an unconfigured optional feature does not mean "not ready"', ( }, }, llm: parseConfig(null).llm, + update: parseConfig(null).update, }, }; } @@ -833,6 +837,7 @@ describe('a corrupt database: every entry point must refuse', () => { dbFile: join(corruptDir, 'ailoud.db'), mediaRoot: join(corruptDir, 'media'), isProjectLibrary: false, + userDataDir: corruptDir, }, config: { stt: { @@ -864,6 +869,7 @@ describe('a corrupt database: every entry point must refuse', () => { model: join(corruptDir, 'llm-model.gguf'), }, }, + update: parseConfig(null).update, }, }; } diff --git a/apps/cli/src/commands/doctor.ts b/apps/cli/src/commands/doctor.ts index cc55640..694428f 100644 --- a/apps/cli/src/commands/doctor.ts +++ b/apps/cli/src/commands/doctor.ts @@ -1,6 +1,6 @@ import { access, constants, stat } from 'node:fs/promises'; import type { Command } from 'commander'; -import { EnvironmentError, installHint } from '@ailoud/core'; +import { EnvironmentError, installHint, isHostedLlm } from '@ailoud/core'; import type { Remedy } from '@ailoud/core'; import { run } from '@ailoud/providers'; import type { CliContext } from '../wiring.js'; @@ -334,8 +334,10 @@ export async function checkLanguageModel( const key = apiKeyFrom(env, variable); const settings = llm.provider === 'anthropic' ? llm.anthropic : llm.openaiCompatible; // A local OpenAI-compatible server needs no key, so its absence is only a - // problem when the endpoint is a hosted one. - const hosted = /api\.(openai|anthropic)\.com/.test(settings.baseUrl); + // problem when the endpoint is a hosted one. By hostname: the pattern this + // replaced matched `https://api.openai.com.example.net` and + // `https://example.net/?x=api.openai.com` alike. + const hosted = isHostedLlm(settings.baseUrl); if (key === undefined && hosted) { return { name, diff --git a/apps/cli/src/commands/groups.ts b/apps/cli/src/commands/groups.ts index 619169d..02c5fc9 100644 --- a/apps/cli/src/commands/groups.ts +++ b/apps/cli/src/commands/groups.ts @@ -12,6 +12,11 @@ export type Register = (parent: Command, context: CliContext) => void; * `gh pr list`, `kubectl get pod`): `ailoud report rm SUM0` reads as removing one * report, while `ailoud reports rm SUM0` reads as removing all of them. * + * `plural` is optional: `self` has no plural. `selves` would put a word in + * `--help` nobody would ever type, and passing `self` as both the name and + * the alias makes commander throw on an alias equal to the name. A noun + * declared without one gets exactly its singular name and nothing else. + * * No `.action()` handler, deliberately. A bare `ailoud audio` already prints * its verb list -- commander does that for any command with subcommands and * no action of its own -- and adding one to force it cost two behaviours that @@ -23,10 +28,12 @@ export type Register = (parent: Command, context: CliContext) => void; export function group( program: Command, name: string, - plural: string, + plural: string | undefined, description: string, ): Command { - return program.command(name).alias(plural).description(description).showHelpAfterError(); + const command = program.command(name).description(description).showHelpAfterError(); + if (plural !== undefined) command.alias(plural); + return command; } /** @@ -70,8 +77,22 @@ export function inGroupAndTopLevel( * The one-letter alias for each second-level verb. * * One table rather than a letter beside each command definition, because the - * risk here is collision and a table is where you can see it: every letter - * below appears exactly once, and the test for that reads this map. + * risk here is collision and a table is where you can see it. + * + * A letter is unique WITHIN A GROUP, not across the table: `summarize` and + * `sync` both take `s`, and that is fine because they live under different + * nouns (`audio summarize`, `self sync`) and commander resolves an alias + * against one parent's children. This comment used to claim every letter + * appeared exactly once, which the table below already contradicted -- and + * the test only checked two of the four groups, so nothing would have caught + * a real collision inside `self` or `template`. It checks all of them now. + * + * `mcp` gets NO letters, and that is deliberate rather than an oversight: + * its verbs are `install`, `uninstall` and `update`, and `uninstall` and + * `update` both want `u`. A set that cannot be made unique and complete is + * better left off entirely than half-assigned, so `program.ts` does not call + * `attachLetters` for that group. Widening the collision test is what made + * this asymmetry visible; it had never been written down. * * The same verb gets the same letter in every group -- `l` lists, `v` views, * `r` removes -- so the letters are worth learning once instead of per noun. @@ -91,6 +112,9 @@ const LETTER: Record = { // `f` for find: `s` is summarize, and search is the verb people reach for // most often after `ls`. search: 'f', + check: 'c', + update: 'u', + sync: 's', }; /** Every letter this build assigns, for the collision test to read. */ diff --git a/apps/cli/src/commands/ls.ts b/apps/cli/src/commands/ls.ts index 8f67f21..6b20cb1 100644 --- a/apps/cli/src/commands/ls.ts +++ b/apps/cli/src/commands/ls.ts @@ -37,7 +37,14 @@ export function registerLs(program: Command, context: CliContext): void { if (recordings.length === 0) { if (options.json === true) { - context.write('[]'); + // Through the Ui, exactly like the non-empty branch below, and + // NOT on the raw channel. Raw looks like it protects the machine + // contract, and does not: `PlainUi` is what runs whenever stdout + // is not a terminal, so a pipe receives a bare `[]` either way. + // What raw does do is print this one line at column 0 while the + // frame is drawn around it -- verified in a sized pty. One branch + // of one command must not render two different ways. + context.ui.content('[]'); return; } context.ui.emptyLibrary(); diff --git a/apps/cli/src/commands/mcp.ts b/apps/cli/src/commands/mcp.ts index f63efca..0c9effd 100644 --- a/apps/cli/src/commands/mcp.ts +++ b/apps/cli/src/commands/mcp.ts @@ -3,6 +3,7 @@ import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js' import type { CliContext } from '../wiring.js'; import { buildMcpServer } from '../mcp/server.js'; import { registerMcpInstall } from './mcpInstall.js'; +import { VERSION } from '../version.js'; export function registerMcp(program: Command, context: CliContext): void { const mcp = program @@ -25,7 +26,7 @@ export function registerMcp(program: Command, context: CliContext): void { // channel. A single stray line of human-facing text would corrupt the // JSON-RPC stream and the client would drop the connection. Every other // command in this codebase writes through context.ui; this one must not. - const { server, close } = buildMcpServer(context, program.version() ?? '0.0.0'); + const { server, close } = buildMcpServer(context, program.version() ?? VERSION); const transport = new StdioServerTransport(); const shutdown = async (): Promise => { diff --git a/apps/cli/src/commands/mcpInstall.ts b/apps/cli/src/commands/mcpInstall.ts index 523fcfd..d9a06bd 100644 --- a/apps/cli/src/commands/mcpInstall.ts +++ b/apps/cli/src/commands/mcpInstall.ts @@ -6,8 +6,10 @@ import { AGENTS, agentIds, findAgent, globalOnly } from '../mcp/agents.js'; import type { AgentTarget, Scope } from '../mcp/agents.js'; import { defaultHome } from '../mcp/agents.js'; import { detect, ensureProjectLibrary, install, uninstall, update } from '../mcp/install.js'; -import type { AgentOutcome } from '../mcp/install.js'; +import type { AgentOutcome, FileOutcome } from '../mcp/install.js'; import { isInteractive } from './setup.js'; +import { rememberProject } from '../projects.js'; +import { VERSION } from '../version.js'; interface Options { readonly target?: string; @@ -95,15 +97,28 @@ async function askScope(agents: readonly AgentTarget[]): Promise { return parseScope(String(answer)); } +/** + * One line for a file a `mcp install`/`uninstall`/`update` action touched (or + * left alone). `created` and `updated` actually changed something on disk, so + * they are marked as successes; the rest -- `unchanged`, `removed`, `cleaned`, + * `absent` -- are informational: true, but not an achievement. + */ +function reportFile(context: CliContext, file: FileOutcome): void { + const line = `${file.action.padEnd(9)} ${file.path}`; + if (file.action === 'created' || file.action === 'updated') { + context.ui.success(line); + } else { + context.ui.note(line); + } +} + /** One line per file touched, so the user can see exactly what changed. */ function report(context: CliContext, outcomes: readonly AgentOutcome[]): void { for (const outcome of outcomes) { - for (const file of outcome.files) { - context.write(`${file.action.padEnd(9)} ${file.path}`); - } + for (const file of outcome.files) reportFile(context, file); } const notes = [...new Set(outcomes.map((outcome) => outcome.note))]; - for (const note of notes) context.write(`note: ${note}`); + for (const note of notes) context.ui.note(`note: ${note}`); } /** @@ -112,6 +127,30 @@ function report(context: CliContext, outcomes: readonly AgentOutcome[]): void { * Silently installing a global-only agent globally while the user asked for * "this project only" would be a surprise; saying so is not. */ +/** + * Records the project a successful `mcp install` just wrote rules into, with + * this build's version -- that is what lets a later `ailoud self sync` say + * "current" for it instead of rewriting bytes that have not changed. + * + * Registration is bookkeeping, not the user's request: a full disk, a + * read-only project directory, or any other write failure here must never + * fail an install that otherwise succeeded. Any error is swallowed down to a + * single debug line. + */ +async function registerAfterInstall(context: CliContext, cwd: string): Promise { + try { + await rememberProject( + { fs: context.fs, clock: context.clock, userDataDir: context.paths.userDataDir }, + { path: cwd, rulesVersion: VERSION }, + ); + } catch (error) { + process.stderr.write( + `ailoud: debug: could not register project "${cwd}": ` + + `${error instanceof Error ? error.message : String(error)}\n`, + ); + } +} + function splitByScope( agents: readonly AgentTarget[], scope: Scope, @@ -146,8 +185,8 @@ export function registerMcpInstall(parent: Command, context: CliContext): void { : await parseTargets(context, 'auto', home()); if (agents.length === 0) { - context.write('No agents selected, so nothing was configured.'); - context.write(`Run again with --target to name one: ${agentIds()}`); + context.ui.warn('No agents selected, so nothing was configured.'); + context.ui.warn(`Run again with --target to name one: ${agentIds()}`); return; } @@ -165,17 +204,24 @@ export function registerMcpInstall(parent: Command, context: CliContext): void { // project rather than joining the per-user collection. if (scope === 'local' && inScope.length > 0) { const library = await ensureProjectLibrary(context.fs, cwd()); - context.write(`${library.action.padEnd(9)} ${library.path}`); + reportFile(context, library); } for (const agent of inScope) { outcomes.push(await install(context.fs, agent, scope, home(), cwd())); } for (const agent of forcedGlobal) { - context.write(`${agent.label} reads no per-project config; configuring it globally.`); + context.ui.note(`${agent.label} reads no per-project config; configuring it globally.`); outcomes.push(await install(context.fs, agent, 'global', home(), cwd())); } + // Only once rules were actually written locally: a run that only + // touched global-only agents wrote nothing into this project, and + // has nothing to register. + if (scope === 'local' && inScope.length > 0) { + await registerAfterInstall(context, cwd()); + } + report(context, outcomes); }); }); @@ -213,11 +259,11 @@ export function registerMcpInstall(parent: Command, context: CliContext): void { if (touched.length === 0) { // Said plainly rather than reported as a success: an uninstall that // claims to have cleaned files it never touched teaches distrust. - context.write('Nothing to remove: no agent here was configured for AILoud.'); + context.ui.warn('Nothing to remove: no agent here was configured for AILoud.'); return; } report(context, outcomes); - context.write( + context.ui.note( 'The .ailoud/ library directory was left alone; delete it by hand if you want it gone.', ); }); @@ -240,8 +286,8 @@ export function registerMcpInstall(parent: Command, context: CliContext): void { } } if (outcomes.length === 0) { - context.write('Nothing to update: no agent here is configured for AILoud.'); - context.write('Run "ailoud mcp install" first.'); + context.ui.warn('Nothing to update: no agent here is configured for AILoud.'); + context.ui.warn('Run "ailoud mcp install" first.'); return; } report(context, outcomes); diff --git a/apps/cli/src/commands/reports.test.ts b/apps/cli/src/commands/reports.test.ts index 415d0e8..c9936a8 100644 --- a/apps/cli/src/commands/reports.test.ts +++ b/apps/cli/src/commands/reports.test.ts @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import { FailureError, UsageError } from '@ailoud/core'; +import { UsageError } from '@ailoud/core'; import type { Summary } from '@ailoud/core'; import { buildProgram } from '../program.js'; import { FIXTURE_PATH, contextWithTranscript } from './testContext.js'; @@ -52,14 +52,37 @@ describe('reportPreview', () => { }); describe('ailoud report ls / show', () => { - it('says so when there are none, rather than printing an empty table', async () => { + it('says so when there are none, and exits 0 rather than failing', async () => { + // Not a failure. `ls` on an empty library, `ls --tag` on a filter that + // matches nothing, and `self sync` with no projects all exit 0, and + // `report ls --json` already exited 0 on this very state -- so the text + // and JSON forms of one command used to disagree. A script cannot tell an + // empty list from a real failure if both exit non-zero. const ctx = await contextWithTranscript({ clearLines: true }); - await expect(buildProgram(ctx).parseAsync(['node', 'ailoud', 'report', 'ls'])).rejects.toThrow( - FailureError, - ); - await expect(buildProgram(ctx).parseAsync(['node', 'ailoud', 'report', 'ls'])).rejects.toThrow( - /No reports yet/, - ); + + await expect( + buildProgram(ctx).parseAsync(['node', 'ailoud', 'report', 'ls']), + ).resolves.toBeDefined(); + + expect(ctx.lines.join('\n')).toMatch(/No reports yet/); + }); + + it('exits 0 when a named recording has no reports', async () => { + const ctx = await contextWithTranscript({ clearLines: true }); + const [recording] = await ctx.store.listRecordings({}); + + await expect( + buildProgram(ctx).parseAsync([ + 'node', + 'ailoud', + 'report', + 'ls', + '--recording', + recording!.id, + ]), + ).resolves.toBeDefined(); + + expect(ctx.lines.join('\n')).toMatch(/No reports cover/); }); it('lists what produced each report, newest first', async () => { @@ -269,13 +292,37 @@ describe('command layout', () => { return hidden !== true; }) .map((command) => command.name()); - expect(visible).toEqual(['audio', 'report', 'template', 'mcp', 'doctor', 'setup']); + expect(visible).toEqual(['audio', 'report', 'template', 'mcp', 'doctor', 'setup', 'self']); }); it('gives every second-level verb a one-letter alias, none colliding', async () => { // Collision is the risk a single table exists to make visible. const ctx = await contextWithTranscript({ skipImport: true }); - for (const groupName of ['audio', 'report']) { + // EVERY group, discovered from the program rather than listed here: the + // old version named `audio` and `report` only, so a collision inside + // `self` or `template` -- the two groups added since -- would have gone + // unnoticed. Discovering them means a group added later is covered the + // day it appears. + const groups = buildProgram(ctx).commands.filter( + (command) => command.commands.length > 0 && command.name() !== 'help', + ); + // A group either assigns letters to ALL its verbs or to none. `mcp` is + // the deliberate none -- `uninstall` and `update` both want `u`, so the + // set cannot be made unique, and a half-assigned set is worse than no + // set. See `attachLetters` in groups.js. + const lettered = groups.filter((command) => + command.commands.some((verb) => verb.name() !== 'help' && verb.aliases().length > 0), + ); + const unlettered = groups.filter((command) => !lettered.includes(command)); + expect(lettered.map((command) => command.name()).sort()).toEqual([ + 'audio', + 'report', + 'self', + 'template', + ]); + expect(unlettered.map((command) => command.name())).toEqual(['mcp']); + + for (const groupName of lettered.map((command) => command.name())) { const found = buildProgram(ctx).commands.find((c) => c.name() === groupName)!; const letters = found.commands .filter((command) => command.name() !== 'help') diff --git a/apps/cli/src/commands/reports.ts b/apps/cli/src/commands/reports.ts index 564afa2..61012f3 100644 --- a/apps/cli/src/commands/reports.ts +++ b/apps/cli/src/commands/reports.ts @@ -1,5 +1,5 @@ import type { Command } from 'commander'; -import { FailureError, formatRecordedAt } from '@ailoud/core'; +import { formatRecordedAt } from '@ailoud/core'; import type { Summary } from '@ailoud/core'; import { page, shouldPage } from '@ailoud/providers'; import type { CliContext } from '../wiring.js'; @@ -112,11 +112,19 @@ export function registerReports(parent: Command, context: CliContext): void { context.ui.content('[]'); return; } - throw new FailureError( + // Exit 0, not a failure. "Nothing here yet" is the same answer + // `ls` gives for an empty library, `ls --tag` for a filter that + // matches nothing, and `self sync` for no registered projects -- + // and `report ls --json` already exited 0 on this very state, so + // the text and JSON forms of one command disagreed. A script that + // treats an empty list as an error cannot tell it from a real + // failure. + context.ui.note( options.recording === undefined ? 'No reports yet. Run "ailoud summarize " to make one.' : `No reports cover ${options.recording}.`, ); + return; } if (options.json === true) { @@ -153,20 +161,20 @@ export function registerReports(parent: Command, context: CliContext): void { const summaries: Summary[] = []; for (const id of ids) summaries.push(await resolveSummary(context.store, id)); - context.write( + context.ui.content( summaries.length === 1 ? 'This will permanently delete 1 report:' : `This will permanently delete ${summaries.length} reports:`, ); for (const summary of summaries) { - context.write( + context.ui.content( ` ${summary.id} ${formatRecordedAt(summary.createdAt)} ${summary.model} ` + reportPreview(summary.body, 40), ); } // The recordings and transcripts stay: a report is derived, and what it // was derived from is the library itself. - context.write('The recordings and their transcripts are not touched.'); + context.ui.note('The recordings and their transcripts are not touched.'); // The same guard rm and setup use, for the same reason: one question, // one answer, and the same refusal with no terminal so a script cannot @@ -182,13 +190,18 @@ export function registerReports(parent: Command, context: CliContext): void { consentFlag: '--force', }); if (!consented) { - context.write('Nothing was deleted.'); + context.ui.warn('Nothing was deleted.'); return; } for (const summary of summaries) { const deleted = await context.store.deleteSummary(summary.id); - context.write(`${summary.id} ${deleted ? 'deleted' : 'was already gone'}`); + const line = `${summary.id} ${deleted ? 'deleted' : 'was already gone'}`; + if (deleted) { + context.ui.success(line); + } else { + context.ui.note(line); + } } }); }); diff --git a/apps/cli/src/commands/rm.ts b/apps/cli/src/commands/rm.ts index 3299667..d13d74a 100644 --- a/apps/cli/src/commands/rm.ts +++ b/apps/cli/src/commands/rm.ts @@ -43,7 +43,7 @@ export function registerRm(program: Command, context: CliContext): void { // ids must not leave the first two gone, and there is no undo. const recordings = await resolveRecordings(context.store, ids); - for (const line of describeDeletion(recordings)) context.write(line); + for (const line of describeDeletion(recordings)) context.ui.content(line); // Reuses setup's guard rather than writing a second one: same // question (may this run change things without being asked?), same @@ -59,7 +59,7 @@ export function registerRm(program: Command, context: CliContext): void { consentFlag: '--force', }); if (!consented) { - context.write('Nothing was deleted.'); + context.ui.warn('Nothing was deleted.'); return; } diff --git a/apps/cli/src/commands/self.test.ts b/apps/cli/src/commands/self.test.ts new file mode 100644 index 0000000..f60bcd2 --- /dev/null +++ b/apps/cli/src/commands/self.test.ts @@ -0,0 +1,926 @@ +import { describe, expect, it } from 'vitest'; +import { FailureError, UsageError } from '@ailoud/core'; +import type { PublishedVersion, VersionSource } from '@ailoud/core'; +import { MemFs, FakeClock } from '@ailoud/core/testing'; +import type { RunOptions, RunResult } from '@ailoud/providers'; +import { buildProgram, exitCodeFor } from '../program.js'; +import { context } from './testContext.js'; +import { findAgent } from '../mcp/agents.js'; +import { install } from '../mcp/install.js'; +import { pruneProjects, readProjects, rememberProject } from '../projects.js'; +import type { SyncDeps } from './self.js'; +import { boundedDetectRun, syncProjects, updateSelf } from './self.js'; +import type { SelfUpdateDeps } from './self.js'; +import { updateLogPath } from '../updateLog.js'; +import { VERSION } from '../version.js'; + +/** + * The build's own version, and one patch above it. + * + * Derived rather than written down: these tests assert what `ailoud` says + * about ITSELF, so a literal `1.0.0` made every one of them fail the moment + * the version was bumped for a release -- and a literal `1.0.1` offered as + * "newer" became OLDER than the build at 1.1.0, which is how a fixture stops + * testing what it claims. `NEWER` strips any pre-release suffix before + * bumping, so it stays above `VERSION` on a snapshot build too. + */ +const NEWER = (() => { + const [major, minor, patch] = VERSION.split('-')[0]!.split('.'); + return `${major}.${minor}.${Number(patch) + 1}`; +})(); + +/** A VersionSource that answers with a fixed list, never touching the network. */ +function source(published: readonly PublishedVersion[]): VersionSource { + return { published: async () => published }; +} + +describe('ailoud self check', () => { + it('reports the target it would move to', async () => { + const ctx = { ...context(), versionSource: source([{ version: NEWER, deprecated: false }]) }; + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'check']); + expect(ctx.lines).toEqual([`ailoud ${VERSION} can update to ${NEWER}.`]); + }); + + it('says so when there is nothing newer', async () => { + const ctx = { ...context(), versionSource: source([{ version: VERSION, deprecated: false }]) }; + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'check']); + expect(ctx.lines).toEqual([`ailoud ${VERSION} is already the newest published version.`]); + // A version check is not a test: it must exit 0 either way. + }); + + it('prints JSON with --json', async () => { + const ctx = { ...context(), versionSource: source([{ version: NEWER, deprecated: false }]) }; + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'check', '--json']); + expect(JSON.parse(ctx.lines.join(''))).toEqual({ + current: VERSION, + target: NEWER, + updatable: true, + }); + }); + + it('prints JSON with no target when there is nothing newer', async () => { + const ctx = { ...context(), versionSource: source([{ version: VERSION, deprecated: false }]) }; + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'check', '--json']); + expect(JSON.parse(ctx.lines.join(''))).toEqual({ + current: VERSION, + target: null, + updatable: false, + }); + }); + + it('fails with the host and the timeout when the registry is unreachable', async () => { + const ctx = { + ...context(), + updateRegistryHost: 'registry.npmjs.org', + updateTimeoutMs: 10_000, + versionSource: { + published: async (): Promise => { + throw new Error('fetch failed'); + }, + }, + }; + // Exit 1: an explicit check that could not run must not read as "up to date". + await expect(buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'check'])).rejects.toThrow( + FailureError, + ); + const error: unknown = await buildProgram(ctx) + .parseAsync(['node', 'ailoud', 'self', 'check']) + .catch((caught: unknown) => caught); + expect(error).toBeInstanceOf(FailureError); + expect((error as Error).message).toContain('registry.npmjs.org'); + // NOT asserting a timeout here: "fetch failed" is a refused connection or + // a bad name, and calling it a timeout sends the user to the wrong place. + // The underlying reason has to survive instead. + expect((error as Error).message).toContain('fetch failed'); + expect((error as Error).message).not.toContain('timed out'); + expect(exitCodeFor(error)).toBe(1); + }); + + it('exists as a hidden top-level alias, "ailoud check"', async () => { + const ctx = { ...context(), versionSource: source([{ version: VERSION, deprecated: false }]) }; + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'check']); + expect(ctx.lines).toEqual([`ailoud ${VERSION} is already the newest published version.`]); + }); + + it('answers to its one-letter alias inside the group', async () => { + const ctx = { ...context(), versionSource: source([{ version: VERSION, deprecated: false }]) }; + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'c']); + expect(ctx.lines).toEqual([`ailoud ${VERSION} is already the newest published version.`]); + }); +}); + +describe('syncProjects', () => { + const USER_DATA_DIR = '/data/ailoud'; + const HOME = '/home/user'; + const claude = findAgent('claude')!; + + function deps(fs: MemFs): SyncDeps { + return { fs, clock: new FakeClock(), userDataDir: USER_DATA_DIR, home: HOME }; + } + + /** Rewrites a project's CLAUDE.md so it no longer matches the current build's block. */ + async function makeStale(fs: MemFs, projectPath: string): Promise { + const rulesPath = `${projectPath}/CLAUDE.md`; + const current = await fs.readTextFile(rulesPath); + const stale = current.replace('## AILoud', '## AILoud (text from an older ailoud build)'); + await fs.writeTextFile(rulesPath, stale); + } + + it('refreshes a rules block and reports it as refreshed', async () => { + const fs = new MemFs({}); + await install(fs, claude, 'local', HOME, '/proj/a'); + const rulesPath = '/proj/a/CLAUDE.md'; + const current = await fs.readTextFile(rulesPath); + await makeStale(fs, '/proj/a'); + + const d = deps(fs); + await rememberProject(d, { path: '/proj/a' }); + + const report = await syncProjects(d); + + expect(report.rows).toEqual([{ path: '/proj/a', status: 'refreshed' }]); + expect(report.failed).toBe(false); + // Rewritten back to exactly the bytes the current build would have + // written on a fresh install -- update() is idempotent by construction. + expect(await fs.readTextFile(rulesPath)).toBe(current); + }); + + it('reports a project whose block is already current, without writing', async () => { + class LoggingFs extends MemFs { + readonly writes: string[] = []; + override async writeTextFile(path: string, content: string): Promise { + this.writes.push(path); + return super.writeTextFile(path, content); + } + } + const fs = new LoggingFs({}); + await install(fs, claude, 'local', HOME, '/proj/a'); + + const d = deps(fs); + await rememberProject(d, { path: '/proj/a' }); + const writesBeforeSync = fs.writes.length; + + const report = await syncProjects(d); + + expect(report.rows).toEqual([{ path: '/proj/a', status: 'current' }]); + expect(report.failed).toBe(false); + // Only bookkeeping (the registry, the log) may be written from here on; + // the project's own rules/config files must be untouched because + // update() already found them byte-identical to the current build. + const newWrites = fs.writes.slice(writesBeforeSync); + expect(newWrites).not.toContain('/proj/a/CLAUDE.md'); + expect(newWrites).not.toContain('/proj/a/.mcp.json'); + }); + + it('reports a project with no rules block as such', async () => { + const fs = new MemFs({}); + fs.dirs.add('/proj/empty'); // the directory exists; ailoud was just never installed into it + + const d = deps(fs); + await rememberProject(d, { path: '/proj/empty' }); + + const report = await syncProjects(d); + + expect(report.rows).toEqual([{ path: '/proj/empty', status: 'no rules here' }]); + expect(report.failed).toBe(false); + }); + + it('continues after one project fails, and exits non-zero', async () => { + // Throws only once armed, so seeding the fixture (which itself writes + // through this same Fs) is not what trips the failure. + class FlakyFs extends MemFs { + armed = false; + override async writeTextFile(path: string, content: string): Promise { + // `includes`, not `===`: the rules file is written to + // `..tmp` and renamed over the target, so matching the + // target exactly would stop injecting the fault altogether and leave + // this test quietly asserting the happy path. + if (this.armed && path.includes('/proj/b/CLAUDE.md')) { + throw new Error('EACCES: permission denied'); + } + return super.writeTextFile(path, content); + } + } + const fs = new FlakyFs({}); + await install(fs, claude, 'local', HOME, '/proj/a'); + await install(fs, claude, 'local', HOME, '/proj/b'); + await makeStale(fs, '/proj/a'); + await makeStale(fs, '/proj/b'); + fs.armed = true; + + const d = deps(fs); + await rememberProject(d, { path: '/proj/a' }); + await rememberProject(d, { path: '/proj/b' }); + + const report = await syncProjects(d); + + expect(report.failed).toBe(true); + const byPath = new Map(report.rows.map((row) => [row.path, row.status])); + // The other nineteen (here: the other one) still get refreshed. + expect(byPath.get('/proj/a')).toBe('refreshed'); + expect(byPath.get('/proj/b')).toMatch(/^failed: /); + expect(byPath.get('/proj/b')).toContain('permission denied'); + }); + + it('prunes a project whose directory is gone', async () => { + const fs = new MemFs({}); + const d = deps(fs); + await rememberProject(d, { path: '/proj/gone' }); + // '/proj/gone' is deliberately never added to fs.dirs. + + const report = await syncProjects(d); + + expect(report.rows).toEqual([{ path: '/proj/gone', status: 'gone' }]); + expect(report.failed).toBe(false); + expect(await readProjects(d)).toEqual([]); + }); + + it('records rulesVersion so the next sync can say "current"', async () => { + const fs = new MemFs({}); + await install(fs, claude, 'local', HOME, '/proj/a'); + await makeStale(fs, '/proj/a'); + + const d = deps(fs); + await rememberProject(d, { path: '/proj/a' }); + + const first = await syncProjects(d); + expect(first.rows).toEqual([{ path: '/proj/a', status: 'refreshed' }]); + + const [entry] = await readProjects(d); + expect(entry?.rulesVersion).toBe(VERSION); + + // Nothing changed the second time: the rows come from update()'s own + // byte comparison, not from re-reading rulesVersion, but recording it is + // what a caller (a future "self status") would use to explain why. + const second = await syncProjects(d); + expect(second.rows).toEqual([{ path: '/proj/a', status: 'current' }]); + }); + + it('appends one line per run to the update log', async () => { + const fs = new MemFs({}); + await install(fs, claude, 'local', HOME, '/proj/a'); + fs.dirs.add('/proj/empty'); + + const d = deps(fs); + await rememberProject(d, { path: '/proj/a' }); + await rememberProject(d, { path: '/proj/empty' }); + + await syncProjects(d); + + const log = await fs.readTextFile(updateLogPath(USER_DATA_DIR)); + const lines = log.split('\n').filter((line) => line.length > 0); + expect(lines).toHaveLength(2); // one line per project row this run produced + expect(lines.some((line) => line.includes('/proj/a'))).toBe(true); + expect(lines.some((line) => line.includes('/proj/empty'))).toBe(true); + }); + + it('caps the log rather than growing it forever', async () => { + const fs = new MemFs({}); + fs.dirs.add('/proj/a'); + const seedLines = Array.from({ length: 20_000 }, (_, i) => `old line ${i} ${'x'.repeat(50)}`); + await fs.ensureDir(USER_DATA_DIR); + await fs.writeTextFile(updateLogPath(USER_DATA_DIR), `${seedLines.join('\n')}\n`); + + const d = deps(fs); + await rememberProject(d, { path: '/proj/a' }); + + await syncProjects(d); + + const log = await fs.readTextFile(updateLogPath(USER_DATA_DIR)); + const lines = log.split('\n').filter((line) => line.length > 0); + expect(lines.length).toBeLessThanOrEqual(500); + expect(lines[lines.length - 1]).toContain('/proj/a'); // the newest action is never lost + }); +}); + +describe('ailoud self sync (CLI)', () => { + it('says so when no project is registered', async () => { + const ctx = context(); + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'sync']); + expect(ctx.lines).toEqual(['No projects registered yet.']); + }); + + it('answers to its one-letter alias inside the group', async () => { + const ctx = context(); + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 's']); + expect(ctx.lines).toEqual(['No projects registered yet.']); + }); + + it('exists as a hidden top-level alias, "ailoud sync"', async () => { + const ctx = context(); + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'sync']); + expect(ctx.lines).toEqual(['No projects registered yet.']); + }); + + it('prints one row per project and exits non-zero when one failed', async () => { + class FlakyFs extends MemFs { + armed = false; + override async writeTextFile(path: string, content: string): Promise { + // `includes`, not `===`: the rules file is written to + // `..tmp` and renamed over the target, so matching the + // target exactly would stop injecting the fault altogether and leave + // this test quietly asserting the happy path. + if (this.armed && path.includes('/proj/a/CLAUDE.md')) { + throw new Error('EACCES: permission denied'); + } + return super.writeTextFile(path, content); + } + } + const fs = new FlakyFs({}); + const claude = findAgent('claude')!; + await install(fs, claude, 'local', '/home/user', '/proj/a'); + const rulesPath = '/proj/a/CLAUDE.md'; + const current = await fs.readTextFile(rulesPath); + await fs.writeTextFile(rulesPath, current.replace('## AILoud', '## AILoud (old)')); + fs.armed = true; + + const ctx = { ...context(), fs }; + await rememberProject( + { fs, clock: ctx.clock, userDataDir: ctx.paths.userDataDir }, + { path: '/proj/a' }, + ); + + const error: unknown = await buildProgram(ctx) + .parseAsync(['node', 'ailoud', 'self', 'sync']) + .catch((caught: unknown) => caught); + + expect(error).toBeInstanceOf(FailureError); + expect(exitCodeFor(error)).toBe(1); + // The row now carries the "warning: " marker `ui.warn` adds for a + // `failed:` status (see registerSelfSync), so the assertion looks for + // that prefix rather than the bare status word. + expect( + ctx.lines.some((line) => line.startsWith('warning: failed:') && line.includes('/proj/a')), + ).toBe(true); + }); +}); + +describe('boundedDetectRun', () => { + it('gives detectInstallMethod a run bounded to 10 seconds', async () => { + const seen: Array<{ command: string; args: readonly string[]; options?: RunOptions }> = []; + const fakeRunImpl = async ( + command: string, + args: readonly string[], + options?: RunOptions, + ): Promise => { + seen.push({ command, args, ...(options === undefined ? {} : { options }) }); + return { code: 0, stdout: '', stderr: '' }; + }; + + const bounded = boundedDetectRun(fakeRunImpl); + await bounded('npm', ['root', '-g']); + + expect(seen).toEqual([ + { command: 'npm', args: ['root', '-g'], options: { timeoutMs: 10_000 } }, + ]); + }); +}); + +describe('updateSelf', () => { + /** `npm root -g` answers this root; `pnpm` is never installed on this fake machine. */ + function fakeDetectRun(roots: { npm?: string; pnpm?: string }) { + return async (command: string, _args: readonly string[]): Promise => { + const root = command === 'npm' ? roots.npm : command === 'pnpm' ? roots.pnpm : undefined; + if (root === undefined) return { code: 1, stdout: '', stderr: `${command}: not found` }; + return { code: 0, stdout: root, stderr: '' }; + }; + } + + /** A fake global npm install: packageRoot sits under the root `run` reports. */ + function npmGlobalDeps( + ctx: ReturnType, + overrides: Partial = {}, + ): SelfUpdateDeps { + return { + context: ctx, + execPath: '/usr/local/bin/node', + packageRoot: '/opt/homebrew/lib/node_modules/ailoud', + realpath: async (p: string) => p, + run: fakeDetectRun({ npm: '/opt/homebrew/lib/node_modules' }), + spawn: async () => 0, + // Every test below is either interactive, or does not reach the + // install spawn at all; the two tests that exercise the non-interactive + // --force path override this explicitly. + runCommand: async () => { + throw new Error('runCommand should not be called while deps.interactive is true'); + }, + interactive: true, + ...overrides, + }; + } + + function withTarget(): ReturnType { + return { ...context(), versionSource: source([{ version: NEWER, deprecated: false }]) }; + } + + it('says so and changes nothing when there is nothing newer', async () => { + const ctx = context(); // default versionSource reports the current VERSION + const calls: Array<[string, readonly string[]]> = []; + const deps = npmGlobalDeps(ctx, { + spawn: async (command, args) => { + calls.push([command, args]); + return 0; + }, + }); + + await updateSelf(deps, {}); + + expect(ctx.lines).toEqual([ + `ailoud ${VERSION} is already the newest version you can update to`, + ]); + expect(calls).toEqual([]); + }); + + it('prints a plan and changes nothing with --dry-run', async () => { + const ctx = withTarget(); + const calls: Array<[string, readonly string[]]> = []; + const deps = npmGlobalDeps(ctx, { + spawn: async (command, args) => { + calls.push([command, args]); + return 0; + }, + }); + + await updateSelf(deps, { dryRun: true }); + + expect(calls).toEqual([]); + expect(ctx.lines.some((line) => line.includes(VERSION))).toBe(true); + expect(ctx.lines.some((line) => line.includes(NEWER))).toBe(true); + expect(ctx.lines.some((line) => line.includes(`npm install -g ailoud@${NEWER}`))).toBe(true); + expect(ctx.lines.some((line) => line.toLowerCase().includes('dry run'))).toBe(true); + expect(await ctx.fs.exists(updateLogPath(ctx.paths.userDataDir))).toBe(false); + }); + + it('refuses an npx install and prints the npx command, exiting 0', async () => { + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { + packageRoot: '/Users/x/.npm/_npx/abcd1234/node_modules/ailoud', + }); + + await expect(updateSelf(deps, {})).resolves.toBeUndefined(); + + expect(ctx.lines).toContain(`npx ailoud@${NEWER}`); + }); + + it('refuses a project dependency, naming the project', async () => { + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { + packageRoot: '/Users/x/code/some-app/node_modules/ailoud', + }); + + await expect(updateSelf(deps, {})).resolves.toBeUndefined(); + + expect(ctx.lines.some((line) => line.includes('/Users/x/code/some-app'))).toBe(true); + }); + + it('names the real target in the unknown-install-method hint', async () => { + // Detection cannot know the target, so its hints carry a `` + // placeholder. By the time one is printed the target IS known, and a + // refusal exists to give the user a command to run -- one they have to + // edit first is a worse version of that. + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { packageRoot: '/opt/somewhere/odd/ailoud' }); + + await expect(updateSelf(deps, {})).resolves.toBeUndefined(); + + const said = ctx.lines.join('\n'); + expect(said).not.toContain(''); + expect(said).toContain(`ailoud@${NEWER}`); + }); + + it('exits non-zero when --force meets an install method it cannot use', async () => { + // A refusal is information; a forced update that cannot happen is an error. + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { + packageRoot: '/Users/x/code/some-app/node_modules/ailoud', + }); + + const error: unknown = await updateSelf(deps, { force: true }).catch( + (caught: unknown) => caught, + ); + + expect(error).toBeInstanceOf(FailureError); + expect(exitCodeFor(error)).toBe(1); + }); + + it('refuses without a terminal and names --force', async () => { + // The rule the shared confirmation helper already enforces for `rm`. + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { interactive: false }); + + const error: unknown = await updateSelf(deps, {}).catch((caught: unknown) => caught); + + expect(error).toBeInstanceOf(UsageError); + expect((error as Error).message).toContain('--force'); + }); + + it('declines when the user says no at the prompt, changing nothing', async () => { + const ctx = withTarget(); + const calls: Array<[string, readonly string[]]> = []; + const deps = npmGlobalDeps(ctx, { + spawn: async (command, args) => { + calls.push([command, args]); + return 0; + }, + confirmImpl: async () => false, + }); + + await updateSelf(deps, {}); + + expect(calls).toEqual([]); + expect(ctx.lines).toContain('Nothing was changed.'); + }); + + it('anchors the npm-global install beside the running node, not to a bare "npm"', async () => { + // detectInstallMethod deliberately anchors on execPath because PATH's + // npm can belong to a different Node than the one running us (nvm, fnm, + // asdf, volta). Throwing that anchor away here and spawning bare 'npm' + // would undo the whole reason execPath was threaded through in the first + // place -- see this file's own doc comment on updateSelf. + const ctx = withTarget(); + const calls: Array<[string, readonly string[]]> = []; + const deps = npmGlobalDeps(ctx, { + spawn: async (command, args) => { + calls.push([command, args]); + return 0; + }, + }); + + await updateSelf(deps, { force: true }); + + expect(calls[0]).toEqual(['/usr/local/bin/npm', ['install', '-g', `ailoud@${NEWER}`]]); + }); + + it('anchors the npm-global sweep beside the running node, not to a bare "ailoud"', async () => { + // The rules text is compiled in, so the process being replaced holds the + // old text. This asserts the spawned command is the installed binary's + // `self sync`, resolved the same way the install command is -- a bare + // 'ailoud' resolved off PATH could be a stale, unrelated install (see the + // machine layout in task-8-review.md). + const ctx = withTarget(); + const calls: Array<[string, readonly string[]]> = []; + const deps = npmGlobalDeps(ctx, { + spawn: async (command, args) => { + calls.push([command, args]); + return 0; + }, + }); + + await updateSelf(deps, { force: true }); + + expect(calls).toHaveLength(2); + expect(calls[1]).toEqual(['/usr/local/bin/ailoud', ['self', 'sync']]); + }); + + it('keeps the pnpm-global install as a bare "pnpm", deliberately', async () => { + // pnpm's global bin comes from PNPM_HOME/corepack, not from any one + // Node's install tree, so there is no execPath-equivalent anchor for the + // install -- bare 'pnpm' IS the right resolution. Asserted explicitly so + // nobody "fixes" this into a broken anchor later. + const ctx = withTarget(); + const calls: Array<[string, readonly string[]]> = []; + const pnpmRun = async (command: string, args: readonly string[]): Promise => { + if (command === 'npm') return { code: 1, stdout: '', stderr: 'npm: not found' }; + if (args[0] === 'root') { + return { code: 0, stdout: '/home/x/.local/share/pnpm/global/5/node_modules', stderr: '' }; + } + return { code: 0, stdout: '/home/x/.local/share/pnpm', stderr: '' }; + }; + const deps = npmGlobalDeps(ctx, { + packageRoot: '/home/x/.local/share/pnpm/global/5/node_modules/ailoud', + run: pnpmRun, + spawn: async (command, args) => { + calls.push([command, args]); + return 0; + }, + }); + + await updateSelf(deps, { force: true }); + + expect(calls[0]).toEqual(['pnpm', ['add', '-g', `ailoud@${NEWER}`]]); + }); + + it('invokes the pnpm-global sweep at the path "pnpm bin -g" reports', async () => { + const ctx = withTarget(); + const calls: Array<[string, readonly string[]]> = []; + const pnpmRun = async (command: string, args: readonly string[]): Promise => { + if (command === 'npm') return { code: 1, stdout: '', stderr: 'npm: not found' }; + if (args[0] === 'root') { + return { code: 0, stdout: '/home/x/.local/share/pnpm/global/5/node_modules', stderr: '' }; + } + // args[0] === 'bin': what sweepCommandFor asks to anchor the sweep. + return { code: 0, stdout: '/home/x/.local/share/pnpm', stderr: '' }; + }; + const deps = npmGlobalDeps(ctx, { + packageRoot: '/home/x/.local/share/pnpm/global/5/node_modules/ailoud', + run: pnpmRun, + spawn: async (command, args) => { + calls.push([command, args]); + return 0; + }, + }); + + await updateSelf(deps, { force: true }); + + expect(calls).toHaveLength(2); + expect(calls[1]).toEqual(['/home/x/.local/share/pnpm/ailoud', ['self', 'sync']]); + }); + + it('prints the command to run by hand when the sweep cannot be spawned', async () => { + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { + spawn: async (command) => { + if (command === '/usr/local/bin/ailoud') throw new Error('ailoud: command not found'); + return 0; + }, + }); + + await expect(updateSelf(deps, { force: true })).resolves.toBeUndefined(); + + expect(ctx.lines.some((line) => line.includes('ailoud self sync'))).toBe(true); + }); + + it('does not sweep when the install failed', async () => { + const ctx = withTarget(); + const calls: Array<[string, readonly string[]]> = []; + const deps = npmGlobalDeps(ctx, { + spawn: async (command, args) => { + calls.push([command, args]); + return 1; + }, + }); + + const error: unknown = await updateSelf(deps, { force: true }).catch( + (caught: unknown) => caught, + ); + + expect(error).toBeInstanceOf(FailureError); + expect(calls).toHaveLength(1); + }); + + it('logs the outcome', async () => { + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { spawn: async () => 0 }); + + await updateSelf(deps, { force: true }); + + const log = await ctx.fs.readTextFile(updateLogPath(ctx.paths.userDataDir)); + expect(log).toContain('self update'); + expect(log).toContain(NEWER); + }); + + it('--force with no TTY fails rather than hangs when the manager wants input', async () => { + // This is given its OWN short vitest timeout, deliberately: if a + // regression ever routes this path back through the unbounded + // `spawn` (runInteractive), that fake below never resolves, and this + // test must show up as a FAILING test rather than hang the whole + // suite -- this repository has been bitten by exactly that before. + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { + interactive: false, + // Would hang forever if updateSelf ever called this non-interactively. + spawn: () => new Promise(() => undefined), + // What run() does when the bounded timeout actually fires: reject, + // never resolve with a code -- so the real timeout can never present + // as an ordinary non-zero exit. + runCommand: async () => { + throw new FailureError('pnpm timed out after 600000 ms and was killed'); + }, + }); + + const error: unknown = await updateSelf(deps, { force: true }).catch( + (caught: unknown) => caught, + ); + + expect(error).toBeInstanceOf(FailureError); + expect((error as Error).message).toContain('timed out'); + }, 2000); + + it('bounds the post-install sweep too when there is no terminal, instead of hanging', async () => { + // Finding C from the review: the sweep spawn called deps.spawn + // (runInteractive) unconditionally, ignoring deps.interactive entirely -- + // unlike the install spawn just above, which is correctly gated. A + // --force run with no TTY whose sweep stalls (a registered project on a + // dead network mount, say) used to hang the parent forever with no + // output. Given its own short vitest timeout for the same reason as the + // install test above: a regression here must show up as a FAILING test, + // not a hung suite. + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { + interactive: false, + runCommand: async (command) => { + // The install succeeds... + if (command === '/usr/local/bin/npm') return { code: 0, stdout: '', stderr: '' }; + // ...but the sweep, bounded the same way, times out. + throw new FailureError('ailoud self sync timed out after 600000 ms and was killed'); + }, + // Would hang forever if the sweep were ever routed through the + // unbounded spawn (runInteractive) non-interactively. + spawn: () => new Promise(() => undefined), + }); + + await expect(updateSelf(deps, { force: true })).resolves.toBeUndefined(); + + expect( + ctx.lines.some((line) => line.includes('ailoud self sync') && line.includes('timed out')), + ).toBe(true); + }, 2000); + + it('prints the sweep output and reports a non-zero exit, bounded and with no terminal', async () => { + const ctx = withTarget(); + const deps = npmGlobalDeps(ctx, { + interactive: false, + runCommand: async (command) => { + if (command === '/usr/local/bin/npm') return { code: 0, stdout: '', stderr: '' }; + return { code: 1, stdout: 'refreshed 2 of 3 projects', stderr: 'one project failed' }; + }, + spawn: () => new Promise(() => undefined), + }); + + await expect(updateSelf(deps, { force: true })).resolves.toBeUndefined(); + + expect(ctx.lines).toContain('refreshed 2 of 3 projects'); + expect(ctx.lines).toContain('one project failed'); + expect(ctx.lines.some((line) => line.includes('Could not run "ailoud self sync"'))).toBe(true); + }, 2000); + + it('logs a throwing install spawn, and does not trigger the sweep', async () => { + // If the install spawn THROWS (e.g. the manager binary was not found) + // rather than resolving with a non-zero code, the sweep is correctly + // skipped -- but this used to leave the failure completely unlogged. + const ctx = withTarget(); + const calls: Array<[string, readonly string[]]> = []; + const deps = npmGlobalDeps(ctx, { + spawn: async (command, args) => { + calls.push([command, args]); + throw new Error('npm: command not found'); + }, + }); + + const error: unknown = await updateSelf(deps, { force: true }).catch( + (caught: unknown) => caught, + ); + + expect(error).toBeInstanceOf(Error); + expect((error as Error).message).toContain('command not found'); + expect(calls).toHaveLength(1); // the install attempt only, never the sweep + + const log = await ctx.fs.readTextFile(updateLogPath(ctx.paths.userDataDir)); + expect(log).toContain('self update'); + expect(log).toContain('command not found'); + }); +}); + +describe('ailoud self update (CLI wiring)', () => { + it('is already the newest version, printed through the real command', async () => { + const ctx = context(); + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'update']); + expect(ctx.lines).toEqual([ + `ailoud ${VERSION} is already the newest version you can update to`, + ]); + }); + + it('answers to its one-letter alias inside the group', async () => { + const ctx = context(); + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'self', 'u']); + expect(ctx.lines).toEqual([ + `ailoud ${VERSION} is already the newest version you can update to`, + ]); + }); + + it('exists as a hidden top-level alias, "ailoud update"', async () => { + const ctx = context(); + await buildProgram(ctx).parseAsync(['node', 'ailoud', 'update']); + expect(ctx.lines).toEqual([ + `ailoud ${VERSION} is already the newest version you can update to`, + ]); + }); +}); + +describe('the failure message found in review', () => { + const throwing = (error: Error) => ({ + published: async (): Promise => { + throw error; + }, + }); + + const messageFrom = async (error: Error): Promise => { + const ctx = { + ...context(), + updateRegistryHost: 'registry.npmjs.org', + updateTimeoutMs: 10_000, + versionSource: throwing(error), + }; + const caught: unknown = await buildProgram(ctx) + .parseAsync(['node', 'ailoud', 'self', 'check']) + .catch((thrown: unknown) => thrown); + expect(caught).toBeInstanceOf(FailureError); + return (caught as Error).message; + }; + + it('does not call an HTTP error a timeout', async () => { + // NpmRegistry already reports a status, or an unreadable body, accurately. + // Framing every failure as "timed out" sent the user off to check a + // network that had answered perfectly well. + const message = await messageFrom(new Error('the npm registry answered 503 for ailoud')); + expect(message).toContain('503'); + expect(message).not.toContain('timed out'); + }); + + it('does call an actual timeout a timeout, with the wait', async () => { + const timeout = new Error('The operation was aborted due to timeout'); + timeout.name = 'TimeoutError'; + const message = await messageFrom(timeout); + expect(message).toContain('timed out after 10000ms'); + }); +}); + +describe('the sweep must survive bookkeeping failures (task 7 review)', () => { + class AnnoyedFs extends MemFs { + public constructor(private readonly failOn: string) { + super({}); + } + override async isDirectory(path: string): Promise { + if (path === this.failOn) throw new Error('EACCES: permission denied'); + return super.isDirectory(path); + } + } + + it('keeps a project whose directory cannot be read, instead of forgetting it', async () => { + // EACCES is not evidence the project is gone -- revoked permissions or a + // dead network mount answer the same way. Dropping the entry would forget + // a project that still holds a rules block, silently and for good. + const fs = new AnnoyedFs('/proj/locked'); + const clock = new FakeClock(); + const deps = { fs, clock, userDataDir: '/data/ailoud' }; + await rememberProject(deps, { path: '/proj/locked' }); + + const dropped = await pruneProjects(deps); + + expect(dropped).toEqual([]); + expect((await readProjects(deps)).map((entry) => entry.path)).toEqual(['/proj/locked']); + }); + + it('reports a failure instead of sweeping nothing when the registry cannot be read', async () => { + // Unguarded, this ended the whole sweep with zero rows and zero log + // lines, which is indistinguishable from "there was nothing to do". + class UnreadableFs extends MemFs { + override async readTextFile(path: string): Promise { + if (path.endsWith('projects.json')) throw new Error('EACCES: permission denied'); + return super.readTextFile(path); + } + } + const fs = new UnreadableFs({}); + await fs.writeTextFile('/data/ailoud/projects.json', '[]\n'); + const report = await syncProjects({ + fs, + clock: new FakeClock(), + userDataDir: '/data/ailoud', + home: '/home/x', + }); + + expect(report.failed).toBe(true); + expect(report.rows.some((row) => row.status.startsWith('failed:'))).toBe(true); + }); +}); + +describe('a partial refresh must not be reported as a plain failure', () => { + const BLOCK = 'keep me\n\nold rules\n\n'; + + /** Rejects writes to one agent's rules file, leaving the other's to succeed. */ + class OneUnwritableAgent extends MemFs { + override async writeTextFile(path: string, content: string): Promise { + // `includes`, not `endsWith`: the write now goes to a temporary file + // beside the target, so an `endsWith` match injects nothing. + if (path.includes('GEMINI.md')) throw new Error('EROFS: read-only file system'); + return super.writeTextFile(path, content); + } + } + + it('says some agents were refreshed, and does not record the rules version', async () => { + // One project can hold configs for several agents. Claude's block is + // rewritten and Gemini's write then fails: reporting a plain `failed` + // would claim nothing changed in a file this command had just edited. + // And recording the version after a partial write would make the NEXT + // sweep call the project `current` and skip the block still left stale. + const fs = new OneUnwritableAgent({ + '/proj/a/CLAUDE.md': BLOCK, + '/proj/a/GEMINI.md': BLOCK, + }); + const clock = new FakeClock(); + const registry = { fs, clock, userDataDir: '/data/ailoud' }; + // The directory has to exist, or the sweep prunes the entry before it + // ever reaches an agent. + await fs.ensureDir('/proj/a'); + await rememberProject(registry, { path: '/proj/a' }); + + const report = await syncProjects({ ...registry, home: '/home/x' }); + + const row = report.rows.find((candidate) => candidate.path === '/proj/a'); + expect(row?.status).toMatch(/^failed:/); + expect(row?.status).toContain('some agents were refreshed'); + expect(report.failed).toBe(true); + const entry = (await readProjects(registry)).find((candidate) => candidate.path === '/proj/a'); + expect(entry?.rulesVersion).toBeUndefined(); + }); +}); diff --git a/apps/cli/src/commands/self.ts b/apps/cli/src/commands/self.ts new file mode 100644 index 0000000..b2bf5a6 --- /dev/null +++ b/apps/cli/src/commands/self.ts @@ -0,0 +1,648 @@ +import { fileURLToPath } from 'node:url'; +import { realpath } from 'node:fs/promises'; +import type { Command } from 'commander'; +import { confirm, isCancel } from '@clack/prompts'; +import { FailureError, UsageError, chooseUpdateTarget } from '@ailoud/core'; +import { + detectInstallMethod, + installCommandFor, + run, + runInteractive, + sweepCommandFor, +} from '@ailoud/providers'; +import type { DetectOptions, RunOptions, RunResult } from '@ailoud/providers'; +import type { CliContext } from '../wiring.js'; +import { VERSION } from '../version.js'; +import { AGENTS, defaultHome } from '../mcp/agents.js'; +import { update } from '../mcp/install.js'; +import type { AgentOutcome } from '../mcp/install.js'; +import { pruneProjects, readProjects, registryPath, rememberProject } from '../projects.js'; +import type { ProjectEntry, ProjectsDeps } from '../projects.js'; +import { appendUpdateLog } from '../updateLog.js'; +import { isInteractive } from './setup.js'; + +/** + * The only package this project ever asks the registry about. Never + * `@ailoud/core` or `@ailoud/providers` -- they arrive as this package's own + * dependencies, so their versions follow whatever `ailoud` itself resolves to. + */ +const PACKAGE_NAME = 'ailoud'; + +interface SelfCheckOptions { + readonly json?: boolean; +} + +export interface SelfCheckResult { + readonly current: string; + readonly target: string | null; + readonly updatable: boolean; +} + +/** + * The version `ailoud` is running, and the newest one it could move to. + * + * Throws a `FailureError` when the lookup itself could not be performed -- + * the registry was unreachable, timed out, or answered with something this + * build cannot read -- naming the host and the timeout so the message never + * reads as "you are up to date" when the truth is "this could not be + * checked". Finding no newer version is not a failure: that is `target === + * null` in an ordinary result, because a version check is not a test. + */ +export async function checkForUpdate(context: CliContext): Promise { + let published; + try { + published = await context.versionSource.published(PACKAGE_NAME); + } catch (error) { + const reason = error instanceof Error ? error.message : String(error); + // Name the timeout only when it WAS one. NpmRegistry already reports an + // HTTP status or an unreadable body accurately, and wrapping those in + // "timed out" sends the user to check a network that answered fine. + const timedOut = + error instanceof Error && (error.name === 'TimeoutError' || error.name === 'AbortError'); + const how = timedOut ? ` (timed out after ${context.updateTimeoutMs}ms)` : ''; + throw new FailureError( + `ailoud could not check ${context.updateRegistryHost} for a newer version${how}: ${reason}`, + ); + } + const target = chooseUpdateTarget(VERSION, published); + return { current: VERSION, target, updatable: target !== null }; +} + +export function registerSelfCheck(parent: Command, context: CliContext): void { + parent + .command('check') + .option('--json', 'print one JSON object instead of text') + .description('Check whether a newer version of ailoud is published') + .action(async (options: SelfCheckOptions) => { + await context.ui.frame('Checking for updates', async () => { + const result = await checkForUpdate(context); + if (options.json === true) { + // Raw, undecorated: a machine reader parses this, the same + // contract `ls --json` keeps by writing straight through here + // rather than through the decorated `ui`. + context.ui.content(JSON.stringify(result)); + return; + } + context.ui.content( + result.target === null + ? `ailoud ${result.current} is already the newest published version.` + : `ailoud ${result.current} can update to ${result.target}.`, + ); + }); + }); +} + +/** What `syncProjects` needs beyond the project registry: where the agents' home lives. */ +export interface SyncDeps extends ProjectsDeps { + readonly home: string; +} + +/** + * One project's outcome, for the table `self sync` prints. + * + * The five shapes below are the whole point of the command: a sweep over + * twenty projects must say which two actually changed, not "20 updated". + */ +export interface SyncRow { + readonly path: string; + readonly status: 'refreshed' | 'current' | 'no rules here' | 'gone' | `failed: ${string}`; +} + +export interface SyncReport { + readonly rows: readonly SyncRow[]; + readonly failed: boolean; +} + +/** One line in the update log, naming only the action taken -- never a path's contents. */ +async function logSyncAction(deps: SyncDeps, path: string, status: string): Promise { + await appendUpdateLog(deps, `${deps.clock.nowIso()} self sync ${status} ${path}`); +} + +/** + * Rewrites the rules block in every registered project with this build's + * text, prunes entries whose directory is gone, and reports one row per + * project. + * + * Reuses `update()` from mcp/install.ts unchanged: it already touches only + * the bytes between the AILOUD markers and is already idempotent, which is + * exactly the contract a sweep across other people's repositories needs. + * Only the `local` scope is swept -- a project entry names one directory, + * and the global agent files it: `~/.claude/CLAUDE.md` and friends -- are + * not properties of any one project, so sweeping them once per registered + * project would be both wrong and, over many projects, wasted work. + * + * `FileOutcome.action` (from `update()`, via `install()`) is what tells + * "refreshed" (bytes changed) from "current" (nothing written) -- not a + * version comparison -- so a sweep over twenty projects never claims twenty + * edits when it made two. + * + * One project failing never stops the sweep: each is wrapped in its own + * try/catch, so an unwritable repository lands one `failed:` row instead of + * aborting the other nineteen. `report.failed` is what tells the caller to + * exit non-zero. + */ +export async function syncProjects(deps: SyncDeps): Promise { + const rows: SyncRow[] = []; + let failed = false; + + // Pruning and reading the registry are bookkeeping, and bookkeeping must + // not be able to cancel the work. Unguarded, an EACCES on one registered + // directory -- or on projects.json itself -- ended the whole sweep with no + // rows and no log lines, which looks exactly like "there was nothing to do". + let dropped: readonly ProjectEntry[] = []; + try { + dropped = await pruneProjects(deps); + } catch (error) { + failed = true; + const reason = error instanceof Error ? error.message : String(error); + rows.push({ path: registryPath(deps.userDataDir), status: `failed: ${reason}` }); + await logSyncAction(deps, registryPath(deps.userDataDir), `failed: ${reason}`); + } + for (const entry of dropped) { + rows.push({ path: entry.path, status: 'gone' }); + await logSyncAction(deps, entry.path, 'gone'); + } + + let projects: readonly ProjectEntry[] = []; + try { + projects = await readProjects(deps); + } catch (error) { + failed = true; + const reason = error instanceof Error ? error.message : String(error); + rows.push({ path: registryPath(deps.userDataDir), status: `failed: ${reason}` }); + await logSyncAction(deps, registryPath(deps.userDataDir), `failed: ${reason}`); + } + for (const entry of projects) { + try { + const outcomes: AgentOutcome[] = []; + const failures: string[] = []; + for (const agent of AGENTS) { + if (!agent.scopes.includes('local')) continue; + // Caught PER AGENT, not around the loop. One project can hold configs + // for several agents, and a throw on the second one used to report the + // whole project as `failed` even though the first one's file had + // already been rewritten -- telling the user nothing changed in a + // directory this command had just edited. + try { + const outcome = await update(deps.fs, agent, 'local', deps.home, entry.path); + if (outcome !== null) outcomes.push(outcome); + } catch (error) { + failures.push(error instanceof Error ? error.message : String(error)); + } + } + + if (failures.length > 0) { + failed = true; + const wrote = outcomes.some((outcome) => + outcome.files.some((file) => file.action !== 'unchanged'), + ); + // Deliberately NOT recording rulesVersion on a partial failure: some + // agent here still has a stale block, and recording the version would + // make the next sweep call this project `current` and skip it. + const status = ( + wrote ? `failed: ${failures[0]} (some agents were refreshed)` : `failed: ${failures[0]}` + ) as SyncRow['status']; + rows.push({ path: entry.path, status }); + await logSyncAction(deps, entry.path, status); + continue; + } + + if (outcomes.length === 0) { + rows.push({ path: entry.path, status: 'no rules here' }); + await logSyncAction(deps, entry.path, 'no rules here'); + continue; + } + + const changed = outcomes.some((outcome) => + outcome.files.some((file) => file.action !== 'unchanged'), + ); + if (changed) { + // Recorded immediately, bypassing rememberProject's 24-hour + // throttle by design (see its own doc comment): a rules write just + // happened, and that is what lets the NEXT sync explain a "current" + // row rather than only report it. + await rememberProject(deps, { + path: entry.path, + ...(entry.libraryDir === undefined ? {} : { libraryDir: entry.libraryDir }), + rulesVersion: VERSION, + }); + rows.push({ path: entry.path, status: 'refreshed' }); + await logSyncAction(deps, entry.path, 'refreshed'); + } else { + rows.push({ path: entry.path, status: 'current' }); + await logSyncAction(deps, entry.path, 'current'); + } + } catch (error) { + failed = true; + const reason = error instanceof Error ? error.message : String(error); + const status = `failed: ${reason}` as const; + rows.push({ path: entry.path, status }); + await logSyncAction(deps, entry.path, status); + } + } + + return { rows, failed }; +} + +export function registerSelfSync(parent: Command, context: CliContext): void { + parent + .command('sync') + .description('Refresh the rules block in every project ailoud has been used in') + .action(async () => { + await context.ui.frame('Syncing rules', async () => { + const report = await syncProjects({ + fs: context.fs, + clock: context.clock, + userDataDir: context.paths.userDataDir, + home: defaultHome(), + }); + + if (report.rows.length === 0) { + context.ui.content('No projects registered yet.'); + return; + } + for (const row of report.rows) { + const line = `${row.status}: ${row.path}`; + if (row.status === 'refreshed') { + context.ui.success(line); + } else if (row.status.startsWith('failed:')) { + context.ui.warn(line); + } else { + context.ui.note(line); + } + } + if (report.failed) { + throw new FailureError( + 'ailoud self sync: at least one project failed to refresh; see the rows above.', + ); + } + }); + }); +} + +/** + * What `updateSelf` needs beyond `checkForUpdate`'s `CliContext`: everything + * that touches the machine or spawns another process, grouped here rather + * than folded into `CliContext` itself so a test can replace every one of + * them. `registerSelfUpdate` below is the only place the real node + * primitives are ever passed in; every test in this file passes fakes + * instead, so no test ever spawns a real process. + */ +export interface SelfUpdateDeps { + readonly context: CliContext; + /** + * `process.execPath`: which Node is running us, so a global install under + * nvm, fnm, asdf or volta is recognised even though the npm on PATH often + * belongs to a different Node version. Required by `DetectOptions` -- see + * its doc comment in `installMethod.ts`. + */ + readonly execPath: string; + /** Where this package sits on disk, resolved from `import.meta.url`. */ + readonly packageRoot: string; + readonly realpath: (path: string) => Promise; + /** + * What `detectInstallMethod` calls to ask `npm`/`pnpm` for their global + * root. Bounded to 10 seconds in production by `boundedDetectRun` -- see + * its own doc comment -- so a hung `npm root -g` cannot hang the whole + * command. + */ + readonly run: DetectOptions['run']; + /** + * Runs a subprocess with the parent's own stdio, so its output streams to + * the real terminal as it happens, and resolves to its exit code. Bound to + * `runInteractive` (from `@ailoud/providers`) in production. This is the + * one seam that would otherwise run a real package manager, so every test + * in this file injects a fake here instead of that binding. + * + * Used for the package-manager install, and for the post-install `self + * sync` sweep, but ONLY when `interactive` is true for either -- `run + * Interactive` has no timeout by design, which is only safe when a real + * terminal is watching and can interrupt it. Neither spawn is safe to + * leave ungated: `self sync` does not prompt for input either, but "does + * not prompt" is not the same as "cannot stall" -- a registered project on + * a dead network mount is enough to hang this unconditionally, which is + * exactly the defect the non-interactive branch below exists to close. + */ + readonly spawn: (command: string, args: readonly string[]) => Promise; + /** + * Runs the package-manager install, and the post-install sweep, BOUNDED -- + * for the one case `spawn` (`runInteractive`) must never be used for + * either: `--force` with no terminal attached. Bound to `run` (from + * `@ailoud/providers`) in production, given a generous but finite timeout + * by `updateSelf` itself -- see its own doc comment for why an unbounded + * wait is not safe there. + */ + readonly runCommand: ( + command: string, + args: readonly string[], + options?: RunOptions, + ) => Promise; + /** + * Whether there is a real terminal to confirm ON, which means STDIN is a + * TTY and this is not CI -- see `isInteractive` in `setup.ts`, which is + * what every caller uses. + * + * Stdin only, deliberately: the question is whether anyone can ANSWER a + * prompt, and that is stdin's business. `ailoud self update > out.txt` from + * a terminal is still interactive, and should be. This comment used to + * claim both ends had to be a TTY, which no call site checks. + */ + readonly interactive: boolean; + /** Overrides `@clack/prompts`' `confirm` in tests. */ + readonly confirmImpl?: (message: string) => Promise; +} + +export interface SelfUpdateOptions { + readonly force?: boolean; + readonly dryRun?: boolean; +} + +/** + * Wraps a `run` implementation so every call `detectInstallMethod` makes + * through it carries a hard 10 second cap. + * + * Detection happens automatically, before anything is printed or confirmed -- + * unlike the package-manager spawn later in `updateSelf`, which only runs + * after the user has already seen and agreed to the plan. `run()`'s own + * default timeout is thirty minutes (see `providers/process/run.ts`); left at + * that default, a hung `npm root -g` would hang `self update` for half an + * hour with nothing on screen to explain why. + */ +export function boundedDetectRun( + runImpl: (command: string, args: readonly string[], options?: RunOptions) => Promise, +): DetectOptions['run'] { + return (command, args) => runImpl(command, args, { timeoutMs: 10_000 }); +} + +/** + * How long the FORCED, non-interactive install is allowed to run before it + * is treated as hung rather than merely slow. Ten minutes is ample for a + * real package install; it exists only to turn "the manager is waiting on a + * prompt nobody can answer" into a failure instead of an infinite wait. See + * `updateSelf`'s own doc comment for the full reasoning. + */ +const FORCE_INSTALL_TIMEOUT_MS = 10 * 60_000; + +/** + * How long the FORCED, non-interactive `self sync` sweep is allowed to run + * before it is treated as hung rather than merely slow. Same duration and + * the same reasoning as `FORCE_INSTALL_TIMEOUT_MS`: this exists only to turn + * a stall -- a registered project sitting on a dead network mount, say -- + * into a failure instead of an infinite, silent wait. See `updateSelf`'s own + * doc comment for the full reasoning. + */ +const FORCE_SWEEP_TIMEOUT_MS = 10 * 60_000; + +/** One line in the update log, naming only the action taken. */ +async function logUpdateAction(context: CliContext, status: string): Promise { + await appendUpdateLog( + { fs: context.fs, userDataDir: context.paths.userDataDir }, + `${context.clock.nowIso()} self update ${status}`, + ); +} + +/** The real `confirm`, the way `commands/setup.ts` uses it: `isCancel` is "no". */ +const defaultConfirm = async (message: string): Promise => { + const answer = await confirm({ message }); + if (isCancel(answer)) return false; + return answer === true; +}; + +/** + * Installs a newer ailoud, then re-syncs the rules block through a fresh + * subprocess -- never in this one. + * + * That last part is load-bearing, not a style choice: the rules text is + * compiled into the code, so the process being replaced still holds the OLD + * text. If IT swept the registered projects, it would write the old rules + * into every one of them -- precisely the staleness `self sync` exists to + * fix. So the sweep only ever happens by spawning `ailoud self sync` as a + * subprocess. + * + * Both the install and the sweep are anchored, never a bare command name + * resolved off PATH: under nvm/fnm/asdf/volta, PATH's `npm`/`ailoud` can + * belong to an entirely different Node install than the one running us, + * which would install the new version into the wrong tree and then sweep + * every registered project with a DIFFERENT, possibly older, binary's + * compiled-in rules -- both while reporting success. `installCommandFor` and + * `sweepCommandFor` (`@ailoud/providers`) are the single places that decide + * those two argvs, anchored to `deps.execPath` for `npm-global` and to + * `pnpm bin -g` for `pnpm-global` -- see their own doc comments. + * + * Both the install AND the sweep only ever wait unboundedly (`deps.spawn`, + * bound to `runInteractive`) when `deps.interactive` is true, i.e. a real + * terminal is attached and can answer a prompt or interrupt it. `--force` + * with no terminal is the one path that can still reach either spawn with + * nobody able to answer a prompt or interrupt a stall -- some package + * managers do prompt on first global use (`pnpm add -g` before its bin/PATH + * setup has run once), and the sweep can stall for reasons of its own (a + * registered project on a dead network mount, say) even though it never + * prompts -- so that path uses `deps.runCommand` (bound to the bounded + * `run`) instead, for both, with a generous but finite timeout, so either + * one stuck waiting FAILS after that timeout, with its output shown, rather + * than hanging forever. This follows the same convention + * `provision/llamaInstall.ts` uses for `runInteractive`: gate on + * interactivity before ever calling it. + * + * Follows the eight steps of the design's `self update` section in order: + * resolve the target, detect the install method (three of its five kinds are + * refusals), print the plan, confirm (skipped by `--force`; `--dry-run` stops + * here, having changed nothing), spawn the package manager, spawn the new + * binary's `self sync` only on success, and log the outcome -- including a + * throwing install spawn, which used to vanish unlogged. + */ +export async function updateSelf(deps: SelfUpdateDeps, options: SelfUpdateOptions): Promise { + const { context } = deps; + + const result = await checkForUpdate(context); + if (result.target === null) { + context.ui.content(`ailoud ${result.current} is already the newest version you can update to`); + return; + } + const target = result.target; + + const method = await detectInstallMethod({ + execPath: deps.execPath, + packageRoot: deps.packageRoot, + realpath: deps.realpath, + run: deps.run, + }); + + // npx, project and unknown all carry a hint and nothing else: a refusal + // that does not say what to do instead is just a failure. Narrowed by + // `method.kind` rather than by `installCommandFor(...) === null`, so + // TypeScript knows `method.hint` exists on every branch that reads it. + if (method.kind === 'npx' || method.kind === 'project' || method.kind === 'unknown') { + // The hints carry a `` placeholder because detection does not + // know the target -- but by here we do, so fill it in. A refusal exists to + // hand the user a command they can run, and one they have to edit first + // is a worse version of that. + const hint = method.hint.replaceAll('', target); + context.ui.content(hint); + if (options.force === true) { + // A refusal is information; a forced update that cannot happen is an + // error, because --force asked for a guarantee this install method + // cannot give. + throw new FailureError(`ailoud self update cannot install this way: ${hint}`); + } + return; + } + + const command = installCommandFor(method, target, deps.execPath); + if (command === null) { + // Unreachable today: installCommandFor only returns null for the three + // kinds handled above. Kept as a real check rather than a cast, so a + // future InstallMethod variant fails loudly here instead of spawning + // `undefined`. + throw new FailureError('ailoud self update: no install command for this install method'); + } + const managerCommand = command[0]; + if (managerCommand === undefined) { + throw new FailureError('ailoud self update: the install command was empty'); + } + const managerArgs = command.slice(1); + + const projects = await readProjects({ + fs: context.fs, + clock: context.clock, + userDataDir: context.paths.userDataDir, + }); + + context.ui.content(`Current version: ${result.current}`); + context.ui.content(`Target version: ${target}`); + context.ui.content(`Install command: ${command.join(' ')}`); + context.ui.content( + projects.length === 0 + ? 'No registered projects to refresh.' + : `${projects.length} registered project${projects.length === 1 ? '' : 's'} will have their rules refreshed.`, + ); + + if (options.dryRun === true) { + context.ui.content('Dry run: nothing was changed.'); + return; + } + + if (options.force !== true) { + if (!deps.interactive) { + throw new UsageError( + `ailoud self update needs confirmation before installing ailoud ${target}, but there is ` + + 'no terminal to ask on. Re-run with --force to confirm in advance.', + ); + } + const confirmImpl = deps.confirmImpl ?? defaultConfirm; + const consented = await confirmImpl(`Install ailoud ${target}?`); + if (!consented) { + context.ui.content('Nothing was changed.'); + return; + } + } + + let code: number; + try { + if (deps.interactive) { + code = await deps.spawn(managerCommand, managerArgs); + } else { + // Only --force reaches here without a terminal (the gate above throws + // otherwise). runInteractive has no timeout and would hang forever if + // the manager wants to prompt, so this bounds the wait instead -- see + // FORCE_INSTALL_TIMEOUT_MS and this function's own doc comment. + const bounded = await deps.runCommand(managerCommand, managerArgs, { + timeoutMs: FORCE_INSTALL_TIMEOUT_MS, + }); + // run() buffers output rather than streaming it live the way + // runInteractive does, so it has to be printed after the fact -- this + // is the only way whoever (or whatever) is watching a --force, + // no-terminal run sees what the manager actually did. + if (bounded.stdout.length > 0) context.ui.content(bounded.stdout); + if (bounded.stderr.length > 0) context.ui.content(bounded.stderr); + code = bounded.code; + } + } catch (error) { + // The install spawn THROWING (ENOENT, or the bounded run's own timeout) + // is different from it exiting non-zero: the sweep is correctly skipped + // either way, but a throw used to reach no log line at all. + const reason = error instanceof Error ? error.message : String(error); + await logUpdateAction(context, `install threw for "${command.join(' ')}": ${reason}`); + throw error; + } + if (code !== 0) { + await logUpdateAction(context, `install failed, "${command.join(' ')}" exited ${code}`); + throw new FailureError(`ailoud self update: "${command.join(' ')}" exited with code ${code}`); + } + await logUpdateAction(context, `installed ${target}`); + context.ui.content(`ailoud updated to ${target}.`); + + // The NEW binary, as a fresh subprocess -- never this one. See this + // function's own doc comment for why that is not optional, and for why the + // command below is anchored rather than a bare 'ailoud'. + const sweep = await sweepCommandFor(method, deps.execPath, deps.run); + const sweepCommand = sweep?.[0]; + if (sweep === null || sweepCommand === undefined) { + context.ui.content('Could not determine the command to refresh rules automatically.'); + context.ui.content('Run it by hand: ailoud self sync'); + return; + } + const sweepArgs = sweep.slice(1); + try { + if (deps.interactive) { + await deps.spawn(sweepCommand, sweepArgs); + } else { + // Same reasoning as the install spawn above: deps.spawn is bound to + // runInteractive, which has no timeout and is only safe to call when a + // real terminal is attached to interrupt it. `self sync` never prompts + // for input, but "does not prompt" is not the same as "cannot stall" -- + // a registered project on a dead network mount is enough to hang this + // unconditionally, and this is the exact path (--force, no TTY) that + // used to reach it ungated. The bounded runCommand fails instead, after + // a generous wait, with its output printed either way. + const bounded = await deps.runCommand(sweepCommand, sweepArgs, { + timeoutMs: FORCE_SWEEP_TIMEOUT_MS, + }); + if (bounded.stdout.length > 0) context.ui.content(bounded.stdout); + if (bounded.stderr.length > 0) context.ui.content(bounded.stderr); + if (bounded.code !== 0) { + throw new FailureError(`"${sweep.join(' ')}" exited with code ${bounded.code}`); + } + } + } catch (error) { + const reason = error instanceof Error ? error.message : String(error); + context.ui.content(`Could not run "ailoud self sync" automatically (${reason}).`); + context.ui.content('Run it by hand: ailoud self sync'); + } +} + +interface SelfUpdateCliOptions { + readonly force?: boolean; + readonly dryRun?: boolean; +} + +export function registerSelfUpdate(parent: Command, context: CliContext): void { + parent + .command('update') + .option( + '--force', + 'skip confirmation; fail rather than refuse quietly when this install cannot be updated', + ) + .option('--dry-run', 'print the plan without installing or syncing anything') + .description('Install a newer ailoud, then refresh the rules block in every registered project') + .action(async (options: SelfUpdateCliOptions) => { + await context.ui.frame('Updating ailoud', async () => { + const deps: SelfUpdateDeps = { + context, + execPath: process.execPath, + // self.ts sits at /dist/commands/self.js once built; + // '../..' from there is the package root itself. + packageRoot: fileURLToPath(new URL('../..', import.meta.url)), + realpath, + run: boundedDetectRun(run), + spawn: runInteractive, + runCommand: run, + interactive: isInteractive(process.env, process.stdin.isTTY === true), + }; + await updateSelf(deps, { + force: options.force === true, + dryRun: options.dryRun === true, + }); + }); + }); +} diff --git a/apps/cli/src/commands/setup.test.ts b/apps/cli/src/commands/setup.test.ts index e232100..2814637 100644 --- a/apps/cli/src/commands/setup.test.ts +++ b/apps/cli/src/commands/setup.test.ts @@ -1002,6 +1002,7 @@ describe('runProvisioning', () => { }, }, llm: parseConfig(null).llm, + update: parseConfig(null).update, }; /** A failing check carrying `remedy`, shaped the way runChecks would emit it. */ @@ -1044,6 +1045,7 @@ describe('runProvisioning', () => { dbFile: join(tmp, 'data', 'ailoud.db'), mediaRoot: join(tmp, 'data', 'media'), isProjectLibrary: false, + userDataDir: join(tmp, 'data'), }; await mkdir(paths.mediaRoot, { recursive: true }); for (const fn of Object.values(providers)) fn.mockReset(); @@ -1191,7 +1193,10 @@ describe('runProvisioning', () => { expect(shownAtConsent).toContain(' Runs: sudo apt-get update'); expect(shownAtConsent).toContain(' Runs: sudo apt-get install -y ffmpeg'); expect(providers.runInteractive).not.toHaveBeenCalled(); - expect(ctx.lines.at(-1)).toBe('Nothing was changed.'); + // Now routed through `ui.warn` (declining consent is a "nothing + // happened" outcome), which PlainUi renders with its "warning: " + // marker prefix. + expect(ctx.lines.at(-1)).toBe('warning: Nothing was changed.'); } finally { if (isTtyDescriptor === undefined) delete (process.stdin as { isTTY?: boolean }).isTTY; else Object.defineProperty(process.stdin, 'isTTY', isTtyDescriptor); @@ -1218,7 +1223,10 @@ describe('runProvisioning', () => { await expect(runProvisioning(ctx, {}, checks, 'linux')).rejects.toThrow(EnvironmentError); - expect(ctx.lines.at(-1)).toBe('Nothing was changed.'); + // Now routed through `ui.warn` (declining consent is a "nothing + // happened" outcome), which PlainUi renders with its "warning: " + // marker prefix. + expect(ctx.lines.at(-1)).toBe('warning: Nothing was changed.'); expect(providers.runInteractive).not.toHaveBeenCalled(); expect(providers.downloadFile).not.toHaveBeenCalled(); } finally { diff --git a/apps/cli/src/commands/setup.ts b/apps/cli/src/commands/setup.ts index 7192d8d..6591f06 100644 --- a/apps/cli/src/commands/setup.ts +++ b/apps/cli/src/commands/setup.ts @@ -436,15 +436,15 @@ export function unfixableChecks(checks: readonly Check[]): readonly Check[] { /** Names the checks provisioning will not touch, with the human fix each carries. */ function reportUnfixable(context: CliContext, checks: readonly Check[]): void { - context.write( + context.ui.warn( checks.length === 1 ? 'One check failed, and it is not something ailoud can repair automatically:' : `${checks.length} checks failed, and none of them are something ailoud can repair ` + 'automatically:', ); for (const check of checks) { - context.write(`FAILED ${check.name} -- ${check.detail}`); - if (check.fix !== undefined) context.write(` ${check.fix}`); + context.ui.warn(`FAILED ${check.name} -- ${check.detail}`); + if (check.fix !== undefined) context.ui.warn(` ${check.fix}`); } } @@ -490,7 +490,7 @@ export async function runProvisioning( // means both entry points refuse first and spend nothing, and neither can // drift away from it again. if (platform === 'win32') { - for (const line of windowsManualSteps(commandName)) context.write(line); + for (const line of windowsManualSteps(commandName)) context.ui.content(line); throw new EnvironmentError( `ailoud ${commandName} cannot provision Windows: follow the manual steps above.`, ); @@ -507,7 +507,7 @@ export async function runProvisioning( remedies: collected, interactive, commandName, - note: (message) => context.write(message), + note: (message) => context.ui.note(message), }); const remedies = remediesForChoice(collected, llmChoice); @@ -519,7 +519,7 @@ export async function runProvisioning( // and nothing else. const unfixable = unfixableChecks(checks); if (unfixable.length === 0) { - context.write('Everything ailoud needs is already in place.'); + context.ui.note('Everything ailoud needs is already in place.'); return; } // `doctor --fix` already rendered the full check list (ui.checks) before @@ -551,11 +551,11 @@ export async function runProvisioning( manager, configFile: context.paths.configFile, }; - for (const line of describePlan(actions, env)) context.write(line); + for (const line of describePlan(actions, env)) context.ui.content(line); const consented = await requireConsent({ yes: options.yes === true, interactive, commandName }); if (!consented) { - context.write('Nothing was changed.'); + context.ui.warn('Nothing was changed.'); // Declining does not undo the checks that failed to get here: remedies // is non-empty at this point (the "nothing to fix" case above already // returned), so the environment is exactly as not-ready as it was before @@ -583,24 +583,28 @@ export async function runProvisioning( dataDir: context.paths.dataDir, manager, interactive, - onStep: (message) => context.write(message), + onStep: (message) => context.ui.note(message), // Coarse-grained on purpose: a line per percent would flood plain output, // and no spinner is used here (see provisionRunner.ts) so there is never // a live display for this to update instead. onProgress: (file, percent) => { - if (percent % 20 === 0) context.write(` ${file}: ${percent}%`); + if (percent % 20 === 0) context.ui.note(` ${file}: ${percent}%`); }, }); for (const outcome of result.outcomes) { - const status = outcome.ok ? 'ok' : 'FAILED'; - context.write(`${status} ${describeAction(outcome.action)} -- ${outcome.detail}`); + const line = `${describeAction(outcome.action)} -- ${outcome.detail}`; + if (outcome.ok) { + context.ui.success(line); + } else { + context.ui.warn(line); + } } const updatedKeys = Object.keys(result.updates); if (updatedKeys.length > 0) { await writeConfigUpdates(context.paths.configFile, result.updates); - context.write(`Updated ${context.paths.configFile}: ${updatedKeys.join(', ')}`); + context.ui.content(`Updated ${context.paths.configFile}: ${updatedKeys.join(', ')}`); } // Re-read unconditionally, even when result.updates was empty: an action diff --git a/apps/cli/src/commands/template.ts b/apps/cli/src/commands/template.ts index 4af8395..a9c97d8 100644 --- a/apps/cli/src/commands/template.ts +++ b/apps/cli/src/commands/template.ts @@ -116,8 +116,8 @@ export function registerTemplate(parent: Command, context: CliContext): void { summary: options.summary ?? base?.summary ?? safe, }), ); - context.write(`Wrote ${path}`); - context.write(`Use it with: ailoud audio summarize --template ${safe}`); + context.ui.success(`Wrote ${path}`); + context.ui.note(`Use it with: ailoud audio summarize --template ${safe}`); }); }); } diff --git a/apps/cli/src/commands/testContext.ts b/apps/cli/src/commands/testContext.ts index 477a762..f85275c 100644 --- a/apps/cli/src/commands/testContext.ts +++ b/apps/cli/src/commands/testContext.ts @@ -1,4 +1,11 @@ -import type { Diarizer, SpeechSegmenter, Summarizer, TranscriptionProvider } from '@ailoud/core'; +import type { + Diarizer, + PublishedVersion, + SpeechSegmenter, + Summarizer, + TranscriptionProvider, + VersionSource, +} from '@ailoud/core'; import { parseConfig } from '../config.js'; import { FakeAudioTool, @@ -50,6 +57,7 @@ export function context(): CliContext & { dbFile: '/d/ailoud.db', mediaRoot: '/d/media', isProjectLibrary: false, + userDataDir: '/d', }, config: { stt: { @@ -67,6 +75,7 @@ export function context(): CliContext & { }, }, llm: parseConfig(null).llm, + update: parseConfig(null).update, }, store: new InMemoryStore(), fs: new MemFs({ [FIXTURE_PATH]: 'AUDIO' }), @@ -112,6 +121,15 @@ export function context(): CliContext & { }; return summarizer; }, + // Reports no update by default -- a fixed list, never the network. + // Specs that care about `self check` override this field directly. + versionSource: { + published: async (): Promise => [ + { version: '1.0.0', deprecated: false }, + ], + } satisfies VersionSource, + updateRegistryHost: 'registry.npmjs.org', + updateTimeoutMs: 10_000, }; } diff --git a/apps/cli/src/config.test.ts b/apps/cli/src/config.test.ts index dda0b81..81c9c34 100644 --- a/apps/cli/src/config.test.ts +++ b/apps/cli/src/config.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest'; import { EnvironmentError } from '@ailoud/core'; -import { parseConfig, resolvePaths } from './config.js'; +import { ConfigSchema, parseConfig, resolvePaths } from './config.js'; describe('resolvePaths', () => { it('honours both XDG variables', () => { @@ -10,6 +10,7 @@ describe('resolvePaths', () => { dbFile: '/d/ailoud/ailoud.db', mediaRoot: '/d/ailoud/media', isProjectLibrary: false, + userDataDir: '/d/ailoud', }); }); @@ -20,6 +21,7 @@ describe('resolvePaths', () => { dbFile: '/h/.local/share/ailoud/ailoud.db', mediaRoot: '/h/.local/share/ailoud/media', isProjectLibrary: false, + userDataDir: '/h/.local/share/ailoud', }); }); @@ -27,6 +29,18 @@ describe('resolvePaths', () => { expect(() => resolvePaths({})).toThrow(/HOME/); expect(() => resolvePaths({})).toThrow(EnvironmentError); }); + + it('reports the per-user data directory even inside a project', () => { + const paths = resolvePaths( + { HOME: '/home/x', XDG_DATA_HOME: '/home/x/.local/share' }, + { cwd: '/repo/sub', exists: (p) => p === '/repo/.ailoud' }, + ); + // dataDir follows the project; userDataDir must not. The registry, the + // update-check cache and the log all live per user, and a registry inside + // a project would list only that project. + expect(paths.dataDir).toBe('/repo/.ailoud'); + expect(paths.userDataDir).toBe('/home/x/.local/share/ailoud'); + }); }); describe('parseConfig', () => { @@ -77,9 +91,16 @@ describe('parseConfig', () => { contextTokens: 200_000, }, }, + update: { + check: true, + }, }); }); + it('defaults the update check to on', () => { + expect(ConfigSchema.parse({}).update.check).toBe(true); + }); + it('reads the whisper binary and model', () => { const config = parseConfig( 'stt:\n provider: whisper-cpp\n whisperCpp:\n binary: /opt/whisper\n model: /m/base.bin\n', diff --git a/apps/cli/src/config.ts b/apps/cli/src/config.ts index 4167469..e8789d0 100644 --- a/apps/cli/src/config.ts +++ b/apps/cli/src/config.ts @@ -9,7 +9,7 @@ import { EnvironmentError, LLM_PROVIDERS, UsageError } from '@ailoud/core'; // contradicts the requirement that `.default({})` fills in the rest, so // nested objects use `.prefault()` instead, which re-parses the default // value through the inner schema (the pre-Zod-4 `.default()` behaviour). -const ConfigSchema = z.object({ +export const ConfigSchema = z.object({ stt: z .object({ provider: z.enum(['whisper-cpp']).default('whisper-cpp'), @@ -84,6 +84,12 @@ const ConfigSchema = z.object({ .prefault({}), }) .prefault({}), + update: z + .object({ + /** Look for a newer version once a day and mention it after a command. */ + check: z.boolean().default(true), + }) + .prefault({}), }); export type AiloudConfig = z.infer; @@ -95,6 +101,14 @@ export interface AiloudPaths { readonly mediaRoot: string; /** True when the library came from a project's `.ailoud/`, not the user's home. */ readonly isProjectLibrary: boolean; + /** + * The user's own `/ailoud`, regardless of `dataDir`. Things that + * are properties of the user rather than of a project -- the registry of + * projects ailoud has been used in, the update-check cache, the update log + * -- read and write here so that being inside a project library never + * scopes them down to that one project. + */ + readonly userDataDir: string; } /** The directory name a project uses to keep its own library. */ @@ -182,6 +196,7 @@ export function resolvePaths( dbFile: `${dataDir}/ailoud.db`, mediaRoot: `${dataDir}/media`, isProjectLibrary: project !== null, + userDataDir: `${dataHome}/ailoud`, }; } diff --git a/apps/cli/src/mcp/install.test.ts b/apps/cli/src/mcp/install.test.ts index d548397..10d055c 100644 --- a/apps/cli/src/mcp/install.test.ts +++ b/apps/cli/src/mcp/install.test.ts @@ -160,3 +160,43 @@ describe('ensureProjectLibrary', () => { expect(await fs.readTextFile(`${CWD}/.ailoud/.gitignore`)).toContain('notes.md'); }); }); + +describe('the rules file is written atomically', () => { + /** Records the order of writes and renames, so the mechanism is checkable. */ + class RecordingFs extends MemFs { + public readonly calls: string[] = []; + public override async writeTextFile(path: string, content: string): Promise { + this.calls.push(`write:${path}`); + return super.writeTextFile(path, content); + } + public override async rename(from: string, to: string): Promise { + this.calls.push(`rename:${from}->${to}`); + return super.rename(from, to); + } + } + + it('writes a temporary file and renames it over the target', async () => { + // The MECHANISM is what this asserts, deliberately. The defect it guards + // against -- `writeTextFile` truncating the target before a failed write + // empties it -- cannot be reproduced with `MemFs`, which either writes or + // throws atomically. Truncation is a property of the real POSIX + // `open(path, 'w')`. + // + // It was demonstrated on a real filesystem instead: on a full 1 MB + // volume, a plain write turned a 25-byte hand-written CLAUDE.md into 0 + // bytes with ENOSPC, while temp-then-rename left it byte-identical. That + // is why this pattern is here, and `self sync` sweeping this writer + // across every registered project unattended is why it matters. + const fs = new RecordingFs({ '/proj/CLAUDE.md': '# My own notes\n' }); + + await install(fs, findAgent('claude')!, 'local', '/home/x', '/proj'); + + const rules = fs.calls.filter((call) => call.includes('CLAUDE.md')); + expect(rules.some((call) => call.startsWith('write:') && call.includes('.tmp'))).toBe(true); + expect( + rules.some((call) => call.startsWith('rename:') && call.endsWith('->/proj/CLAUDE.md')), + ).toBe(true); + // And never a direct write to the target itself. + expect(rules).not.toContain('write:/proj/CLAUDE.md'); + }); +}); diff --git a/apps/cli/src/mcp/install.ts b/apps/cli/src/mcp/install.ts index 1047ea2..fcc9820 100644 --- a/apps/cli/src/mcp/install.ts +++ b/apps/cli/src/mcp/install.ts @@ -1,3 +1,4 @@ +import { randomUUID } from 'node:crypto'; import { dirname, join } from 'node:path'; import type { Fs } from '@ailoud/core'; import { PROJECT_DIR } from '../config.js'; @@ -54,9 +55,36 @@ async function readIfPresent(fs: Fs, path: string): Promise { return (await fs.exists(path)) ? fs.readTextFile(path) : null; } +/** + * Writes a file without ever leaving it half-written: a temporary file beside + * it, then a rename over the top. + * + * `writeTextFile` truncates before it writes, so a failure part-way through -- + * ENOSPC is the realistic one -- leaves the target EMPTY. That was survivable + * while these files were only touched by an interactive `mcp install` the user + * was watching. It is not survivable now: `self sync` sweeps this writer + * across every registered project unattended, and the file it rewrites is + * often a repository's own hand-written `CLAUDE.md` or `AGENTS.md`. Truncating + * one of those and then reporting `failed` destroys the user's content while + * telling them nothing happened. + * + * Same pattern as `writeRegistry` in `apps/cli/src/projects.ts`, and for the + * same reason. The temporary name is randomised so two concurrent writers + * cannot corrupt each other's, and it sits in the target's own directory so + * the rename stays on one filesystem and therefore stays atomic. + */ async function write(fs: Fs, path: string, content: string): Promise { await fs.ensureDir(dirname(path)); - await fs.writeTextFile(path, content); + const temp = `${path}.${randomUUID()}.tmp`; + try { + await fs.writeTextFile(temp, content); + } catch (error) { + // The target has not been touched yet, so there is nothing to undo. Clear + // the partial temporary file rather than leaving litter beside a config. + await fs.removeFile(temp); + throw error; + } + await fs.rename(temp, path); } /** diff --git a/apps/cli/src/program.test.ts b/apps/cli/src/program.test.ts index 9e68e16..97e7148 100644 --- a/apps/cli/src/program.test.ts +++ b/apps/cli/src/program.test.ts @@ -1,3 +1,4 @@ +import type { Command } from 'commander'; import { afterEach, describe, expect, it } from 'vitest'; import { parseConfig } from './config.js'; import { EnvironmentError, FailureError, UsageError } from '@ailoud/core'; @@ -14,6 +15,8 @@ import { SqliteStore } from '@ailoud/providers'; import { buildProgram, exitCodeFor } from './program.js'; import type { CliContext } from './wiring.js'; import { PlainUi } from './ui/plain.js'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; describe('exitCodeFor', () => { it('maps each domain error to its documented code', () => { @@ -109,6 +112,7 @@ describe('buildProgram', () => { dbFile: ':memory:', mediaRoot: '/fake/data/media', isProjectLibrary: false, + userDataDir: '/fake/data', }, config: { stt: { @@ -128,6 +132,7 @@ describe('buildProgram', () => { }, }, llm: parseConfig(null).llm, + update: parseConfig(null).update, }, store, fs: new MemFs(), @@ -145,6 +150,11 @@ describe('buildProgram', () => { contextTokens: 8192, complete: async () => 'x', }), + // A fixed list, never the network: this suite drives buildProgram + // end to end, and no test here exercises `self check` itself. + versionSource: { published: async () => [{ version: '1.0.0', deprecated: false }] }, + updateRegistryHost: 'registry.npmjs.org', + updateTimeoutMs: 10_000, }; } @@ -154,15 +164,41 @@ describe('buildProgram', () => { expect(program.description()).toContain('audio-to-text'); }); + it('reports the version in its own manifest', () => { + // It used to report the literal 0.0.0, so the published 1.0.0-dev.1 + // answered `ailoud --version` with 0.0.0 and told MCP clients the same. + // Asserting against the manifest rather than a literal keeps this test + // from needing an edit at every release -- which is what would make it + // rot into agreeing with whatever is there. + const manifest: unknown = JSON.parse( + readFileSync(fileURLToPath(new URL('../package.json', import.meta.url)), 'utf8'), + ); + const version = (manifest as { version: string }).version; + expect(version).toMatch(/^\d+\.\d+\.\d+/); + expect(buildProgram(makeContext()).version()).toBe(version); + }); + + /** + * Commander writes its usage errors straight to stderr, which is right for + * a CLI and wrong for a test run: two of the tests below made every + * `pnpm test` print "error: unknown option", so a real error in the log had + * to be picked out of expected noise. Only stderr is silenced -- writeOut is + * how buildProgram routes help through context.write, which one of these + * tests asserts on. + */ + function quiet(program: Command): Command { + return program.configureOutput({ writeErr: () => {} }); + } + it('throws instead of exiting the process on an unknown flag', async () => { - const program = buildProgram(makeContext()); + const program = quiet(buildProgram(makeContext())); await expect(program.parseAsync(['node', 'ailoud', '--bogus'])).rejects.toMatchObject({ code: 'commander.unknownOption', }); }); it('maps an unknown flag to exit code 2 end to end', async () => { - const program = buildProgram(makeContext()); + const program = quiet(buildProgram(makeContext())); const error: unknown = await program .parseAsync(['node', 'ailoud', '--bogus']) .catch((caught: unknown) => caught); @@ -186,7 +222,7 @@ describe('buildProgram', () => { // raise 'commander.help' with its own exitCode of 1. That is a usage // error (nothing was told what to do), not a normal failure, so it must // map to 2 here, not fall through as 1. - const program = buildProgram(makeContext()); + const program = quiet(buildProgram(makeContext())); const error: unknown = await program .parseAsync(['node', 'ailoud']) .catch((caught: unknown) => caught); diff --git a/apps/cli/src/program.ts b/apps/cli/src/program.ts index 64f36d6..9c78e20 100644 --- a/apps/cli/src/program.ts +++ b/apps/cli/src/program.ts @@ -14,7 +14,9 @@ import { registerSearch } from './commands/search.js'; import { registerMcp } from './commands/mcp.js'; import { attachLetters, group, inGroupAndTopLevel } from './commands/groups.js'; import { registerTranscribe } from './commands/transcribe.js'; +import { registerSelfCheck, registerSelfSync, registerSelfUpdate } from './commands/self.js'; import type { CliContext } from './wiring.js'; +import { VERSION } from './version.js'; /** * Reads the commander error code off an unknown thrown value without @@ -57,7 +59,7 @@ export function buildProgram(context: CliContext): Command { program .name('ailoud') .description('Multilingual audio-to-text with a local recording library') - .version('0.0.0') + .version(VERSION) .exitOverride(); // throw instead of calling process.exit // Commander's own help/version text stays plain in both modes: routed // through context.write, not context.ui, so it is never decorated. @@ -100,5 +102,12 @@ export function buildProgram(context: CliContext): Command { registerMcp(program, context); registerDoctor(program, context); registerSetup(program, context); + + const self = group(program, 'self', undefined, 'Manage this installation of ailoud'); + inGroupAndTopLevel(program, self, registerSelfCheck, context); + inGroupAndTopLevel(program, self, registerSelfUpdate, context); + inGroupAndTopLevel(program, self, registerSelfSync, context); + attachLetters(self); + return program; } diff --git a/apps/cli/src/projects.test.ts b/apps/cli/src/projects.test.ts new file mode 100644 index 0000000..180a73f --- /dev/null +++ b/apps/cli/src/projects.test.ts @@ -0,0 +1,314 @@ +import { describe, expect, it } from 'vitest'; +import type { Clock } from '@ailoud/core'; +import { MemFs } from '@ailoud/core/testing'; +import { + pruneProjects, + readProjects, + registryPath, + rememberProject, + type ProjectsDeps, +} from './projects.js'; + +const DATA_DIR = '/data/ailoud'; +const DAY_MS = 24 * 60 * 60 * 1000; + +class StubClock implements Clock { + public constructor(private ms: number) {} + public nowIso(): string { + return new Date(this.ms).toISOString(); + } + public advance(ms: number): void { + this.ms += ms; + } +} + +/** A `MemFs` that records every write/rename/removeFile call, in order. */ +class LoggingFs extends MemFs { + readonly calls: string[] = []; + override async writeTextFile(path: string, content: string): Promise { + this.calls.push(`write:${path}`); + return super.writeTextFile(path, content); + } + override async rename(from: string, to: string): Promise { + this.calls.push(`rename:${from}->${to}`); + return super.rename(from, to); + } + override async removeFile(path: string): Promise { + this.calls.push(`removeFile:${path}`); + return super.removeFile(path); + } +} + +function deps(overrides: Partial = {}): ProjectsDeps { + return { + fs: new MemFs({}), + clock: new StubClock(Date.parse('2026-01-01T00:00:00.000Z')), + userDataDir: DATA_DIR, + ...overrides, + }; +} + +describe('rememberProject / readProjects', () => { + it('creates the file on first registration', async () => { + const d = deps(); + + await rememberProject(d, { path: '/proj/a' }); + + expect(await d.fs.exists(registryPath(DATA_DIR))).toBe(true); + const projects = await readProjects(d); + expect(projects).toEqual([ + { + path: '/proj/a', + firstSeen: '2026-01-01T00:00:00.000Z', + lastSeen: '2026-01-01T00:00:00.000Z', + }, + ]); + }); + + it('records firstSeen once and moves lastSeen', async () => { + const clock = new StubClock(Date.parse('2026-01-01T00:00:00.000Z')); + const d = deps({ clock }); + + await rememberProject(d, { path: '/proj/a' }); + clock.advance(DAY_MS + 1); // comfortably past the 24-hour throttle + await rememberProject(d, { path: '/proj/a' }); + + const [entry] = await readProjects(d); + expect(entry).toBeDefined(); + expect(entry?.firstSeen).toBe('2026-01-01T00:00:00.000Z'); + expect(entry?.lastSeen).toBe( + new Date(Date.parse('2026-01-01T00:00:00.000Z') + DAY_MS + 1).toISOString(), + ); + }); + + it('does not write again within 24 hours', async () => { + const clock = new StubClock(Date.parse('2026-01-01T00:00:00.000Z')); + const fs = new LoggingFs({}); + const d = deps({ clock, fs }); + + await rememberProject(d, { path: '/proj/a' }); + const writesAfterFirstCall = fs.calls.length; + + clock.advance(1000); // one second later: well within the 24-hour window + await rememberProject(d, { path: '/proj/a' }); + + expect(fs.calls.length).toBe(writesAfterFirstCall); // no additional write or rename + const [entry] = await readProjects(d); + expect(entry?.lastSeen).toBe('2026-01-01T00:00:00.000Z'); // unchanged + }); + + it('writes again once lastSeen is older than 24 hours', async () => { + const clock = new StubClock(Date.parse('2026-01-01T00:00:00.000Z')); + const fs = new LoggingFs({}); + const d = deps({ clock, fs }); + + await rememberProject(d, { path: '/proj/a' }); + const writesAfterFirstCall = fs.calls.length; + + clock.advance(DAY_MS); // exactly 24 hours later + await rememberProject(d, { path: '/proj/a' }); + + expect(fs.calls.length).toBeGreaterThan(writesAfterFirstCall); + const [entry] = await readProjects(d); + expect(entry?.lastSeen).toBe( + new Date(Date.parse('2026-01-01T00:00:00.000Z') + DAY_MS).toISOString(), + ); + }); + + it('records rulesVersion when rules were written', async () => { + const clock = new StubClock(Date.parse('2026-01-01T00:00:00.000Z')); + const d = deps({ clock }); + + await rememberProject(d, { path: '/proj/a', rulesVersion: '1.2.0' }); + const [afterFirst] = await readProjects(d); + expect(afterFirst?.rulesVersion).toBe('1.2.0'); + + // A rules write is significant enough to record immediately, even though + // the 24-hour throttle would otherwise skip a plain touch. + clock.advance(1000); + await rememberProject(d, { path: '/proj/a', rulesVersion: '1.3.0' }); + const [afterSecond] = await readProjects(d); + expect(afterSecond?.rulesVersion).toBe('1.3.0'); + expect(afterSecond?.lastSeen).toBe(clock.nowIso()); + }); + + it('writes through a temporary file and renames', async () => { + const fs = new LoggingFs({}); + const d = deps({ fs }); + + await rememberProject(d, { path: '/proj/a' }); + + const writeIndex = fs.calls.findIndex((c) => c.startsWith('write:')); + const renameIndex = fs.calls.findIndex((c) => c.startsWith('rename:')); + expect(writeIndex).toBeGreaterThanOrEqual(0); + expect(renameIndex).toBeGreaterThan(writeIndex); // write happens strictly before rename + + const writtenPath = fs.calls[writeIndex]!.slice('write:'.length); + const [, renamedFrom, renamedTo] = fs.calls[renameIndex]!.match(/^rename:(.*)->(.*)$/) ?? []; + expect(renamedFrom).toBe(writtenPath); + expect(renamedTo).toBe(registryPath(DATA_DIR)); + expect(writtenPath).not.toBe(registryPath(DATA_DIR)); // a real, distinct temp file + // Nothing observed the target path directly written to; it only appears + // as the rename's destination. + expect(fs.calls.some((c) => c === `write:${registryPath(DATA_DIR)}`)).toBe(false); + }); + + it('recovers from a corrupt file by moving it aside', async () => { + const fs = new MemFs({ [registryPath(DATA_DIR)]: '{not valid json' }); + const d = deps({ fs }); + + expect(await readProjects(d)).toEqual([]); + expect(await fs.exists(`${DATA_DIR}/projects.json.bad`)).toBe(true); + expect(await fs.exists(registryPath(DATA_DIR))).toBe(false); // moved, not copied + }); + + it('recovers from a file that parses as JSON but not as a registry', async () => { + const fs = new MemFs({ [registryPath(DATA_DIR)]: JSON.stringify({ oops: true }) }); + const d = deps({ fs }); + + expect(await readProjects(d)).toEqual([]); + expect(await fs.exists(`${DATA_DIR}/projects.json.bad`)).toBe(true); + }); +}); + +describe('pruneProjects', () => { + async function seedRegistry(fs: MemFs, entries: unknown[]): Promise { + await fs.ensureDir(DATA_DIR); + await fs.writeTextFile(registryPath(DATA_DIR), JSON.stringify(entries)); + } + + it('prunes an entry whose directory is gone and reports it', async () => { + const fs = new MemFs({}); + await seedRegistry(fs, [ + { + path: '/gone', + firstSeen: '2026-01-01T00:00:00.000Z', + lastSeen: '2026-01-01T00:00:00.000Z', + }, + ]); + // '/gone' is deliberately absent from fs.dirs. + const d = deps({ fs }); + + const dropped = await pruneProjects(d); + + expect(dropped).toHaveLength(1); + expect(dropped[0]?.path).toBe('/gone'); + expect(await readProjects(d)).toEqual([]); + }); + + it('keeps an entry whose .ailoud is gone but whose directory remains', async () => { + const fs = new MemFs({}); + fs.dirs.add('/proj/b'); // the project directory still exists... + // ...but its library, '/proj/b/.ailoud', is never added: it is gone. + await seedRegistry(fs, [ + { + path: '/proj/b', + libraryDir: '/proj/b/.ailoud', + firstSeen: '2026-01-01T00:00:00.000Z', + lastSeen: '2026-01-01T00:00:00.000Z', + }, + ]); + const d = deps({ fs }); + + const dropped = await pruneProjects(d); + + expect(dropped).toEqual([]); + const projects = await readProjects(d); + expect(projects).toHaveLength(1); + expect(projects[0]?.path).toBe('/proj/b'); + }); + + it('never deletes anything on disk while pruning', async () => { + const fs = new LoggingFs({}); + fs.dirs.add('/proj/kept'); + // '/proj/gone' is absent, so it will be dropped from the registry. + await seedRegistry(fs, [ + { + path: '/proj/kept', + firstSeen: '2026-01-01T00:00:00.000Z', + lastSeen: '2026-01-01T00:00:00.000Z', + }, + { + path: '/proj/gone', + firstSeen: '2026-01-01T00:00:00.000Z', + lastSeen: '2026-01-01T00:00:00.000Z', + }, + ]); + const d = deps({ fs }); + + const dropped = await pruneProjects(d); + + expect(dropped).toHaveLength(1); + expect(fs.calls.some((c) => c.startsWith('removeFile:'))).toBe(false); + }); +}); + +describe('the three concurrency and throttle defects found in review', () => { + it('keeps an entry another process committed while we were writing', async () => { + // Two processes registering DIFFERENT projects both read the registry, + // both write, and the second rename wins -- so the first project used to + // vanish outright, not merely with a stale lastSeen. The merge re-reads + // immediately before the rename and keeps paths it had not seen. + class RacingFs extends MemFs { + private reads = 0; + override async readTextFile(path: string): Promise { + this.reads += 1; + // The second read is the one inside writeRegistry, standing in for a + // rival process that committed /proj/b in the meantime. + if (this.reads === 2 && path === registryPath(DATA_DIR)) { + return `${JSON.stringify([ + { + path: '/proj/b', + firstSeen: '2026-01-01T00:00:00.000Z', + lastSeen: '2026-01-01T00:00:00.000Z', + }, + ])}\n`; + } + return super.readTextFile(path); + } + } + const fs = new RacingFs({}); + await fs.writeTextFile(registryPath(DATA_DIR), '[]\n'); + const d = deps({ fs }); + + await rememberProject(d, { path: '/proj/a' }); + + const paths = (await readProjects(d)).map((entry) => entry.path).sort(); + expect(paths).toEqual(['/proj/a', '/proj/b']); + }); + + it('does not fail the command when quarantining a corrupt registry fails', async () => { + // Two processes meeting one corrupt registry both try to move it aside. + // The loser's source is already gone, so rename answers ENOENT. Letting + // that escape would turn a bad bookkeeping file into a broken command, + // which is the one thing this path exists to prevent. + class UnmovableFs extends MemFs { + override async rename(): Promise { + throw new Error('ENOENT: no such file or directory'); + } + } + const fs = new UnmovableFs({}); + await fs.writeTextFile(registryPath(DATA_DIR), '{ not json'); + const d = deps({ fs }); + + await expect(readProjects(d)).resolves.toEqual([]); + }); + + it('does not bypass the throttle when the rules version has not changed', async () => { + // Presence is not enough: a caller passing the same version on every + // touch would write on every command, which is the throttle gone. + const fs = new LoggingFs({}); + const clock = new StubClock(Date.parse('2026-01-01T00:00:00.000Z')); + const d = deps({ fs, clock }); + + await rememberProject(d, { path: '/proj/a', rulesVersion: '1.0.0' }); + const afterFirst = fs.calls.length; + clock.advance(1000); + await rememberProject(d, { path: '/proj/a', rulesVersion: '1.0.0' }); + expect(fs.calls.length).toBe(afterFirst); + + // A CHANGED version is a real event and must be recorded at once. + await rememberProject(d, { path: '/proj/a', rulesVersion: '1.0.1' }); + expect(fs.calls.length).toBeGreaterThan(afterFirst); + }); +}); diff --git a/apps/cli/src/projects.ts b/apps/cli/src/projects.ts new file mode 100644 index 0000000..1be2420 --- /dev/null +++ b/apps/cli/src/projects.ts @@ -0,0 +1,214 @@ +import { randomUUID } from 'node:crypto'; +import { z } from 'zod'; +import type { Clock, Fs } from '@ailoud/core'; + +/** One project ailoud has been used in, as remembered across runs. */ +export interface ProjectEntry { + readonly path: string; + readonly libraryDir?: string; + readonly firstSeen: string; + readonly lastSeen: string; + readonly rulesVersion?: string; +} + +export interface ProjectsDeps { + readonly fs: Fs; + readonly clock: Clock; + /** + * The per-user data directory (`AiloudPaths.userDataDir`), never + * `AiloudPaths.dataDir`. `dataDir` follows the current project's `.ailoud/` + * when there is one, and a registry stored inside one project would only + * ever list that project. + */ + readonly userDataDir: string; +} + +/** How long a registration is left untouched before a re-run bumps `lastSeen` again. */ +const REFRESH_AFTER_MS = 24 * 60 * 60 * 1000; + +const ProjectEntrySchema = z.object({ + path: z.string(), + libraryDir: z.string().optional(), + firstSeen: z.string(), + lastSeen: z.string(), + rulesVersion: z.string().optional(), +}); + +const RegistrySchema = z.array(ProjectEntrySchema); + +export function registryPath(userDataDir: string): string { + return `${userDataDir}/projects.json`; +} + +/** Where a registry that failed to parse is moved, so it never blocks a run. */ +function quarantinePath(userDataDir: string): string { + return `${userDataDir}/projects.json.bad`; +} + +/** + * Writes the registry atomically: a temporary file beside the real one, then + * a rename over the top. Writing in place would let a concurrent reader see a + * half-written file, and the temporary name is randomised per call so two + * writers never share -- and corrupt -- one temporary file. + * + * `merge` exists because a plain read-modify-write loses more than a + * timestamp. Two processes registering DIFFERENT projects both read the + * registry, both write, and the second rename wins -- so the first project + * disappears entirely, not merely with a stale `lastSeen`. Re-reading + * immediately before the rename and keeping any path we had not seen narrows + * that window to the rename itself, and a project lost inside it re-registers + * the next time ailoud runs there. + * + * Pruning passes `merge: false`, deliberately: it is REMOVING entries, and + * merging would read the very entries it just dropped back in. + */ +async function writeRegistry( + deps: ProjectsDeps, + entries: readonly ProjectEntry[], + options: { readonly merge: boolean }, +): Promise { + const path = registryPath(deps.userDataDir); + let next = entries; + if (options.merge) { + const mine = new Set(entries.map((entry) => entry.path)); + const latest = await readProjects(deps); + const theirs = latest.filter((entry) => !mine.has(entry.path)); + next = [...entries, ...theirs]; + } + const tempPath = `${path}.${randomUUID()}.tmp`; + await deps.fs.ensureDir(deps.userDataDir); + await deps.fs.writeTextFile(tempPath, `${JSON.stringify(next, null, 2)}\n`); + await deps.fs.rename(tempPath, path); +} + +/** + * Moves an unparseable registry aside, and never throws doing it. + * + * The move itself can fail: two processes reading the same corrupt registry + * both try it, and the loser's source is already gone, so `rename` answers + * ENOENT. Letting that escape would crash the command -- turning a bad + * bookkeeping file into a broken `ailoud ls`, which is exactly what this + * whole quarantine path exists to prevent. + */ +async function quarantine(deps: ProjectsDeps, path: string): Promise { + try { + await deps.fs.rename(path, quarantinePath(deps.userDataDir)); + } catch { + // Someone else moved it, or the directory is not writable. Either way the + // caller gets an empty list and the command carries on. + } +} + +/** + * Every project ailoud has been used in. + * + * A registry is a convenience, not a source of truth: a file that fails to + * parse must never fail the command that asked for it. It is instead moved + * aside to `projects.json.bad` so a human can look at it, and reading + * proceeds as though there were no registry yet. + */ +export async function readProjects(deps: ProjectsDeps): Promise { + const path = registryPath(deps.userDataDir); + if (!(await deps.fs.exists(path))) return []; + + const raw = await deps.fs.readTextFile(path); + let document: unknown; + try { + document = JSON.parse(raw); + } catch { + await quarantine(deps, path); + return []; + } + + const result = RegistrySchema.safeParse(document); + if (!result.success) { + await quarantine(deps, path); + return []; + } + return result.data; +} + +/** + * Records that a project was just used, creating the registry on first use. + * + * Skips the write when the project was already seen within the last 24 + * hours: the hot path is a command that touches a project library reading + * this file on every run, and writing on every run would put a disk write on + * something as routine as `ailoud ls`. `rulesVersion` is the exception -- + * when it is given, rules were just written to the project, which is worth + * recording immediately rather than waiting out the throttle. + */ +export async function rememberProject( + deps: ProjectsDeps, + project: { path: string; libraryDir?: string; rulesVersion?: string }, +): Promise { + const projects = await readProjects(deps); + const now = deps.clock.nowIso(); + const index = projects.findIndex((entry) => entry.path === project.path); + + if (index === -1) { + const entry: ProjectEntry = { + path: project.path, + firstSeen: now, + lastSeen: now, + ...(project.libraryDir === undefined ? {} : { libraryDir: project.libraryDir }), + ...(project.rulesVersion === undefined ? {} : { rulesVersion: project.rulesVersion }), + }; + await writeRegistry(deps, [...projects, entry], { merge: true }); + return; + } + + const existing = projects[index]!; + const staleMs = Date.parse(now) - Date.parse(existing.lastSeen); + // Presence is not enough: a caller that passes the same version on every + // touch would bypass the throttle on every command, which is the throttle + // gone. Only a CHANGE means rules were just rewritten. + const rulesJustWritten = + project.rulesVersion !== undefined && project.rulesVersion !== existing.rulesVersion; + if (staleMs < REFRESH_AFTER_MS && !rulesJustWritten) return; + + const updated: ProjectEntry = { + ...existing, + lastSeen: now, + ...(project.libraryDir === undefined ? {} : { libraryDir: project.libraryDir }), + ...(project.rulesVersion === undefined ? {} : { rulesVersion: project.rulesVersion }), + }; + const next = [...projects]; + next[index] = updated; + await writeRegistry(deps, next, { merge: true }); +} + +/** + * Drops entries whose project directory is gone, and returns what was + * dropped so the caller can report it. + * + * Checked against `entry.path` itself, not a `.ailoud/` beneath it: a project + * whose library is gone can still hold the rules block sync exists to + * refresh, so losing the library is not reason enough to forget the project. + * Never removes anything on disk -- only the registry file is rewritten, and + * only when there is something to drop. + */ +export async function pruneProjects(deps: ProjectsDeps): Promise { + const projects = await readProjects(deps); + const kept: ProjectEntry[] = []; + const dropped: ProjectEntry[] = []; + for (const entry of projects) { + // A throw here is NOT evidence the project is gone. `isDirectory` rethrows + // EACCES -- revoked permissions, an unreachable network mount -- and + // dropping the entry on that would forget a project that still exists and + // still holds a rules block. So an unanswerable question keeps the entry, + // and only a definite "not a directory" drops it. + let present: boolean; + try { + present = await deps.fs.isDirectory(entry.path); + } catch { + present = true; + } + if (present) kept.push(entry); + else dropped.push(entry); + } + // merge: false, deliberately. Pruning REMOVES entries, and merging would + // re-read the very entries it just dropped straight back in. + if (dropped.length > 0) await writeRegistry(deps, kept, { merge: false }); + return dropped; +} diff --git a/apps/cli/src/setupLock.test.ts b/apps/cli/src/setupLock.test.ts index 78e5e25..60c6f00 100644 --- a/apps/cli/src/setupLock.test.ts +++ b/apps/cli/src/setupLock.test.ts @@ -1,5 +1,8 @@ import { describe, expect, it } from 'vitest'; -import { mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises'; +import { spawn } from 'node:child_process'; +import { existsSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import { lockPath, withProvisioningLock } from './setupLock.js'; @@ -72,6 +75,41 @@ describe('withProvisioningLock', () => { await expect(withProvisioningLock(dir, async () => 'took over')).resolves.toBe('took over'); }); + it('leaves no scratch file behind after taking over a stale lock', async () => { + // The takeover writes its own lock beside the path and renames over it, + // because `rm` then create loses the race it looks like it wins. A rename + // that failed, or a path built wrong, would leave the scratch file in the + // data directory. + const dir = await dataDir(); + await writeFile( + lockPath(dir), + JSON.stringify({ pid: 4_194_304, startedAt: '2020-01-01T00:00:00.000Z' }), + 'utf8', + ); + let duringBody: string[] = []; + await withProvisioningLock(dir, async () => { + duringBody = await readdir(dir); + }); + expect(duringBody).toEqual(['provisioning.lock']); + expect(await readdir(dir)).toEqual([]); + }); + + it('records the taking-over process as the holder, not the stale one', async () => { + const dir = await dataDir(); + await writeFile( + lockPath(dir), + JSON.stringify({ pid: 4_194_304, startedAt: '2020-01-01T00:00:00.000Z' }), + 'utf8', + ); + let held: unknown; + await withProvisioningLock(dir, async () => { + held = JSON.parse(await readFile(lockPath(dir), 'utf8')); + }); + // The read-back after the rename is what makes a losing takeover + // detectable; this is the winning side of it. + expect(held).toMatchObject({ pid: process.pid }); + }); + it('treats an empty lock file as stale', async () => { const dir = await dataDir(); // A run that died between creating the lock and writing to it. @@ -97,3 +135,82 @@ describe('withProvisioningLock', () => { await rm(dir, { recursive: true, force: true }); }); }); + +/** + * Two runs at once, against the COMPILED lock, with a stale lock in place. + * + * Every other test here is single-threaded, which is why none of them caught + * that the first two takeover implementations let both runs proceed -- a + * concurrency test found 34 overlaps in 60 attempts. This is that test, small + * enough to keep in the suite. + */ +describe('withProvisioningLock under contention', () => { + const ITERATIONS = 12; + const HOLD_MS = 120; + + it('lets exactly one of two simultaneous runs into the body', async () => { + const dist = fileURLToPath(new URL('../dist/setupLock.js', import.meta.url)); + expect(existsSync(dist)).toBe(true); // the gate builds before it tests + + const scratch = await mkdtemp(join(tmpdir(), 'ailoud-lock-race-')); + const worker = join(scratch, 'worker.mjs'); + await writeFile( + worker, + `import { appendFileSync } from "node:fs"; + import { withProvisioningLock } from ${JSON.stringify(dist)}; + const [dir, log] = process.argv.slice(2); + try { + await withProvisioningLock(dir, async () => { + appendFileSync(log, \`ENTER \${process.pid}\n\`); + await new Promise((r) => setTimeout(r, ${HOLD_MS})); + appendFileSync(log, \`LEAVE \${process.pid}\n\`); + }); + } catch { + appendFileSync(log, \`REFUSED \${process.pid}\n\`); + }`, + 'utf8', + ); + + let refusals = 0; + for (let round = 0; round < ITERATIONS; round += 1) { + const dir = join(scratch, `round-${round}`); + await mkdir(dir, { recursive: true }); + // A stale lock: pid 2^22 is above every default pid_max, so nothing + // holds it. This is the state a Ctrl-C or a crash leaves behind, and the + // only state in which a takeover happens at all. + await writeFile( + lockPath(dir), + JSON.stringify({ pid: 4_194_304, startedAt: '2020-01-01T00:00:00.000Z' }), + 'utf8', + ); + const log = join(dir, 'log.txt'); + await writeFile(log, '', 'utf8'); + + await Promise.all( + [0, 1].map( + () => + new Promise((resolve) => { + const child = spawn(process.execPath, [worker, dir, log], { stdio: 'ignore' }); + child.on('exit', () => resolve()); + }), + ), + ); + + const lines = (await readFile(log, 'utf8')).split('\n').filter(Boolean); + refusals += lines.filter((l) => l.startsWith('REFUSED')).length; + // Nesting is the failure: ENTER, ENTER means both are inside the body. + let inside = 0; + for (const line of lines) { + if (line.startsWith('ENTER')) inside += 1; + if (line.startsWith('LEAVE')) inside -= 1; + expect(inside, `round ${round}: ${lines.join(' | ')}`).toBeLessThanOrEqual(1); + } + expect(lines.filter((l) => l.startsWith('ENTER'))).toHaveLength(1); + } + + // Without this the test could pass by never contending at all. + expect(refusals).toBeGreaterThan(0); + + await rm(scratch, { recursive: true, force: true }); + }, 60_000); +}); diff --git a/apps/cli/src/setupLock.ts b/apps/cli/src/setupLock.ts index 014d7e8..de40670 100644 --- a/apps/cli/src/setupLock.ts +++ b/apps/cli/src/setupLock.ts @@ -65,6 +65,12 @@ async function readHolder(path: string): Promise { * would leave a window for the other process to win in between, which is * exactly the race being closed. * + * Taking over a stale lock cannot use `wx`, since the file is there, so the + * takeover runs under a second lock (`provisioning.lock.steal`) created with + * `wx`. That makes the takeover itself exclusive -- one winner, and the loser + * told to try again. Checking afterwards who won is not enough: two runs can + * each check after their own write and each see themselves. + * * A live lock is refused immediately rather than waited on. Provisioning is * interactive and can sit on a consent prompt for minutes, so a queued * second run would look like a hang. The refusal names the holder's pid and @@ -78,36 +84,99 @@ export async function withProvisioningLock(dataDir: string, body: () => Promi const path = lockPath(dataDir); await mkdir(dirname(path), { recursive: true }); - let handle; + const holder: LockHolder = { pid: process.pid, startedAt: new Date().toISOString() }; + const mine = JSON.stringify(holder); + try { - handle = await open(path, 'wx'); + const handle = await open(path, 'wx'); + try { + await handle.writeFile(mine, 'utf8'); + } finally { + await handle.close(); + } } catch (error) { if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error; - const holder = await readHolder(path); - if (holder !== null && isRunning(holder.pid)) { + const existing = await readHolder(path); + if (existing !== null && isRunning(existing.pid)) { throw new FailureError( - `another ailoud provisioning run is already in progress (pid ${holder.pid}, started ` + - `${holder.startedAt}). Wait for it to finish, or stop it, then try again.`, + `another ailoud provisioning run is already in progress (pid ${existing.pid}, started ` + + `${existing.startedAt}). Wait for it to finish, or stop it, then try again.`, ); } + // Stale: the holder is gone, or never finished writing who it was. - await rm(path, { force: true }); - handle = await open(path, 'wx'); - } + // + // Two earlier attempts at this were wrong in the same way -- they made the + // takeover look exclusive without making it exclusive. `rm` then create + // let a second run become a live holder in between and then deleted its + // lock. Rename-then-read-back looked safer but is not: the interleaving + // A.rename, A.read, B.rename, B.read leaves each run reading its own pid, + // and both proceed. A concurrency test found 34 overlaps in 60 runs. + // + // Exclusion has to be on the takeover itself, so it runs through a second + // lock created with `wx` -- one atomic syscall, one winner. The loser is + // told to try again rather than being allowed to guess. + const steal = `${path}.steal`; + let stealHandle; + try { + stealHandle = await open(steal, 'wx'); + } catch (stealError) { + if ((stealError as NodeJS.ErrnoException).code !== 'EEXIST') throw stealError; + throw new FailureError( + 'another ailoud provisioning run is taking over a stale lock right now. Try again.', + ); + } - try { - const holder: LockHolder = { pid: process.pid, startedAt: new Date().toISOString() }; - await handle.writeFile(JSON.stringify(holder), 'utf8'); - } finally { - await handle.close(); + try { + // Re-read under the steal lock: between the check above and here, the + // stale lock may have been taken by a run that is now alive. + const current = await readHolder(path); + if (current !== null && isRunning(current.pid)) { + throw new FailureError( + `another ailoud provisioning run is already in progress (pid ${current.pid}, started ` + + `${current.startedAt}). Wait for it to finish, or stop it, then try again.`, + ); + } + await rm(path, { force: true }); + // Still `wx`: a run on the fast path can create the lock in the instant + // after that `rm`, and it is then the holder. Losing to it is the + // correct outcome, not something to overwrite. + // + // CodeQL reads this as a check-then-use race (js/file-system-race) and + // cannot see that the exclusion is held by `${path}.steal` above. The + // rule is excluded in .github/codeql/codeql-config.yml, where the + // reasoning lives; what actually guards this is the two-process test in + // setupLock.test.ts. + try { + const handle = await open(path, 'wx'); + try { + await handle.writeFile(mine, 'utf8'); + } finally { + await handle.close(); + } + } catch (createError) { + if ((createError as NodeJS.ErrnoException).code !== 'EEXIST') throw createError; + throw new FailureError( + 'another ailoud provisioning run took the lock at the same moment. Try again.', + ); + } + } finally { + await stealHandle.close(); + await rm(steal, { force: true }); + } } try { return await body(); } finally { - // force: a lock already gone is the outcome we wanted anyway, and - // failing to clean up must never mask what the body was doing. - await rm(path, { force: true }); + // Only our own lock. Removing it unconditionally would delete the lock of + // a run that legitimately took over after ours was declared stale, letting + // a third run in while that one is still working. + // + // force: a lock already gone is the outcome we wanted anyway, and failing + // to clean up must never mask what the body was doing. + const held = await readHolder(path); + if (held === null || held.pid === process.pid) await rm(path, { force: true }); } } diff --git a/apps/cli/src/ui/plain.ts b/apps/cli/src/ui/plain.ts index 71896d3..e4cd5b0 100644 --- a/apps/cli/src/ui/plain.ts +++ b/apps/cli/src/ui/plain.ts @@ -127,6 +127,14 @@ export class PlainUi implements Ui { if (note !== null) this.write(`note: ${note}`); } + public success(message: string): void { + // Same "ok " prefix `checks()` uses for a passing check, not a new + // vocabulary: one greppable shape for "this succeeded" across the whole + // plain renderer, since PlainUi is what runs whenever stdout is not a + // terminal. + this.write(`ok ${message}`); + } + public warn(message: string): void { this.write(`warning: ${message}`); } diff --git a/apps/cli/src/ui/pretty.ts b/apps/cli/src/ui/pretty.ts index d8ff537..1921f60 100644 --- a/apps/cli/src/ui/pretty.ts +++ b/apps/cli/src/ui/pretty.ts @@ -346,6 +346,10 @@ export class PrettyUi implements Ui { if (note !== null) log.info(this.wrap(note)); } + public success(message: string): void { + log.success(this.wrap(message)); + } + public warn(message: string): void { log.warn(this.wrap(message)); } diff --git a/apps/cli/src/ui/types.ts b/apps/cli/src/ui/types.ts index bd028c5..4c7f7af 100644 --- a/apps/cli/src/ui/types.ts +++ b/apps/cli/src/ui/types.ts @@ -159,6 +159,17 @@ export interface Ui { /** `doctor` finished running its checks: render the full report. */ checks(checks: readonly Check[]): void; + /** + * A status outcome that succeeded -- e.g. one file `mcp install` wrote, or + * one project `self sync` actually refreshed. Distinct from `content()`, + * which reports payload rather than an outcome, and from the frame's own + * success status, which speaks for the whole command rather than one + * outcome inside it. Used only where the thing being reported genuinely + * carries an enumerated status -- decorating a line that is not one of + * several possible outcomes would make this marker mean nothing. + */ + success(message: string): void; + /** * A non-fatal problem worth the user's attention, e.g. `--diarize` failing * to produce speaker labels. Distinct from the frame's own failure diff --git a/apps/cli/src/updateLog.test.ts b/apps/cli/src/updateLog.test.ts new file mode 100644 index 0000000..db46c84 --- /dev/null +++ b/apps/cli/src/updateLog.test.ts @@ -0,0 +1,145 @@ +import { describe, expect, it } from 'vitest'; +import { MemFs } from '@ailoud/core/testing'; +import { appendUpdateLog, updateLogPath } from './updateLog.js'; + +const DATA_DIR = '/data/ailoud'; + +describe('updateLogPath', () => { + it('sits directly under the per-user data directory', () => { + expect(updateLogPath(DATA_DIR)).toBe('/data/ailoud/update.log'); + }); +}); + +describe('appendUpdateLog', () => { + it('creates the log on first use', async () => { + const fs = new MemFs({}); + expect(await fs.exists(updateLogPath(DATA_DIR))).toBe(false); + + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'first line'); + + expect(await fs.exists(updateLogPath(DATA_DIR))).toBe(true); + expect(await fs.readTextFile(updateLogPath(DATA_DIR))).toBe('first line\n'); + }); + + it('appends subsequent lines rather than overwriting', async () => { + const fs = new MemFs({}); + + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'one'); + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'two'); + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'three'); + + expect(await fs.readTextFile(updateLogPath(DATA_DIR))).toBe('one\ntwo\nthree\n'); + }); + + it('does not touch a log under the 1 MB threshold', async () => { + const fs = new MemFs({}); + + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'small'); + const content = await fs.readTextFile(updateLogPath(DATA_DIR)); + + expect(content).toBe('small\n'); + }); + + it('caps the log at the last 500 lines once it passes 1 MB', async () => { + const fs = new MemFs({}); + // 20,000 lines at roughly 60 bytes each is comfortably over 1 MB. + const seedLines = Array.from({ length: 20_000 }, (_, i) => `old line ${i} ${'x'.repeat(50)}`); + await fs.ensureDir(DATA_DIR); + await fs.writeTextFile(updateLogPath(DATA_DIR), `${seedLines.join('\n')}\n`); + + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'the newest line'); + + const content = await fs.readTextFile(updateLogPath(DATA_DIR)); + const lines = content.split('\n').filter((line) => line.length > 0); + expect(lines.length).toBeLessThanOrEqual(500); + // The truncation drops the OLDEST lines, never the write that triggered it. + expect(lines[lines.length - 1]).toBe('the newest line'); + expect(lines[0]).not.toBe(seedLines[0]); + }); + + it('never contains more bytes than the 1 MB cap plus one line, after truncating', async () => { + const fs = new MemFs({}); + const seedLines = Array.from({ length: 20_000 }, (_, i) => `old line ${i} ${'x'.repeat(50)}`); + await fs.ensureDir(DATA_DIR); + await fs.writeTextFile(updateLogPath(DATA_DIR), `${seedLines.join('\n')}\n`); + + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'the newest line'); + + const content = await fs.readTextFile(updateLogPath(DATA_DIR)); + expect(Buffer.byteLength(content, 'utf8')).toBeLessThan(1024 * 1024); + }); + + it('never loses an entry to a second writer racing on the same file', async () => { + // Simulates two processes (a scheduled `self sync` and a manual `self + // update`, say) both calling appendUpdateLog close together. `RacingFs` + // captures what THIS call read as its starting point, then -- while + // that call is still building its own write -- lets a second, complete + // call to appendUpdateLog run and commit first. A plain read-modify-write + // (no re-check before committing) would then have the first call + // overwrite the second's line outright when it finally writes, which is + // exactly the defect found in review: "two processes both read, and the + // second write drops the first's line with no error." + class RacingFs extends MemFs { + private reads = 0; + private armed = true; + override async readTextFile(path: string): Promise { + this.reads += 1; + // Snapshot BEFORE letting the racing call run, so this call sees + // the bytes as they were at ITS read, not after the race. + const snapshot = await super.readTextFile(path); + if (this.reads === 1 && this.armed) { + this.armed = false; + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'second process'); + } + return snapshot; + } + } + const fs = new RacingFs({}); + await fs.ensureDir(DATA_DIR); + await fs.writeTextFile(updateLogPath(DATA_DIR), 'start\n'); + + await appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'first process'); + + const content = await fs.readTextFile(updateLogPath(DATA_DIR)); + const lines = content.split('\n').filter((l) => l.length > 0); + expect(lines).toContain('second process'); + expect(lines).toContain('first process'); + }); +}); + +describe('the append gives up rather than spinning forever', () => { + it('drops the line after a bounded number of lost races', async () => { + // The loop retries only on genuine contention, so hanging needs a + // continuous stream of rival writers -- implausible for a once-per-update + // diagnostic, and cheap to rule out entirely. A rival that ALWAYS commits + // between the re-read and the rename is what an unbounded loop could not + // survive. + // + // What this test proves, exactly: that the bound EXISTS, because it + // returns. It does NOT turn a regression into a clean failure -- + // measured: with the bound removed this test HANGS, and vitest's + // per-test timeout does not preempt it, because the loop awaits only + // already-resolved promises and never yields to a timer. So if this ever + // hangs instead of failing, the bound is what went missing. Stated + // plainly because the first version of this comment claimed the timeout + // would catch it, and that was untrue. + class AlwaysLoses extends MemFs { + public reads = 0; + public override async readTextFile(path: string): Promise { + this.reads += 1; + // Every second read -- the verification read -- reports different + // bytes, so the attempt always looks lost. + if (path.endsWith('update.log') && this.reads % 2 === 0) { + return `rival line ${this.reads}\n`; + } + return super.readTextFile(path).catch(() => ''); + } + public override async exists(): Promise { + return true; + } + } + const fs = new AlwaysLoses({}); + + await expect(appendUpdateLog({ fs, userDataDir: DATA_DIR }, 'mine')).resolves.toBeUndefined(); + }, 5000); +}); diff --git a/apps/cli/src/updateLog.ts b/apps/cli/src/updateLog.ts new file mode 100644 index 0000000..10d0edb --- /dev/null +++ b/apps/cli/src/updateLog.ts @@ -0,0 +1,119 @@ +import { randomUUID } from 'node:crypto'; +import type { Fs } from '@ailoud/core'; + +/** The log is truncated once it grows past this many bytes. */ +const MAX_BYTES = 1024 * 1024; // 1 MB + +/** How many of the most recent lines survive a truncation. */ +/** How many times an append will retry a lost race before dropping the line. */ +const MAX_ATTEMPTS = 5; + +const MAX_LINES = 500; + +export interface UpdateLogDeps { + readonly fs: Fs; + readonly userDataDir: string; +} + +/** + * Where the plain-text log of `self sync` and `self update` actions lives. + * Per-user, like the project registry: it must never live inside a project's + * own `.ailoud/`, or it would only ever record that one project's history. + */ +export function updateLogPath(userDataDir: string): string { + return `${userDataDir}/update.log`; +} + +/** + * Keeps only the last `maxLines` lines of `text`. + * + * `text` always ends in exactly one trailing newline (every write this + * module makes ends that way), so splitting on "\n" leaves one empty + * element at the end that is not a line of its own -- it is dropped before + * slicing, and the `join` below adds the trailing newline back. + */ +function keepLastLines(text: string, maxLines: number): string { + const lines = text.split('\n'); + const last = lines[lines.length - 1]; + const withoutTrailingEmpty = last === '' ? lines.slice(0, -1) : lines; + const kept = withoutTrailingEmpty.slice(-maxLines); + return kept.length === 0 ? '' : `${kept.join('\n')}\n`; +} + +/** + * Appends one line to the update log, creating it (and its directory) on + * first use. + * + * Two processes racing here used to lose an entry outright: both read the + * same bytes, both built "everything so far, plus my line", and whichever + * wrote last replaced the file with only ITS line appended -- the other's + * was gone, with no error anywhere. The same read-modify-write race + * `projects.ts` was redesigned to avoid for the project registry. + * + * An O_APPEND write is the textbook fix for exactly this shape of race: the + * kernel serializes small appends, so neither writer's bytes are ever + * discarded no matter how the two calls interleave. The `Fs` port has no + * such primitive, though -- only whole-file `writeTextFile`/`rename` -- so + * that is not available here. Instead this detects the conflict itself: + * after building the candidate content and + * writing it to a temp file, it re-reads the real log and checks whether it + * still matches what this call read at the start. If another writer + * committed in between, the temp file is discarded and the whole + * read-build-write is retried against the fresh content, until a rename + * lands against the same bytes it was built from. That narrows the race to + * the gap between that last re-read and the rename -- not zero, the same + * residual window `writeRegistry` (`projects.ts`) documents and accepts for + * the same reason -- without a lock: this is an occasional diagnostic + * append, not a hot path, so a retry costs nothing worth avoiding. + * + * Truncated to the last 500 lines once the file passes 1 MB, so a machine + * ailoud has run on for years does not grow this file forever. The check + * runs after every append rather than on a schedule, because the file is + * never large enough for that read-and-measure to be a cost worth avoiding. + * + * Callers must never pass a credential or any transcript text: this line is + * meant to be safe to paste into a bug report. + */ +export async function appendUpdateLog(deps: UpdateLogDeps, line: string): Promise { + const path = updateLogPath(deps.userDataDir); + await deps.fs.ensureDir(deps.userDataDir); + + // Bounded, deliberately. The loop only spins on genuine contention -- a + // persistent failure throws instead -- so an unbounded `for (;;)` needs a + // continuous stream of rival writers to hang, which a once-per-update + // diagnostic append will not see. Two lines is a cheap price for removing + // the possibility altogether, and dropping the line is the right answer on + // giving up: this file exists to explain what happened, and losing one + // entry is incomparably better than hanging the command that was writing + // it. The same rule the project registry follows. + for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt += 1) { + const before = (await deps.fs.exists(path)) ? await deps.fs.readTextFile(path) : ''; + const appended = `${before}${line}\n`; + const next = + Buffer.byteLength(appended, 'utf8') > MAX_BYTES + ? keepLastLines(appended, MAX_LINES) + : appended; + + const tempPath = `${path}.${randomUUID()}.tmp`; + try { + await deps.fs.writeTextFile(tempPath, next); + } catch (error) { + // The target has not been touched yet, so there is nothing to undo. + // Clear the partial temporary file rather than leaving litter behind. + await deps.fs.removeFile(tempPath); + throw error; + } + + const current = (await deps.fs.exists(path)) ? await deps.fs.readTextFile(path) : ''; + if (current !== before) { + // Someone else committed while this attempt was being built: discard + // it and retry against what is actually on disk now, rather than + // renaming over -- and silently erasing -- their line. + await deps.fs.removeFile(tempPath); + continue; + } + await deps.fs.rename(tempPath, path); + return; + } + // Every attempt lost the race. Dropping the line is correct -- see above. +} diff --git a/apps/cli/src/updateNotice.test.ts b/apps/cli/src/updateNotice.test.ts new file mode 100644 index 0000000..165700a --- /dev/null +++ b/apps/cli/src/updateNotice.test.ts @@ -0,0 +1,358 @@ +import { describe, expect, it, vi } from 'vitest'; +import type { PublishedVersion } from '@ailoud/core'; +import { MemFs, FakeClock } from '@ailoud/core/testing'; +import { startUpdateCheck, updateCachePath } from './updateNotice.js'; +import type { NoticeDeps } from './updateNotice.js'; + +const DATA_DIR = '/data/ailoud'; + +/** Lets every pending microtask run, so an "already settled" fake fetch or + * cache read has actually resolved before a test calls `finish()`. Real + * usage never needs this: the command being run does real work between + * `startUpdateCheck` and `finish()`, which is what this stands in for. */ +function flush(): Promise { + return new Promise((resolve) => setImmediate(resolve)); +} + +/** A `published` that resolves with a fixed list, never touching a network. */ +function published(versions: readonly PublishedVersion[]): NoticeDeps['published'] { + return () => Promise.resolve(versions); +} + +/** A `published` that never settles -- the shape a hung connection has from + * this module's point of view, whether or not it is ever aborted. */ +function hangingPublished(): NoticeDeps['published'] { + return () => new Promise(() => undefined); +} + +/** A `published` that behaves like the real `registryPublished`: it never + * settles on its own within a test's lifetime, but REJECTS the instant its + * signal is aborted -- exactly what `https.request`'s own `signal` option + * does. `hangingPublished` above does not model this at all (it ignores the + * signal entirely), which is exactly why the review found the abort-poisons + * -the-cache defect survived every existing test: none of them exercised a + * fetch that actually reacts to being aborted. */ +function abortablePublished(): NoticeDeps['published'] { + return (signal) => + new Promise((_resolve, reject) => { + signal.addEventListener('abort', () => { + const error = new Error('aborted'); + error.name = 'AbortError'; + reject(error); + }); + }); +} + +function baseDeps(overrides: Partial = {}): NoticeDeps { + return { + fs: new MemFs(), + clock: new FakeClock(), + userDataDir: DATA_DIR, + currentVersion: '1.0.0', + argv: ['ls'], + env: {}, + stderrIsTTY: true, + checkEnabled: true, + published: published([{ version: '1.1.0', deprecated: false }]), + ...overrides, + }; +} + +describe('startUpdateCheck', () => { + it('prints nothing and waits for nothing when the fetch has not settled', async () => { + // The whole point. Awaiting the network before exit would add up to two + // seconds to `ailoud ls`. + const notice = startUpdateCheck(baseDeps({ published: hangingPublished() })); + expect(await notice.finish()).toBeNull(); + }); + + it('prints from the cache when this run could not refresh it', async () => { + const fs = new MemFs({ + [updateCachePath(DATA_DIR)]: JSON.stringify({ + checkedAt: '2026-01-01T00:00:00.000Z', + target: '1.2.3', + }), + }); + // A fetch that would hang forever if it were ever started: a fresh cache + // must answer without going anywhere near the network. No `await + // flush()` here: real usage never performs one between + // `startUpdateCheck()` and `finish()`, and a test that needs one to pass + // is proving nothing about production -- see `finish()`'s own bounded + // wait, which is what actually gives this cache read a chance to win. + const deps = baseDeps({ fs, published: hangingPublished() }); + + const notice = startUpdateCheck(deps); + + expect(await notice.finish()).toBe('1.2.3'); + }); + + it('bounds a slow disk read instead of waiting for it', async () => { + // Stands in for a slow or momentarily unresponsive filesystem (a + // network home directory, a sleeping external drive, a contended or + // nearly full disk): the read resolves, but only long after any + // budget this module may spend waiting for it. Real usage has no + // signal to cancel this with -- `Fs` offers none -- so the only + // available fix is bounding how long anything here waits for it, + // never assuming the disk itself can be made to stop. Unref'd so this + // fixture alone cannot hold the test process open: the property under + // test is whether the SOURCE gives up on it, not whether this fake + // does. + const fs = new MemFs(); + fs.exists = (): Promise => + new Promise((resolve) => { + const timer = setTimeout(() => resolve(false), 3000); + timer.unref(); + }); + let called = false; + const deps = baseDeps({ + fs, + published: () => { + called = true; + return Promise.resolve([{ version: '1.1.0', deprecated: false }]); + }, + }); + + const notice = startUpdateCheck(deps); + await notice.finish(); + // Well past this module's own ~50ms budget, but nowhere near the + // disk's 3 second delay: if the disk read were still being awaited, + // the registry would not have been reached yet, because `readCache` + // runs before it on every path. + await new Promise((resolve) => setTimeout(resolve, 200)); + + expect(called).toBe(true); + }, 2500); + + it('unrefs the bounded wait timer, so it alone cannot keep the process alive', async () => { + const createdTimers: NodeJS.Timeout[] = []; + const realSetTimeout = globalThis.setTimeout; + const spy = vi.spyOn(globalThis, 'setTimeout'); + spy.mockImplementation(((...args: Parameters) => { + const timer = realSetTimeout(...args); + createdTimers.push(timer); + return timer; + }) as typeof setTimeout); + + try { + // A fetch that never settles, so `finish()` can only ever be answered + // by the bounded wait's own timer -- exactly the timer under test. + const notice = startUpdateCheck(baseDeps({ published: hangingPublished() })); + expect(await notice.finish()).toBeNull(); + } finally { + spy.mockRestore(); + } + + expect(createdTimers.length).toBeGreaterThan(0); + for (const timer of createdTimers) { + // hasRef() is Node's own answer to "does this timer count against the + // event loop staying open": false is what lets a command whose own + // work is already done exit immediately, unheld by this wait. + expect(timer.hasRef()).toBe(false); + } + }); + + it('caches a failure too, so a broken network costs one attempt a day', async () => { + const fs = new MemFs(); + const deps = baseDeps({ + fs, + published: () => Promise.reject(new Error('registry unreachable')), + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBeNull(); + const raw = fs.files.get(updateCachePath(DATA_DIR)); + expect(raw).toBeDefined(); + expect(JSON.parse(raw!)).toMatchObject({ target: null }); + }); + + it('never caches a null caused by its own abort, so the next run retries', async () => { + // The defect found in review: BOUND_MS covers the whole check, so a + // fast command's fetch is always still in flight when finish() gives + // up and aborts it. `abortablePublished` rejects on that abort exactly + // as the real HTTPS client does -- if the catch block cached that + // rejection as a genuine "no update", the notice could never fire. + const fs = new MemFs(); + const deps = baseDeps({ fs, published: abortablePublished() }); + + const notice = startUpdateCheck(deps); + expect(await notice.finish()).toBeNull(); + // Give the abort's rejection a chance to reach the catch block; a real + // command's own work would take far longer than this in practice. + await flush(); + + expect(fs.files.has(updateCachePath(DATA_DIR))).toBe(false); + }, 2000); + + it('fills the cache with a genuine answer once the fetch actually settles', async () => { + // The consequence documented on startUpdateCheck: a command slow enough + // to outlast the round trip is the one that fills the cache. Simulated + // here by letting the fetch resolve (via flush()) before finish() is + // even called, the way a real multi-second command would without a + // test needing to wait multiple seconds. + const fs = new MemFs(); + const deps = baseDeps({ + fs, + published: published([{ version: '1.2.3', deprecated: false }]), + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBe('1.2.3'); + const raw = fs.files.get(updateCachePath(DATA_DIR)); + expect(raw).toBeDefined(); + expect(JSON.parse(raw!)).toMatchObject({ target: '1.2.3' }); + }); + + it('is silent when stderr is not a TTY', async () => { + let called = false; + const deps = baseDeps({ + stderrIsTTY: false, + published: () => { + called = true; + return Promise.resolve([{ version: '1.1.0', deprecated: false }]); + }, + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBeNull(); + expect(called).toBe(false); + }); + + it('is silent with --json', async () => { + let called = false; + const deps = baseDeps({ + argv: ['ls', '--json'], + published: () => { + called = true; + return Promise.resolve([{ version: '1.1.0', deprecated: false }]); + }, + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBeNull(); + expect(called).toBe(false); + }); + + it('is silent with --format json', async () => { + // `show --format json` is machine-readable output too, and with + // stdout and stderr merged this notice would land inside it just like + // `--json` would. + let called = false; + const deps = baseDeps({ + argv: ['show', 'abc123', '--format', 'json'], + published: () => { + called = true; + return Promise.resolve([{ version: '1.1.0', deprecated: false }]); + }, + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBeNull(); + expect(called).toBe(false); + }); + + it('is silent in the MCP server', async () => { + let called = false; + const deps = baseDeps({ + argv: ['mcp'], + published: () => { + called = true; + return Promise.resolve([{ version: '1.1.0', deprecated: false }]); + }, + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBeNull(); + expect(called).toBe(false); + }); + + it('is silent when AILOUD_NO_UPDATE_CHECK is set', async () => { + let called = false; + const deps = baseDeps({ + env: { AILOUD_NO_UPDATE_CHECK: '1' }, + published: () => { + called = true; + return Promise.resolve([{ version: '1.1.0', deprecated: false }]); + }, + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBeNull(); + expect(called).toBe(false); + }); + + it('is silent when config update.check is false', async () => { + let called = false; + const deps = baseDeps({ + checkEnabled: false, + published: () => { + called = true; + return Promise.resolve([{ version: '1.1.0', deprecated: false }]); + }, + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBeNull(); + expect(called).toBe(false); + }); + + it('is silent for self check and self update themselves', async () => { + const argvs = [ + ['self', 'check'], + ['self', 'c'], + ['self', 'update'], + ['self', 'u'], + ['check'], + ['update'], + ]; + for (const argv of argvs) { + let called = false; + const deps = baseDeps({ + argv, + published: () => { + called = true; + return Promise.resolve([{ version: '1.1.0', deprecated: false }]); + }, + }); + + const notice = startUpdateCheck(deps); + await flush(); + + expect(await notice.finish()).toBeNull(); + expect(called).toBe(false); + } + }); +}); + +describe('the MCP suppression must not catch a run that only prints help', () => { + it('stays silent for the bare mcp server', async () => { + const notice = startUpdateCheck(baseDeps({ argv: ['mcp'] })); + expect(await notice.finish()).toBeNull(); + }); + + it.each([['--help'], ['-h'], ['--version']])( + 'still speaks for `mcp %s`, which starts no server', + async (flag) => { + // Stripping every `-` argument before deciding made this look like the + // bare `mcp` command. A suppression rule that fires on the wrong input + // is how one eventually fails to fire on the right one. + const notice = startUpdateCheck(baseDeps({ argv: ['mcp', flag] })); + expect(await notice.finish()).toBe('1.1.0'); + }, + ); +}); diff --git a/apps/cli/src/updateNotice.ts b/apps/cli/src/updateNotice.ts new file mode 100644 index 0000000..fcd4492 --- /dev/null +++ b/apps/cli/src/updateNotice.ts @@ -0,0 +1,305 @@ +import { z } from 'zod'; +import type { Clock, Fs, PublishedVersion } from '@ailoud/core'; +import { chooseUpdateTarget } from '@ailoud/core'; + +/** How long a cached answer is trusted before this run asks the registry again. */ +const TTL_MS = 24 * 60 * 60 * 1000; + +/** Resolves at once with whichever the fetch or nothing settles first. */ +const SENTINEL = Symbol('update-check-not-settled'); + +/** + * How long any single `Fs` call on this path may take. A local cache read is + * sub-millisecond, so this is generous for a healthy disk and decisive on a + * sick one (a network home directory, a sleeping external drive, a + * contended or nearly full disk). Any timeout here means "say nothing" -- + * never an error, and never a claim of being up to date. + * + * `finish()` below reuses this same value to bound how much longer it will + * wait, past its own call, for the in-flight check to settle -- but that is + * NOT the same as bounding the registry round trip itself to 50ms. By the + * time `finish()` runs, the disk read has already had the command's own + * work (plus this much again) to complete, so a cache hit almost always + * wins outright; a cache MISS instead falls through to the live fetch, + * which this bound then gives up WAITING for. Giving up still aborts the + * fetch (so the process can exit right away instead of lingering on the + * network), but see `startUpdateCheck`'s own doc comment for why an abort + * must never be confused with a genuine answer worth caching. + */ +const BOUND_MS = 50; + +/** + * Races `promise` against an unref'd timer of `BOUND_MS`, rejecting if the + * timer wins. Every caller on this path already treats any rejection here + * identically to a real I/O error, so a slow disk costs at most `BOUND_MS`, + * never the disk's own unbounded worst case. The timer is unref'd for the + * same reason `sentinelAfter`'s is below: it must never itself be the thing + * that keeps a command's process alive. + */ +function withFsTimeout(promise: Promise): Promise { + return new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error('fs call exceeded its bound')), BOUND_MS); + timer.unref(); + promise.then( + (value) => { + clearTimeout(timer); + resolve(value); + }, + (error: unknown) => { + clearTimeout(timer); + reject(error); + }, + ); + }); +} + +/** + * Resolves to SENTINEL after `ms`, and never before. The timer backing this + * is unref'd: an unref'd timer does not count against Node keeping the + * event loop open, so a command whose own work is already done still exits + * right away, exactly as if this timer did not exist. Without `unref()` + * here, this bounded wait would itself reintroduce the very delay this + * module exists to avoid -- do not "simplify" it away. + */ +function sentinelAfter(ms: number): Promise { + return new Promise((resolve) => { + const timer = setTimeout(() => resolve(SENTINEL), ms); + timer.unref(); + }); +} + +/** Everything `startUpdateCheck` needs, so a test never touches a real clock, + * filesystem, network, environment or `process.argv`. */ +export interface NoticeDeps { + readonly fs: Fs; + readonly clock: Clock; + /** `AiloudPaths.userDataDir`: always per-user, never a project's `.ailoud/`. */ + readonly userDataDir: string; + readonly currentVersion: string; + /** `process.argv.slice(2)`: the words after the node binary and script path. */ + readonly argv: readonly string[]; + readonly env: Record; + /** Whether stderr is attached to a real terminal, not a pipe or a log file. */ + readonly stderrIsTTY: boolean; + /** The `update.check` config key. */ + readonly checkEnabled: boolean; + /** Fetches the published versions, honoring `signal` for cancellation. */ + readonly published: (signal: AbortSignal) => Promise; +} + +/** What `finish()` answers: the version to mention, or null for "say nothing". */ +export interface UpdateCheck { + /** Resolves at once: the settled result, or null if it has not arrived. */ + finish(): Promise; +} + +/** True when `--json` appears anywhere on the command line, or when + * `--format json` does (`ailoud show --format json`): both put + * machine-readable output on stdout, and with stdout and stderr merged (a + * redirect, or any tool that captures both) this notice would land inside + * it either way. Machine output stays machine output. */ +function hasJsonFlag(argv: readonly string[]): boolean { + if (argv.includes('--json')) return true; + for (let i = 0; i < argv.length; i += 1) { + const arg = argv[i]; + if (arg === '--format') return argv[i + 1] === 'json'; + if (arg !== undefined && arg.startsWith('--format=')) { + return arg.slice('--format='.length) === 'json'; + } + } + return false; +} + +/** True only for the bare `mcp` command, which speaks the MCP protocol over + * stdout for as long as the client stays connected -- never `mcp install`, + * which is an ordinary one-shot CLI command like any other. */ +function isServingMcp(argv: readonly string[]): boolean { + // `--help` and `--version` never start a server, so they must not be + // stripped before deciding. Stripping every `-` argument first made + // `ailoud mcp --help` look like the bare `mcp` command and suppressed the + // notice on a run that prints ordinary help. Harmless in effect, but a + // suppression rule that fires on the wrong input is how one eventually + // fails to fire on the right one. + const words = argv.filter((arg) => !arg.startsWith('-')); + if (words[0] !== 'mcp' || words[1] !== undefined) return false; + return !argv.some((arg) => arg === '--help' || arg === '-h' || arg === '--version'); +} + +/** True for `self check` and `self update` themselves (every spelling: the + * group form, its one-letter alias, and the hidden top-level alias) -- they + * just said it in full, so a passive mention right after would be noise. */ +function isSelfCheckOrUpdate(argv: readonly string[]): boolean { + const words = argv.filter((arg) => !arg.startsWith('-')); + const [first, second] = words; + if (first === 'check' || first === 'update') return true; + if (first !== 'self') return false; + return second === 'check' || second === 'c' || second === 'update' || second === 'u'; +} + +function suppressed(deps: NoticeDeps): boolean { + if (!deps.stderrIsTTY) return true; + if (hasJsonFlag(deps.argv)) return true; + if (isServingMcp(deps.argv)) return true; + if (deps.env['AILOUD_NO_UPDATE_CHECK'] !== undefined) return true; + if (!deps.checkEnabled) return true; + if (isSelfCheckOrUpdate(deps.argv)) return true; + return false; +} + +const CacheSchema = z.object({ + checkedAt: z.string(), + target: z.string().nullable(), +}); + +type CacheEntry = z.infer; + +/** Where the once-a-day cache lives: per-user, like the project registry and + * the update log, so being inside a project library never scopes it down. */ +export function updateCachePath(userDataDir: string): string { + return `${userDataDir}/update-check.json`; +} + +/** The cached answer, or null when there is none, it does not parse, or it + * has aged past the 24 hour TTL. A cache that fails to read never throws: + * the worst outcome is one extra check, not a broken command. */ +async function readCache(deps: NoticeDeps): Promise { + const path = updateCachePath(deps.userDataDir); + let raw: string; + try { + if (!(await withFsTimeout(deps.fs.exists(path)))) return null; + raw = await withFsTimeout(deps.fs.readTextFile(path)); + } catch { + return null; + } + let document: unknown; + try { + document = JSON.parse(raw); + } catch { + return null; + } + const result = CacheSchema.safeParse(document); + if (!result.success) return null; + const ageMs = Date.parse(deps.clock.nowIso()) - Date.parse(result.data.checkedAt); + if (!Number.isFinite(ageMs) || ageMs < 0 || ageMs > TTL_MS) return null; + return result.data; +} + +/** Writes the cache, on a success or a failure alike -- see this module's own + * doc comment on `startUpdateCheck` for why a failure is cached too. Never + * throws: a directory that cannot be written costs one extra check a day, + * not a crash of whatever command triggered it. */ +async function writeCache(deps: NoticeDeps, target: string | null): Promise { + const entry: CacheEntry = { checkedAt: deps.clock.nowIso(), target }; + try { + await withFsTimeout(deps.fs.ensureDir(deps.userDataDir)); + await withFsTimeout( + deps.fs.writeTextFile(updateCachePath(deps.userDataDir), `${JSON.stringify(entry)}\n`), + ); + } catch { + // Best effort, as above. + } +} + +/** + * Starts a once-a-day, best-effort check for a newer `ailoud`, without ever + * making a command slower. + * + * The fetch (or the cache read behind it) is started here and deliberately + * NOT awaited: awaiting the network before a command's own work is done + * would add up to `updateTimeoutMs` to every single run of, say, `ailoud + * ls`. Suppression is checked FIRST, synchronously, so a suppressed run + * never starts a fetch at all -- there is nothing to abort and nothing to + * cache. + * + * `finish()` is how a caller collects the answer without ever waiting long + * for it: it races the in-flight check against `sentinelAfter(BOUND_MS)`, an + * unref'd timer of about 50ms -- not, as an earlier version of this module + * did, an ALREADY-RESOLVED sentinel promise. That older race was a single + * microtask, which sounds free but was not: a real fetch or disk read can + * never win a race against a promise that has already resolved, because the + * sentinel's continuation runs before the event loop even reaches the phase + * where that answer could arrive. Racing against a real, if small, timer + * instead gives a fast, healthy check a genuine chance to settle first. If + * it does, its answer wins. If it does not, the timer wins instead: + * `finish()` aborts the in-flight check (so the process is free to exit + * right away, rather than lingering until the network gives up on its own) + * and returns null. The timer being unref'd is what keeps this bounded wait + * from ever being able to make a command slower than `BOUND_MS` on its own: + * an unref'd timer never counts against the event loop staying open, so a + * command whose own work is already done still exits immediately. The next + * run's cache read is what prints from the attempt this one could not + * finish. + * + * A GENUINE failure is cached as null too, exactly like "no update found": a + * version check must never read as news, and caching the failure is what + * limits a broken registry (a bad HTTP status, an unreadable body, a + * connection that fails on its own) to one wasted attempt a day rather than + * one per command. + * + * An ABORT -- `finish()` giving up on this check and calling + * `controller.abort()` -- is deliberately NOT a genuine failure and is never + * cached. Caching it would be indistinguishable from a completed check that + * found nothing, and for a fast command (`ailoud ls`, `ailoud search`, most + * of them) the fetch is *always* aborted before a real registry round trip + * can finish: bounding `finish()`'s wait is what keeps this module from ever + * making a command slower, but it also means a fast command can basically + * never be the one to complete its own fetch. Caching that as "no update" + * would have made the notice permanently unreachable -- exactly the defect + * this doc comment now exists to prevent. Instead, `finish()` still aborts + * the fetch when it gives up (so the process is free to exit right away), + * but the abort itself writes NOTHING to the cache: the next run (of + * anything) starts fresh and tries again. The consequence to accept is that + * the cache only ever gets a genuine answer from a command slow enough to + * outlast the round trip -- `transcribe`, `summarize`, `setup`, or any run + * against an already-warm connection -- and every fast command in between + * prints from whatever one of those left behind. That is the honest shape + * of "never delay, but still eventually say something": only the WAIT for + * the fetch is bounded, never the fetch's own right to keep running and to + * report a genuine answer whenever it actually settles. + */ +export function startUpdateCheck(deps: NoticeDeps): UpdateCheck { + if (suppressed(deps)) return { finish: async () => null }; + + const controller = new AbortController(); + + const inflight = (async (): Promise => { + const cached = await readCache(deps); + if (cached !== null) return cached.target; + try { + const versions = await deps.published(controller.signal); + const target = chooseUpdateTarget(deps.currentVersion, versions); + await writeCache(deps, target); + return target; + } catch { + if (controller.signal.aborted) { + // WE did this, by giving up in finish() below -- not the registry. + // An abort says nothing about whether a newer version exists, so + // nothing is written: see this function's own doc comment for why + // caching it here is exactly the defect that made the notice + // unreachable in the first place. + return null; + } + // A genuine failure: the registry answered, badly, or the connection + // failed on its own. Caching it is what limits a broken network to + // one wasted attempt a day rather than one per command. + await writeCache(deps, null); + return null; + } + })(); + + return { + async finish(): Promise { + // Whichever wins: the check, or the bounded wait. The wait's timer is + // unref'd (see `sentinelAfter`), so it alone can never keep the + // process alive -- do not replace it with a plain timer "for + // simplicity"; that would reintroduce exactly the delay this module + // exists to avoid. + const raced = await Promise.race([inflight, sentinelAfter(BOUND_MS)]); + if (raced === SENTINEL) { + controller.abort(); + return null; + } + return raced; + }, + }; +} diff --git a/apps/cli/src/version.ts b/apps/cli/src/version.ts new file mode 100644 index 0000000..e199fbc --- /dev/null +++ b/apps/cli/src/version.ts @@ -0,0 +1,28 @@ +import { readFileSync } from 'node:fs'; + +/** + * The CLI's own version, read from its manifest. + * + * It used to be the literal `0.0.0` passed to commander, so the published + * 1.0.0-dev.1 answered `ailoud --version` with 0.0.0 and told every MCP client + * the same. The manifest is the one place a release already updates, and it + * ships inside the tarball, so it is the only copy that cannot go stale. + * + * Resolved relative to this module rather than to the working directory: + * `dist/version.js` sits one level below the package root both in the + * repository and in an installed package. + */ +export const VERSION: string = readVersion(); + +function readVersion(): string { + const manifest = new URL('../package.json', import.meta.url); + const parsed: unknown = JSON.parse(readFileSync(manifest, 'utf8')); + const version = + typeof parsed === 'object' && parsed !== null + ? (parsed as { version?: unknown }).version + : undefined; + if (typeof version !== 'string' || version === '') { + throw new Error(`no version in ${manifest.pathname}`); + } + return version; +} diff --git a/apps/cli/src/wiring.test.ts b/apps/cli/src/wiring.test.ts index c898ffb..84cc570 100644 --- a/apps/cli/src/wiring.test.ts +++ b/apps/cli/src/wiring.test.ts @@ -1,9 +1,15 @@ -import { mkdir, mkdtemp, rm, stat, writeFile } from 'node:fs/promises'; +import { existsSync } from 'node:fs'; +import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { afterEach, describe, expect, it } from 'vitest'; +import { afterEach, describe, expect, it, vi } from 'vitest'; import { EnvironmentError } from '@ailoud/core'; +import { NodeFs } from '@ailoud/providers'; import { createContext } from './wiring.js'; +import { buildProgram } from './program.js'; +import { registryPath } from './projects.js'; +import { VERSION } from './version.js'; +import { context as fakeCliContext } from './commands/testContext.js'; describe('createContext', () => { const dirs: string[] = []; @@ -181,3 +187,142 @@ describe('createContext', () => { } }); }); + +describe('project registration (task 10)', () => { + const dirs: string[] = []; + + afterEach(async () => { + for (const dir of dirs.splice(0)) await rm(dir, { recursive: true, force: true }); + vi.restoreAllMocks(); + }); + + async function tempHome(): Promise { + const dir = await mkdtemp(join(tmpdir(), 'ailoud-wiring-reg-home-')); + dirs.push(dir); + return dir; + } + + /** A directory with its own `.ailoud/`, so `createContext` treats it as a project library. */ + async function tempProject(): Promise { + const dir = await mkdtemp(join(tmpdir(), 'ailoud-wiring-reg-project-')); + dirs.push(dir); + await mkdir(join(dir, '.ailoud'), { recursive: true }); + return dir; + } + + /** A plain directory, with no `.ailoud/` anywhere above it, for the per-user path. */ + async function tempPlainDir(): Promise { + const dir = await mkdtemp(join(tmpdir(), 'ailoud-wiring-reg-plain-')); + dirs.push(dir); + return dir; + } + + async function readRegistry(home: string): Promise { + const raw = await readFile(registryPath(join(home, '.local', 'share', 'ailoud')), 'utf8'); + return JSON.parse(raw) as unknown[]; + } + + it('registers a project when a command resolves its library', async () => { + const home = await tempHome(); + const project = await tempProject(); + const cwdSpy = vi.spyOn(process, 'cwd').mockReturnValue(project); + const context = await createContext({ HOME: home }, () => {}); + try { + expect(context.paths.isProjectLibrary).toBe(true); + const entries = (await readRegistry(home)) as Array<{ + path: string; + libraryDir?: string; + }>; + expect(entries).toEqual([ + expect.objectContaining({ path: project, libraryDir: join(project, '.ailoud') }), + ]); + } finally { + context.store.close(); + cwdSpy.mockRestore(); + } + }); + + it('does not register the per-user library', async () => { + // It always exists, so listing it would be noise. + const home = await tempHome(); + const plain = await tempPlainDir(); + const cwdSpy = vi.spyOn(process, 'cwd').mockReturnValue(plain); + const context = await createContext({ HOME: home }, () => {}); + try { + expect(context.paths.isProjectLibrary).toBe(false); + expect(existsSync(registryPath(join(home, '.local', 'share', 'ailoud')))).toBe(false); + } finally { + context.store.close(); + cwdSpy.mockRestore(); + } + }); + + it('registers the project mcp install wrote rules into, with the version', async () => { + const ctx = fakeCliContext(); + const cwdSpy = vi.spyOn(process, 'cwd').mockReturnValue('/proj/a'); + try { + await buildProgram(ctx).parseAsync([ + 'node', + 'ailoud', + 'mcp', + 'install', + '--yes', + '--target', + 'claude', + '--location', + 'local', + ]); + const raw = await ctx.fs.readTextFile(registryPath(ctx.paths.userDataDir)); + const entries = JSON.parse(raw) as Array<{ path: string; rulesVersion?: string }>; + const entry = entries.find((candidate) => candidate.path === '/proj/a'); + expect(entry?.rulesVersion).toBe(VERSION); + } finally { + cwdSpy.mockRestore(); + } + }); + + it('registers at most once a day', async () => { + // The hot-path rule: `createContext` runs before every command, and + // writing the registry on every single one of them would put a disk + // write on something as routine as `ailoud ls`. + const home = await tempHome(); + const project = await tempProject(); + const cwdSpy = vi.spyOn(process, 'cwd').mockReturnValue(project); + const writeSpy = vi.spyOn(NodeFs.prototype, 'writeTextFile'); + try { + const first = await createContext({ HOME: home }, () => {}); + first.store.close(); + const second = await createContext({ HOME: home }, () => {}); + second.store.close(); + + const registryWrites = writeSpy.mock.calls.filter(([path]) => path.includes('projects.json')); + expect(registryWrites).toHaveLength(1); + } finally { + cwdSpy.mockRestore(); + writeSpy.mockRestore(); + } + }); + + it('never fails a command because the registry could not be written', async () => { + // `ailoud ls` (or any other command) must not die because a bookkeeping + // file could not be written -- a full disk, a read-only home, or any + // other permission problem. + const home = await tempHome(); + const project = await tempProject(); + const cwdSpy = vi.spyOn(process, 'cwd').mockReturnValue(project); + const writeSpy = vi + .spyOn(NodeFs.prototype, 'writeTextFile') + .mockRejectedValue(new Error('ENOSPC: no space left on device')); + try { + const context = await createContext({ HOME: home }, () => {}); + try { + expect(context.paths.isProjectLibrary).toBe(true); + } finally { + context.store.close(); + } + } finally { + cwdSpy.mockRestore(); + writeSpy.mockRestore(); + } + }); +}); diff --git a/apps/cli/src/wiring.ts b/apps/cli/src/wiring.ts index ac35d5a..8e821b4 100644 --- a/apps/cli/src/wiring.ts +++ b/apps/cli/src/wiring.ts @@ -9,15 +9,19 @@ import type { SpeechSegmenter, Summarizer, TranscriptionProvider, + VersionSource, } from '@ailoud/core'; import { existsSync, statSync } from 'node:fs'; -import { EnvironmentError } from '@ailoud/core'; +import { EnvironmentError, isHostedLlm } from '@ailoud/core'; import { AnthropicSummarizer, ClaudeCliSummarizer, + DEFAULT_REGISTRY, + DEFAULT_TIMEOUT_MS, FfmpegAudioTool, LlamaCppSummarizer, NodeFs, + NpmRegistry, OpenAiCompatibleSummarizer, SherpaDiarizer, SystemClock, @@ -26,11 +30,58 @@ import { WhisperVadSegmenter, openStore, } from '@ailoud/providers'; -import { parseConfig, resolvePaths } from './config.js'; +import { PROJECT_DIR, parseConfig, resolvePaths } from './config.js'; +import type { RegistryTransport } from '@ailoud/providers'; import type { AiloudConfig, AiloudPaths } from './config.js'; import { createUi } from './ui/index.js'; import type { Ui } from './ui/index.js'; import { apiKeyFrom } from './apiKey.js'; +import { rememberProject } from './projects.js'; + +/** + * Where `ailoud self check` looks for newer versions, and how long it waits. + * Not read from AiloudConfig: the schema's `update.check` key is only + * whether to look, never where -- there is one npm registry this project + * publishes to, and no user has a reason to point ailoud at another one. + */ +const UPDATE_REGISTRY = DEFAULT_REGISTRY; +const UPDATE_TIMEOUT_MS = DEFAULT_TIMEOUT_MS; + +/** + * Stubs the npm registry answer with a JSON fixture, when `AILOUD_PACKUMENTS` + * names one, instead of a real network call. + * + * Same environment variable and the same fixture shape -- + * `{ "": }` -- that `scripts/retire-prereleases.mjs` + * reads for the identical reason: a file an end-to-end test can point at, + * never a server a test could leave open. A thrown test there once skipped + * the server's own `close()`, and the leaked handle hung the whole suite + * with no failing test to point at, because a per-test timeout does not + * apply to a handle nobody closed. This reads a file instead, on every call, + * so there is never a handle to leak. + */ +function packumentFixtureTransport(fixturePath: string): RegistryTransport { + // Announced on stderr, every run, deliberately. This hook ships INSIDE the + // binary -- unlike the identical variable in scripts/retire-prereleases.mjs, + // which only maintainers run -- so it can substitute where `self check` and + // `self update` get their version facts. Nobody can set it in your + // environment without already being able to do worse, and it cannot cause a + // bad install because the install itself still resolves against the real + // registry. What it CAN do is hide that an update exists. A silent + // substitution of trusted data is the part worth refusing, so it is made + // impossible: if this is in effect, you are told. + process.stderr.write( + `ailoud: reading npm versions from the fixture ${fixturePath} (AILOUD_PACKUMENTS is set), not from the registry\n`, + ); + return async (url) => { + const name = decodeURIComponent(new URL(url).pathname.slice(1)); + const raw = await readFile(fixturePath, 'utf8'); + const all = JSON.parse(raw) as Record; + const packument = all[name]; + if (packument === undefined) return { status: 404, body: '' }; + return { status: 200, body: JSON.stringify(packument) }; + }; +} export interface CliContext { readonly paths: AiloudPaths; @@ -85,6 +136,22 @@ export interface CliContext { * missing. */ createDiarizer(): Diarizer; + /** + * What versions of ailoud are published, for `ailoud self check`. A port, + * not `NpmRegistry` directly, the same way every other engine on this + * context is: `createContext` is the only place that knows which provider + * backs it. + */ + readonly versionSource: VersionSource; + /** + * The registry host and timeout `versionSource` was built with. Kept + * alongside it rather than read back off it: `VersionSource` only + * promises `published()`, so a failed lookup could not otherwise name + * where it looked or how long it waited before giving up -- exactly what + * a check that could not run must report. + */ + readonly updateRegistryHost: string; + readonly updateTimeoutMs: number; } async function readConfigFile(path: string): Promise { @@ -95,6 +162,37 @@ async function readConfigFile(path: string): Promise { } } +/** + * Records that this run resolved a project's own library, so a later + * `ailoud self sync` has something to sweep. The per-user library is never + * entered here: it always exists, so listing it in the registry would only + * ever be noise. + * + * Registration is bookkeeping, not the user's request -- `createContext` runs + * before every command, including `doctor` and `ls`, and a full disk, a + * read-only home, or any other write failure here must never fail the + * command that triggered it. Any error is swallowed down to a single debug + * line. + */ +async function registerProjectLibrary(fs: Fs, clock: Clock, paths: AiloudPaths): Promise { + if (!paths.isProjectLibrary) return; + // `paths.dataDir` IS the project's `.ailoud/` in this branch (see + // `AiloudPaths.dataDir`'s own doc comment), so the project itself is that + // directory with the trailing `/.ailoud` stripped back off. + const projectPath = paths.dataDir.slice(0, -`/${PROJECT_DIR}`.length); + try { + await rememberProject( + { fs, clock, userDataDir: paths.userDataDir }, + { path: projectPath, libraryDir: paths.dataDir }, + ); + } catch (error) { + process.stderr.write( + `ailoud: debug: could not register project "${projectPath}": ` + + `${error instanceof Error ? error.message : String(error)}\n`, + ); + } +} + export async function createContext( env: Record, write: (line: string) => void = (line) => process.stdout.write(`${line}\n`), @@ -108,13 +206,16 @@ export async function createContext( const raw = await readConfigFile(paths.configFile); const config = parseConfig(raw); await mkdir(paths.mediaRoot, { recursive: true }); + const fs = new NodeFs(); + const clock = new SystemClock(); + await registerProjectLibrary(fs, clock, paths); return { paths, config, store: openStore(paths.dbFile), - fs: new NodeFs(), + fs, audio: new FfmpegAudioTool(), - clock: new SystemClock(), + clock, ids: new UlidIds(), write, ui: createUi(write), @@ -165,7 +266,7 @@ export async function createContext( if (llm.provider === 'openai-compatible') { const settings = llm.openaiCompatible; const apiKey = apiKeyFrom(env, 'OPENAI_API_KEY'); - if (settings.baseUrl.startsWith('https://api.openai.com') && apiKey === undefined) { + if (isHostedLlm(settings.baseUrl) && apiKey === undefined) { throw new EnvironmentError( 'No API key for the language model. Set AILOUD_LLM_API_KEY (or OPENAI_API_KEY) in ' + 'your environment. It is read from the environment on purpose and never from ' + @@ -235,5 +336,14 @@ export async function createContext( threads: config.stt.diarization.threads, }); }, + versionSource: new NpmRegistry({ + registry: UPDATE_REGISTRY, + timeoutMs: UPDATE_TIMEOUT_MS, + ...(env['AILOUD_PACKUMENTS'] === undefined || env['AILOUD_PACKUMENTS'] === '' + ? {} + : { transport: packumentFixtureTransport(env['AILOUD_PACKUMENTS']) }), + }), + updateRegistryHost: new URL(UPDATE_REGISTRY).host, + updateTimeoutMs: UPDATE_TIMEOUT_MS, }; } diff --git a/docs/development/architecture.md b/docs/development/architecture.md index d2765be..7cfc146 100644 --- a/docs/development/architecture.md +++ b/docs/development/architecture.md @@ -59,16 +59,16 @@ only ever move forward. ## The summary prompt -The prompt is measured, not guessed. `scripts/eval-summary-prompt.mjs` runs -variants three times each over seven transcripts -- English, Russian, -code-switched, long, multi-recording, undiarized, and a language override -- -across haiku, sonnet and opus, scoring each run for stated facts, invented -ones, language and length. - ``` node scripts/eval-summary-prompt.mjs --runs 3 node scripts/eval-summary-prompt.mjs --models haiku --cases one-on-one ``` +The prompt is measured, not guessed. Each variant runs three times over eight +transcripts -- English, Russian, code-switched, long, multi-recording, +one-on-one, undiarized, and a language override -- across haiku, sonnet and +opus. Every run +is scored for stated facts, invented facts, language and length. + Change the prompt or a template's headings, then re-run it. The previous measurement does not carry over. diff --git a/docs/development/releasing.md b/docs/development/releasing.md index 3cd0233..f47ae43 100644 --- a/docs/development/releasing.md +++ b/docs/development/releasing.md @@ -10,10 +10,16 @@ ## Tags -| Tag | Means | -| ------------------- | ---------------------------------------- | -| `v-rc.` | a release candidate, tagged on `develop` | -| `v` | a final release, tagged on `main` only | +| Tag | Means | npm dist-tag | +| -------------------- | ---------------------------------------- | ------------ | +| `v-dev.` | a snapshot, tagged on any branch | `dev` | +| `v-rc.` | a release candidate, tagged on `develop` | `next` | +| `v` | a final release, tagged on `main` only | `latest` | + +Only a final tag moves `latest`, so `npm install ailoud` never returns a +pre-release -- except for a package's very first publish, where npm sets +`latest` whatever `--tag` says. `latest` can be moved but not removed, so that +first pre-release answers `npm install ailoud` until a final version exists. ## Changelog limits @@ -28,25 +34,50 @@ bug, there is nothing to say. See ## Steps 1. Run the `pre-release-check` skill. It runs the whole gate plus the - documentation, changelog and version checks. + documentation, dependency, changelog and version checks. + 2. Run the `bump-version` skill. It sets the version across every - `package.json`, promotes the CHANGES.md Development section, and makes the - release commit. It does not tag or push. -3. Extract the release body: + `package.json` and promotes the CHANGES.md Development section. It does not + tag or push. + +3. Fold the pre-release sections in. `bump-version` comes first: ``` - node scripts/release-notes.mjs v1.2.3 + node scripts/fold-prereleases.mjs 1.2.3 + node scripts/check-changelog.mjs v1.2.3 ``` - It reads the `## Version 1.2.3` section and writes `RELEASE_NOTES.md`. It - exits non-zero if that section is missing, empty, or over the hard limit. + Folding first would leave `bump-version` an empty Development section to + promote, giving a second `## Version 1.2.3` heading that fails the check. -4. Merge to `main` and tag: + To read the notes as they will appear, `node scripts/release-notes.mjs +v1.2.3` writes them to `RELEASE_NOTES.md`. The release itself does not need + this -- `publish.yml` runs the same script. + +4. Commit, tag and push: ``` + git commit -am "chore: 1.2.3" git tag -s v1.2.3 -m "v1.2.3" - git push origin main --tags + git push origin main + git push origin v1.2.3 + ``` + + The tag is what starts everything else: `publish.yml` publishes the three + packages and creates the GitHub release from CHANGES.md; `docs.yml` then + publishes the site. Retiring the superseded snapshots is the one step left + to a human -- see below. + +5. Retire the snapshots this release supersedes, by hand: + ``` + pnpm retire 1.2.3 # prints the plan + NPM_TOKEN=npm_... pnpm retire 1.2.3 --yes # carries it out + ``` + + Deprecates every pre-release of that version -- `1.2.3-dev.*` and + `1.2.3-rc.*` alike -- moves the `dev` dist-tag onto the release, + and deletes the tags. See "Retiring pre-releases" below. ## Publishing to npm @@ -59,6 +90,17 @@ mints a short-lived OIDC token for the run, npm exchanges it for a credential good for minutes, and provenance is attached automatically. Nothing long-lived is stored, so there is no 90-day expiry to renew. +Except once, per package: a trusted publisher cannot be attached to a package +that does not exist yet, so the first version of each goes out on a token in +the `NPM_TOKEN` secret. The workflow uses the secret when present and OIDC when +not, so deleting the secret is the whole of the switch. + +It will not let that drift: a **pre-release** published on the token logs a +warning, and a **final release** with the secret still set fails before +publishing anything. Attaching the publisher (organization `lorem-dev`, +repository `ailoud`, workflow `publish.yml`, environment empty) on all three +package pages and deleting the secret clears it. + One-time setup on npmjs.com, per package -- Package, then Settings, then Trusted publisher, then GitHub Actions: @@ -74,24 +116,57 @@ half of what is needed: `pnpm pack` rewrites the `workspace:*` dependencies into real versions, which npm requires and will not do itself, and `npm publish` is the one with OIDC and provenance. -Before publishing it checks that all three manifests agree with the tag, then -runs the whole gate. A pre-release tag publishes under the `next` dist-tag, so -`npm install ailoud` keeps returning the last stable version. +Before publishing, and before anything is built, it refuses a release that +should not happen: a changelog unfit to release, an open high or critical code +scanning alert, a manifest that disagrees with the tag, or the `NPM_TOKEN` +secret still set on a final tag. Then it runs the whole gate. + +A code scanning finding blocks only while it is open. One that has been +reviewed is dismissed with its reason and does not block. + +## Retiring pre-releases + +After a final release, retire the snapshots it supersedes: + +``` +pnpm retire 1.0.0 # prints the plan +NPM_TOKEN=npm_... pnpm retire 1.0.0 --yes # carries it out +``` + +Run it after every final release. Nothing prompts for it, and until it runs +`npm install ailoud@dev` hands out an older build than `npm install ailoud`. +`NPM_TOKEN` is a granular access token with read-and-write on the three +packages; without it npm asks for a 2FA code on each of the twelve writes. + +Deprecating, not unpublishing: a deprecated version keeps every pinned install +working and prints a notice on the next one. It also drops the `dev` dist-tag, +and deletes the tags -- but only those whose commit is reachable from `main`, +because the published provenance attests that commit. The rest are reported and +left in place. + +This is a manual step, run under `npm login`: trusted publishing authenticates +`npm publish` and nothing else, so the token it returns cannot deprecate. + +If the npm side does not complete, the script leaves the tags alone and exits +non-zero. The tags are what name which pre-releases to retire, so deleting them +after a failed deprecation would destroy the only record of what was missed. ## What a tag triggers -Pushing a final tag runs -[`.github/workflows/docs.yml`](https://github.com/lorem-dev/ailoud/blob/main/.github/workflows/docs.yml), -which publishes the documentation for that version to the `gh-pages` branch -with [mike](https://github.com/jimporter/mike) and moves the `latest` alias -that the site root redirects to. +Pushing a tag runs `publish.yml`. When it succeeds, +[`.github/workflows/docs.yml`](https://github.com/lorem-dev/ailoud/blob/main/.github/workflows/docs.yml) +runs on its completion and publishes the documentation for that version to the +`gh-pages` branch with [mike](https://github.com/jimporter/mike), moving the +`latest` alias that the site root redirects to. `publish.yml` also creates the +GitHub release, with the body taken from the `## Version ` section of +CHANGES.md by `scripts/release-notes.mjs`. Nothing else publishes documentation. A push to a branch publishes nothing, so what is online always describes a version someone can install. -A pre-release tag (`v1.2.3-rc.1`, or any tag with a `-` qualifier) publishes -nothing either. The workflow refuses it twice: the tag filter never starts it, -and the job checks again in case `workflow_dispatch` was pointed at one. +A pre-release publishes nothing either. It reaches docs.yml -- `publish.yml` +runs for pre-releases too -- and the job stops once it reads the version from +the published commit's manifest and finds a `-` in it. ## One-time repository setup diff --git a/docs/index.md b/docs/index.md index 505d7ae..262eda5 100644 --- a/docs/index.md +++ b/docs/index.md @@ -16,10 +16,10 @@ ailoud audio summarize ID001 --template one-on-one ## Start here - [Getting Started](getting-started.md) -- install it, transcribe your first file. +- [MCP](mcp.md) -- let an agent use your library. - [Recordings](usage/recordings.md) -- import, transcribe, tag, annotate. - [Search](usage/search.md) -- find where something was said. - [Summaries](usage/summaries.md) -- make reports and read them back. -- [MCP](mcp.md) -- let an agent use your library. ## What it is good at diff --git a/docs/mcp.md b/docs/mcp.md index c8c8a31..2eabaf1 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -73,24 +73,9 @@ Your own files are safe: is not JSON at all is refused with a message rather than rewritten. !!! note -Comments in a JSON or `.jsonc` MCP config do not survive an edit -- the file -is parsed and re-serialised. TOML and YAML keep theirs. -Your own config files are safe: - -- Only the text between the two markers is ever rewritten. A file that merely - _mentions_ `` in prose keeps everything around it. -- A `config.toml` that already defines an `ailoud` server some other way is - refused, not edited. Two definitions of one key is a TOML error, and it - would break your whole Codex config rather than just this server. -- A Hermes `config.yaml` with your own settings or comments is rewritten, never - deleted. Only a file holding nothing but AILoud's own keys is removed. -- A trailing comma or a byte-order mark in a `.jsonc` is tolerated. A file that - is not JSON at all is refused with a message, not rewritten. - -!!! note -Comments in a JSON or `.jsonc` MCP config do not survive an edit -- the -file is parsed and re-serialised. TOML and YAML keep theirs. + Comments in a JSON or `.jsonc` MCP config do not survive an edit -- the + file is parsed and re-serialised. TOML and YAML keep theirs. ### Supported agents @@ -123,21 +108,10 @@ model files, which are not a property of a project. ## By hand -=== "Claude Code" - - `.mcp.json` in your project, or `~/.claude.json` for every project: - - ```json - { - "mcpServers": { - "ailoud": { "command": "ailoud", "args": ["mcp"] } - } - } - ``` - -=== "Claude Desktop" +=== "Claude Code / Claude Desktop" - `claude_desktop_config.json`: + `.mcp.json` in your project, `~/.claude.json` for every project, or + `claude_desktop_config.json` for Claude Desktop: ```json { @@ -173,8 +147,7 @@ Check it works: echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}' | ailoud mcp ``` -It serves the same library the CLI uses. Anything you import in the shell is -visible to the agent, and the other way round. +It serves the same library the CLI uses, in both directions. ## Ask it things diff --git a/docs/usage/cli.md b/docs/usage/cli.md index 7a343e9..dd4c587 100644 --- a/docs/usage/cli.md +++ b/docs/usage/cli.md @@ -44,7 +44,7 @@ ailoud report l The old top-level spellings still work: `ailoud ls`, `ailoud show`, `ailoud rm`, `ailoud annotate`, `ailoud import`, `ailoud transcribe`, -`ailoud summarize`. +`ailoud summarize`, `ailoud search`. ## audio import diff --git a/docs/usage/configuration.md b/docs/usage/configuration.md index 846a02f..8e4b3de 100644 --- a/docs/usage/configuration.md +++ b/docs/usage/configuration.md @@ -125,9 +125,10 @@ ailoud setup --llm claude-api --llm-model claude-opus-5 --yes ``` !!! warning "Context size is not adjusted for you" -No provider reports a model's context window, so switching to a -small-context model needs `contextTokens` set by hand. The symptom is a -context error from the API on a long transcript. + + No provider reports a model's context window, so switching to a + small-context model needs `contextTokens` set by hand. The symptom is a + context error from the API on a long transcript. ## When doctor is unhappy diff --git a/docs/usage/recordings.md b/docs/usage/recordings.md index 70d2460..50e09b9 100644 --- a/docs/usage/recordings.md +++ b/docs/usage/recordings.md @@ -1,5 +1,31 @@ # Recordings +## Flags + +Where a flag means different things to different verbs, it gets a row each. + +| Flag | Verb | Does | +| ------------------------ | ---------------- | -------------------------------------------------------------------------------------------------- | +| `--tag ` | import | tag the imported recordings; repeatable | +| `--tag ` | transcribe | group these recordings under a tag; repeatable | +| `--tag ` | annotate | group this recording under a tag; repeatable | +| `--tag ` | ls | only recordings carrying this tag; repeatable | +| `--title ` | import, annotate | the recording's title | +| `--notes ` | import, annotate | free-form context about the recording | +| `--lang ` | transcribe | spoken language, several comma-separated, or `auto`. Naming two or more turns on multilingual mode | +| `--multilingual` | transcribe | segment by speech and language, transcribing each run separately | +| `--model ` | transcribe | override the configured model | +| `--diarize` | transcribe | attribute segments to speakers | +| `--speakers ` | transcribe | known number of speakers, to help the diarizer | +| `--speakers` | show | list who spoke, instead of the transcript -- takes no value | +| `--speaker ` | annotate | a real name for one diarizer label; repeatable | +| `--speaker ` | show | only this speaker, by label or by the name you gave them | +| `--transcript ` | show | a specific transcript instead of the newest; a prefix will do | +| `--format ` | show | `text`, `json`, `srt`, `vtt` (default `text`) | +| `--json` | ls | print one JSON array of rows instead of a table | +| `--force` | transcribe | re-transcribe recordings that already have a transcript | +| `--force` | rm | delete without asking | + ## Import ``` @@ -13,8 +39,9 @@ subdirectories. The file you point at is never moved or changed. AILoud keeps its own copy. !!! tip "Always pass `--tag`" -Tags are how you find a recording later by context. The easiest moment to -add one is now, while you know what the file is. See [Tags](#tags). + + Tags are how you find a recording later by context. The easiest moment to + add one is now, while you know what the file is. See [Tags](#tags). ## Transcribe @@ -86,8 +113,6 @@ ailoud audio ls --tag release ailoud audio ls --tag release --tag backend # both, not either ``` -Several tags narrow. A recording must carry all of them. - ## Titles and notes ``` diff --git a/docs/usage/templates.md b/docs/usage/templates.md index f45498b..36173aa 100644 --- a/docs/usage/templates.md +++ b/docs/usage/templates.md @@ -89,5 +89,6 @@ A template needs a `context` sentence and at least two headings. One heading is a title, not a shape. !!! tip -Before writing a template, try `--context` on an existing one. It adjusts -a summary without adding a shape you then have to maintain. + + Before writing a template, try `--context` on an existing one. It adjusts + a summary without adding a shape you then have to maintain. diff --git a/docs/usage/updating.md b/docs/usage/updating.md new file mode 100644 index 0000000..e873da5 --- /dev/null +++ b/docs/usage/updating.md @@ -0,0 +1,42 @@ +# Updating ailoud + +```shell +ailoud self update +``` + +`self update` asks the registry itself, so nothing has to be run before it. It +installs the newer version if there is one, then refreshes the rules block in +every registered project. + +```shell +ailoud self check # only look, change nothing +ailoud self check --json # the same answer, for a script +ailoud self sync # refresh the rules without updating +``` + +`self check` is for looking without installing; `self update` does not need +it. + +## What counts as newer + +| You are on | Can move to | +| --------------- | ----------------------------------------------------- | +| a final release | any newer final release | +| `X.Y.Z-dev.N` | a newer `dev` of the same `X.Y.Z`, or any newer final | +| `X.Y.Z-rc.N` | a newer `rc` of the same `X.Y.Z`, or any newer final | + +A deprecated version is never offered. + +## Turning off the passive notice + +Other commands mention an update at most once a day. `self check` and +`self update` themselves are unaffected by either switch. + +```shell +export AILOUD_NO_UPDATE_CHECK=1 +``` + +```yaml +update: + check: false +``` diff --git a/e2e/src/cli.ts b/e2e/src/cli.ts index 9541c1a..15c8d72 100644 --- a/e2e/src/cli.ts +++ b/e2e/src/cli.ts @@ -2,12 +2,12 @@ // built binary through here, and nowhere else. `makeSandbox()` is the only // export that can start the binary, and the process it spawns always gets // a throwaway HOME, XDG_CONFIG_HOME, and XDG_DATA_HOME. There is no -// exported raw spawn and no way to pass a caller-supplied env that could -// override those three variables -- forgetting any one of them would let a -// spec write into the developer's real library, which is exactly the -// failure this file exists to prevent. +// exported raw spawn, and `run`'s own `env` option cannot override those +// three variables no matter what a caller passes -- forgetting any one of +// them would let a spec write into the developer's real library, which is +// exactly the failure this file exists to prevent. import { spawn } from 'node:child_process'; -import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'; +import { mkdir, mkdtemp, realpath, rm, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { dirname, join } from 'node:path'; @@ -23,6 +23,26 @@ export interface CliResult { readonly stderr: string; } +export interface RunOptions { + /** + * Extra variables for this one call, layered UNDER the sandbox's own + * HOME/XDG_CONFIG_HOME/XDG_DATA_HOME -- those three always win, even if a + * caller names them here. What this is for: threading `AILOUD_PACKUMENTS` + * (a JSON fixture path, never a server -- see `packumentFixture` in + * `self-update.spec.ts`) or a stub-tool directory prepended to `PATH`. + */ + readonly env?: Record; + /** + * Overrides the child's cwd for this one call, still somewhere under the + * sandbox. Defaults to `projectDir`. Load-bearing for the specs that + * register more than one project directory in the same sandbox: `self + * sync` sweeps `projects.json`, whose entries are real paths, so testing a + * pruned or a concurrently-registered project needs more than one such + * path to exist. + */ + readonly cwd?: string; +} + export interface Sandbox { /** The sandboxed $HOME. Nothing outside it should ever be touched. */ readonly home: string; @@ -40,7 +60,7 @@ export interface Sandbox { */ readonly projectDir: string; /** Runs the built binary with this sandbox's environment. The only way a spec invokes it. */ - run(args: readonly string[]): Promise; + run(args: readonly string[], options?: RunOptions): Promise; /** Writes config.yaml inside the sandbox, creating its parent directory. */ writeConfig(yaml: string): Promise; /** Removes the sandbox directory. Call once the test is done with it. */ @@ -73,8 +93,33 @@ function runProcess( }); } +/** + * The parent's own environment, minus the variables that have bitten this + * project's tests before: every `GITHUB_` variable (CI sets several that + * change script behaviour -- see `scripts/testing/harness.mjs`) and + * `AILOUD_NO_UPDATE_CHECK`, so a spec's outcome never depends on whatever + * happened to be exported in the shell -- or the CI job -- that ran it. + */ +function scrubbedEnv(): NodeJS.ProcessEnv { + return Object.fromEntries( + Object.entries(process.env).filter( + ([key]) => !key.startsWith('GITHUB_') && key !== 'AILOUD_NO_UPDATE_CHECK', + ), + ); +} + export async function makeSandbox(): Promise { - const home = await mkdtemp(join(tmpdir(), 'ailoud-e2e-')); + // Resolved through realpath immediately: on macOS, os.tmpdir() answers + // under /var/folders, but the OS reports a spawned child's own cwd already + // canonicalised to /private/var/folders -- the same directory, a + // different string. Left unresolved here, `sandbox.projectDir` would not + // byte-for-byte equal a path the CLI itself prints or records (a project + // registry entry, `self sync`'s own report), while a bare `toContain()` + // check could still pass by accident: the unresolved form is a plain + // substring of the canonical one. Resolving once, here, is what makes + // every path this sandbox hands out compare equal to what the binary + // actually sees. + const home = await realpath(await mkdtemp(join(tmpdir(), 'ailoud-e2e-'))); const configHome = join(home, 'config'); const dataHome = join(home, 'data'); const configFile = join(configHome, 'ailoud', 'config.yaml'); @@ -82,23 +127,27 @@ export async function makeSandbox(): Promise { const projectDir = join(home, 'project'); await mkdir(projectDir, { recursive: true }); - // Every one of these three variables matters: dropping any single one - // falls back to the real $HOME-derived default in apps/cli/src/config.ts - // and points the binary at the developer's actual library. - const env: NodeJS.ProcessEnv = { - ...process.env, - HOME: home, - XDG_CONFIG_HOME: configHome, - XDG_DATA_HOME: dataHome, - }; + const base = scrubbedEnv(); return { home, configFile, dataDir, projectDir, - run(args) { - return runProcess(args, env, projectDir); + run(args, options) { + // Every one of these three matters: dropping any single one falls back + // to the real $HOME-derived default in apps/cli/src/config.ts and + // points the binary at the developer's actual library. Applied LAST, + // after any caller-supplied `options.env`, so nothing above can shadow + // them. + const env: NodeJS.ProcessEnv = { + ...base, + ...options?.env, + HOME: home, + XDG_CONFIG_HOME: configHome, + XDG_DATA_HOME: dataHome, + }; + return runProcess(args, env, options?.cwd ?? projectDir); }, async writeConfig(yaml) { await mkdir(dirname(configFile), { recursive: true }); diff --git a/e2e/tests/self-update.spec.ts b/e2e/tests/self-update.spec.ts new file mode 100644 index 0000000..2cc180f --- /dev/null +++ b/e2e/tests/self-update.spec.ts @@ -0,0 +1,418 @@ +// End-to-end coverage of `ailoud self check|update|sync` and the project +// registry behind `self sync`, driven through the built binary rather than +// in-memory fakes. Tasks 1-12 already proved the logic against fakes; this +// is what proves the wiring around it -- process spawning, file paths, +// `projects.json` -- is real. +// +// No spec here ever reaches a real `npm install -g` or `pnpm add -g`. Run +// from its own checkout, the repository's own binary is never an installed +// `npm-global` or `pnpm-global` copy, so `self update` naturally refuses with +// a hint instead of installing anything -- see the "refuses" specs below, +// which exercise that natural path with no stubbing at all. The one spec +// that needs the OTHER branch, to see `--dry-run`'s plan, reaches it by +// putting a stub `npm` first on PATH that only ever answers `root -g` with +// the repository's own directory. Even that cannot lead to a real install: +// the install command `--dry-run` prints is built from `installCommandFor` +// (packages/providers/src/update/installMethod.ts), anchored to the real +// node binary's own directory, never to anything found on PATH, and +// `--dry-run` returns before any command is ever spawned regardless. +import { chmodSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { mkdir, readFile, rm, writeFile } from 'node:fs/promises'; +import { execFileSync } from 'node:child_process'; +import { join } from 'node:path'; +import { makeSandbox } from '../src/cli'; +import type { Sandbox } from '../src/cli'; + +const REPO_ROOT = join(__dirname, '..', '..'); + +/** A quick, local command; a few seconds is already generous. */ +const GIT_STATUS_TIMEOUT_MS = 10_000; + +const START = ''; + +jest.setTimeout(120_000); + +function gitStatus(): string { + return execFileSync('git', ['status', '--porcelain'], { + cwd: REPO_ROOT, + encoding: 'utf8', + timeout: GIT_STATUS_TIMEOUT_MS, + }); +} + +// Captured before any sandbox runs, so the final check below can prove THIS +// suite left the repository exactly as it found it -- not that the tree was +// clean to begin with. A developer with uncommitted work is the common case, +// not an edge case, and a check that fails for that reason is a check that +// gets ignored. +const statusBeforeSuite = gitStatus(); + +const read = (path: string): Promise => readFile(path, 'utf8'); + +async function exists(path: string): Promise { + try { + await readFile(path); + return true; + } catch { + return false; + } +} + +/** Mirrors `ProjectEntry` (apps/cli/src/projects.ts), for reading and + * rewriting `projects.json` directly in a spec. */ +interface RegistryEntry { + readonly path: string; + readonly firstSeen: string; + readonly lastSeen: string; + readonly libraryDir?: string; + readonly rulesVersion?: string; +} + +async function readRegistry(path: string): Promise { + return JSON.parse(await read(path)) as readonly RegistryEntry[]; +} + +/** The version this checkout's own manifest names -- never hard-coded, so a + * release bump never leaves this file asserting a stale number. */ +function currentVersion(): string { + const manifest = join(REPO_ROOT, 'apps', 'cli', 'package.json'); + const parsed = JSON.parse(readFileSync(manifest, 'utf8')) as { version?: unknown }; + if (typeof parsed.version !== 'string' || parsed.version === '') { + throw new Error(`no version in ${manifest}`); + } + return parsed.version; +} + +/** A newer final release than `version`, which must itself be a final + * release: `chooseUpdateTarget` only ever offers a final release a newer + * final release (packages/core/src/domain/version.ts), never a pre-release. */ +function newerRelease(version: string): string { + const match = /^(\d+)\.(\d+)\.(\d+)$/.exec(version); + if (match === null) { + throw new Error(`apps/cli/package.json's version is not a plain release: ${version}`); + } + const [, major, minor, patch] = match; + return `${major}.${minor}.${Number(patch) + 1}`; +} + +/** + * The registry's document for `ailoud`, written to a fixture file -- never a + * server. See `AILOUD_PACKUMENTS` in `scripts/retire-prereleases.test.mjs`, + * whose own doc comment tells the story this follows: an HTTP stub tried + * here once left a throwing test's server handle open, and the leaked + * handle hung the entire suite with no failing test to point at, because a + * per-test timeout does not apply to a handle nobody closed. A file behind + * an environment variable cannot leak that way -- there is no handle to + * leave open. + */ +function packumentFixture(dir: string, versions: readonly string[]): string { + const path = join(dir, 'packuments.json'); + const one = { versions: Object.fromEntries(versions.map((v) => [v, {}])) }; + writeFileSync(path, JSON.stringify({ ailoud: one })); + return path; +} + +/** + * A directory holding an `npm` that only ever answers `root -g`, with + * `rootDir` -- for putting first on PATH so `self update --dry-run` detects + * an `npm-global` install instead of this repository's natural refusal, the + * one other branch the design calls for exercising. Anything else asked of + * it fails loudly rather than doing something unexpected. + */ +function stubNpmRoot(dir: string, rootDir: string): string { + const bin = join(dir, 'stub-bin'); + mkdirSync(bin, { recursive: true }); + const script = join(bin, 'npm'); + writeFileSync( + script, + `#!/bin/sh\n` + + `if [ "$1" = "root" ] && [ "$2" = "-g" ]; then\n` + + ` echo "${rootDir}"\n` + + ` exit 0\n` + + `fi\n` + + `echo "stub npm: unexpected args: $@" >&2\n` + + `exit 1\n`, + ); + chmodSync(script, 0o755); + return bin; +} + +describe('ailoud self check', () => { + let sandbox: Sandbox; + + beforeEach(async () => { + sandbox = await makeSandbox(); + }); + + afterEach(async () => { + await sandbox.cleanup(); + }); + + it('reports the target it would take, against a stub registry', async () => { + const current = currentVersion(); + const target = newerRelease(current); + const fixture = packumentFixture(sandbox.home, [current, target]); + + const result = await sandbox.run(['self', 'check', '--json'], { + env: { AILOUD_PACKUMENTS: fixture }, + }); + + expect(result.code).toBe(0); + const parsed = JSON.parse(result.stdout.trim()) as { + current: string; + target: string | null; + updatable: boolean; + }; + expect(parsed.current).toBe(current); + expect(parsed.target).toBe(target); + expect(parsed.updatable).toBe(true); + }); +}); + +describe('ailoud self update', () => { + let sandbox: Sandbox; + + beforeEach(async () => { + sandbox = await makeSandbox(); + }); + + afterEach(async () => { + await sandbox.cleanup(); + }); + + it('prints a plan and changes nothing with --dry-run', async () => { + const current = currentVersion(); + const target = newerRelease(current); + const fixture = packumentFixture(sandbox.home, [current, target]); + const stubBin = stubNpmRoot(sandbox.home, REPO_ROOT); + + const registryBefore = await exists(join(sandbox.dataDir, 'projects.json')); + expect(registryBefore).toBe(false); + + const result = await sandbox.run(['self', 'update', '--dry-run'], { + env: { + AILOUD_PACKUMENTS: fixture, + PATH: `${stubBin}:${process.env['PATH'] ?? ''}`, + }, + }); + + expect(result.code).toBe(0); + expect(result.stdout).toContain(`Current version: ${current}`); + expect(result.stdout).toContain(`Target version: ${target}`); + expect(result.stdout).toMatch(/Install command: .*npm .*install -g ailoud@/); + expect(result.stdout).toContain('Dry run: nothing was changed.'); + + // No install: nothing this test could observe short of a real global + // write, which the anchored, never-PATH-resolved install command and + // the early dry-run return both already rule out. + // No sweep, no log write, no registry write. + expect(await exists(join(sandbox.dataDir, 'update.log'))).toBe(false); + expect(await exists(join(sandbox.dataDir, 'projects.json'))).toBe(false); + }); + + it('refuses to update a project dependency and names the command to run', async () => { + const current = currentVersion(); + const target = newerRelease(current); + const fixture = packumentFixture(sandbox.home, [current, target]); + + const result = await sandbox.run(['self', 'update'], { + env: { AILOUD_PACKUMENTS: fixture }, + }); + + // This repository's own checkout is never an npm-global or pnpm-global + // install, so detectInstallMethod refuses -- naturally, with no stubbing + // -- and names a command to run instead of installing anything. + expect(result.code).toBe(0); + expect(result.stdout).toMatch( + /npm install -g ailoud@|pnpm add -g ailoud@|add command in|npx ailoud@/, + ); + }); + + it('refuses under --force with a non-zero exit', async () => { + const current = currentVersion(); + const target = newerRelease(current); + const fixture = packumentFixture(sandbox.home, [current, target]); + + const result = await sandbox.run(['self', 'update', '--force'], { + env: { AILOUD_PACKUMENTS: fixture }, + }); + + expect(result.code).not.toBe(0); + expect(result.stderr).toMatch(/cannot install this way/); + }); +}); + +describe('ailoud self sync', () => { + let sandbox: Sandbox; + + beforeEach(async () => { + sandbox = await makeSandbox(); + }); + + afterEach(async () => { + await sandbox.cleanup(); + }); + + it('refreshes a rules block, and is idempotent on a second run', async () => { + const claudeMd = join(sandbox.projectDir, 'CLAUDE.md'); + + // `mcp install --location local` registers the project itself, with this + // build's rules version, the moment it succeeds (see + // `registerAfterInstall` in apps/cli/src/commands/mcpInstall.ts) -- no + // second command is needed to put it in projects.json. + await sandbox.run(['mcp', 'install', '--target', 'claude', '--location', 'local']); + + // Simulate an older ailoud having written a different block, the same + // way e2e/tests/mcp-install.spec.ts does for `mcp update`. + const before = await read(claudeMd); + const stale = before.replace( + /[\s\S]*/, + `${START}\nold text\n`, + ); + await writeFile(claudeMd, stale, 'utf8'); + + const first = await sandbox.run(['self', 'sync']); + expect(first.code).toBe(0); + expect(first.stdout).toContain(`refreshed: ${sandbox.projectDir}`); + const refreshed = await read(claudeMd); + expect(refreshed).not.toContain('old text'); + expect(refreshed).toContain('search_transcripts'); + + // Idempotent: a second sweep with nothing stale must say `current`, not + // `refreshed` -- a sweep over many projects must never claim an edit it + // did not make, which is the whole reason this command exists. + const second = await sandbox.run(['self', 'sync']); + expect(second.code).toBe(0); + expect(second.stdout).toContain(`current: ${sandbox.projectDir}`); + expect(second.stdout).not.toContain(`refreshed: ${sandbox.projectDir}`); + expect(await read(claudeMd)).toBe(refreshed); + }); + + it('reports and prunes a project whose directory is gone', async () => { + const goneDir = join(sandbox.home, 'gone-project'); + await mkdir(goneDir, { recursive: true }); + await sandbox.run(['mcp', 'install', '--target', 'claude', '--location', 'local'], { + cwd: goneDir, + }); + + const registryPath = join(sandbox.dataDir, 'projects.json'); + const before = await readRegistry(registryPath); + expect(before.some((entry) => entry.path === goneDir)).toBe(true); + + await rm(goneDir, { recursive: true, force: true }); + + const result = await sandbox.run(['self', 'sync']); + expect(result.code).toBe(0); + expect(result.stdout).toContain(`gone: ${goneDir}`); + + const after = await readRegistry(registryPath); + expect(after.some((entry) => entry.path === goneDir)).toBe(false); + }); +}); + +describe('the project registry', () => { + let sandbox: Sandbox; + + beforeEach(async () => { + sandbox = await makeSandbox(); + }); + + afterEach(async () => { + await sandbox.cleanup(); + }); + + it('records a project in projects.json after a command uses its library', async () => { + const registryPath = join(sandbox.dataDir, 'projects.json'); + expect(await exists(registryPath)).toBe(false); + + // `mcp install --location local` both creates the project library AND + // registers it (`registerAfterInstall` in + // apps/cli/src/commands/mcpInstall.ts) -- Task 10's own proof, kept + // honest here against the real binary. + const result = await sandbox.run([ + 'mcp', + 'install', + '--target', + 'claude', + '--location', + 'local', + ]); + expect(result.code).toBe(0); + + const registry = await readRegistry(registryPath); + expect(registry).toHaveLength(1); + expect(registry[0]?.path).toBe(sandbox.projectDir); + expect(registry[0]?.rulesVersion).toBeDefined(); + }); + + it('keeps projects.json valid when two commands run at once', async () => { + const dirA = join(sandbox.home, 'project-a'); + const dirB = join(sandbox.home, 'project-b'); + await mkdir(dirA, { recursive: true }); + await mkdir(dirB, { recursive: true }); + + // Each project's OWN local library is created first, sequentially: with + // neither directory holding a `.ailoud/` yet, `mcp install` falls back to + // opening the shared PER-USER library just long enough to write one, and + // running that step for both projects at once would race two processes + // on THAT single sqlite file -- a real hazard, but a different one from + // what this spec is about. One at a time here isolates the race to the + // one file this spec targets: projects.json. + await sandbox.run(['mcp', 'install', '--target', 'claude', '--location', 'local'], { + cwd: dirA, + }); + await sandbox.run(['mcp', 'install', '--target', 'claude', '--location', 'local'], { + cwd: dirB, + }); + + const registryPath = join(sandbox.dataDir, 'projects.json'); + // Back-dated past rememberProject's 24-hour throttle (projects.ts), so + // the concurrent commands below actually write instead of silently + // no-op'ing on an entry seen moments ago. + const justRegistered = await readRegistry(registryPath); + const backdated = justRegistered.map((entry) => ({ + ...entry, + lastSeen: '2000-01-01T00:00:00.000Z', + })); + await writeFile(registryPath, `${JSON.stringify(backdated, null, 2)}\n`, 'utf8'); + + // NOW genuinely concurrent: two processes, each already holding its own + // project-local library (no shared sqlite file left to race on), both + // touching the one file that IS shared -- projects.json -- at once. + // `writeRegistry`'s own doc comment (projects.ts) is explicit that its + // re-read-before-rename only NARROWS the lost-update window to the + // rename itself, rather than closing it -- a write landing inside that + // window is expected to lose its OWN timestamp bump, self-healing on the + // next run, and is not this test's concern. What must never happen is + // the file going invalid, or either project's entry disappearing + // outright, which is what is asserted below. + // + // And this spec is NOT where the race guarantee is pinned. Two real + // subprocesses may simply not overlap inside the critical section, so a + // green run here is evidence rather than proof. The deterministic case -- + // a rival that commits between the re-read and the rename, every time -- + // lives in `apps/cli/src/projects.test.ts` under `RacingFs`. Read that + // one if you are changing `writeRegistry`; this one only catches a + // regression crude enough to survive real scheduling. + await Promise.all([sandbox.run(['ls'], { cwd: dirA }), sandbox.run(['ls'], { cwd: dirB })]); + + const registry = await readRegistry(registryPath); + expect(Array.isArray(registry)).toBe(true); + expect(registry).toHaveLength(2); + expect(registry.some((entry) => entry.path === dirA)).toBe(true); + expect(registry.some((entry) => entry.path === dirB)).toBe(true); + }); +}); + +it('leaves the repository working tree exactly as it found it', () => { + // Runs after every other spec's sandbox has been torn down. Compared + // against the snapshot taken before the suite started, not against + // "empty": asserting `git status --porcelain` is empty fails for any + // developer with uncommitted work already in progress, which is most of + // the time, and a check that fails for reasons unrelated to its subject + // is a check that gets ignored. Identical before/after still fails the + // instant any spec, or the harness itself, writes into the repository + // rather than into its own sandbox -- the one thing this test exists to + // catch. + expect(gitStatus()).toBe(statusBeforeSuite); +}); diff --git a/eslint.config.mjs b/eslint.config.mjs index f263d02..5a63ff7 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -12,19 +12,44 @@ const LAYER_MAY_IMPORT = { export default tseslint.config( { - // scripts/** holds plain Node utility scripts (release tooling, not - // part of the typed packages/apps source tree); node --check is their - // syntax gate instead. ignores: [ '**/dist/**', '**/coverage/**', '**/node_modules/**', // mkdocs build output: third-party minified JS, not ours to lint. 'site/**', - '**/*.config.{js,mjs,cjs,ts}', - 'scripts/**', ], }, + { + // The build's configuration and the release scripts. Both were ignored, + // which meant every editor reported "File ignored because of a matching + // ignore pattern" on opening one, and a mistake in them was caught by + // nothing -- `node --check` was the only gate on scripts/, and it sees + // syntax, not an unused variable or a misspelled identifier. They are + // Node, not part of the typed source tree, so they get the recommended + // rules and Node globals rather than the type-aware config. + files: ['*.config.{js,mjs,cjs,ts}', '**/*.config.{js,mjs,cjs,ts}', 'scripts/**/*.mjs'], + languageOptions: { + // The Node globals these files actually use. Spelled out rather than + // pulled from a globals package: it is a short list, and a new name + // appearing here should be a deliberate addition. + globals: { + console: 'readonly', + process: 'readonly', + module: 'writable', + require: 'readonly', + __dirname: 'readonly', + Buffer: 'readonly', + URL: 'readonly', + fetch: 'readonly', + }, + }, + rules: { + // A .cjs file uses require/module, which the type-aware rules would + // otherwise flag in a "type": "module" workspace. + '@typescript-eslint/no-require-imports': 'off', + }, + }, js.configs.recommended, ...tseslint.configs.recommended, { @@ -40,13 +65,28 @@ export default tseslint.config( files: ['packages/*/src/**/*.ts', 'apps/*/src/**/*.ts'], plugins: { boundaries }, settings: { + // Both src and dist. A cross-package import can be written two ways: + // `../../providers/src/index.js` resolves to a source file, but + // `@ailoud/providers` resolves through node_modules to the built + // `dist/index.js` -- which matched no element, so the rule classified it + // as unknown and said nothing at all. Listing dist under the same type + // makes both forms the same violation. It does mean the second form is + // only caught once dist exists, which is why the gate and CI both build + // before they lint. 'boundaries/elements': [ - { type: 'core', pattern: 'packages/core/src/**' }, - { type: 'providers', pattern: 'packages/providers/src/**' }, - { type: 'cli', pattern: 'apps/cli/src/**' }, + { type: 'core', pattern: ['packages/core/src/**', 'packages/core/dist/**'] }, + { + type: 'providers', + pattern: ['packages/providers/src/**', 'packages/providers/dist/**'], + }, + { type: 'cli', pattern: ['apps/cli/src/**', 'apps/cli/dist/**'] }, ], + // The root tsconfig, which includes every package's sources, rather + // than a glob over the per-package ones: the resolver warns about + // multiple projects, and one project covering the whole workspace is + // both quieter and faster. 'import/resolver': { - typescript: { project: ['packages/*/tsconfig.json', 'apps/*/tsconfig.json'] }, + typescript: { project: 'tsconfig.json' }, }, }, rules: { @@ -54,10 +94,12 @@ export default tseslint.config( 'error', { default: 'disallow', - policies: Object.entries(LAYER_MAY_IMPORT).map(([from, to]) => ({ - from: { element: { type: from } }, - allow: { to: { element: { types: { anyOf: to } } } }, - })), + policies: [ + ...Object.entries(LAYER_MAY_IMPORT).map(([from, to]) => ({ + from: { element: { type: from } }, + allow: { to: { element: { types: { anyOf: to } } } }, + })), + ], }, ], }, diff --git a/fixtures/en-short.mkv b/fixtures/en-short.mkv new file mode 100644 index 0000000..0e38a2d --- /dev/null +++ b/fixtures/en-short.mkv @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1f05f1d481205693d249df834dfb8fd7ef706db41932d07afca49a7cc3bdf2e3 +size 9879 diff --git a/fixtures/en-short.mov b/fixtures/en-short.mov new file mode 100644 index 0000000..24e5ea0 --- /dev/null +++ b/fixtures/en-short.mov @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fc8f99f5540c772a26017328a69c05f6a8c4d07e6f105fb00b03a3cd10176214 +size 13266 diff --git a/fixtures/en-short.mp4 b/fixtures/en-short.mp4 new file mode 100644 index 0000000..2009cac --- /dev/null +++ b/fixtures/en-short.mp4 @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c984e208272ef1ae391e07d0a57fd5fa505ca40f4a0932429a366b968b414de6 +size 13215 diff --git a/fixtures/en-short.webm b/fixtures/en-short.webm new file mode 100644 index 0000000..57a0da4 --- /dev/null +++ b/fixtures/en-short.webm @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c77373b0cd1330b4e64d1c120828efd3306a3856382c2737e175a73091d791f2 +size 9117 diff --git a/jest.config.cjs b/jest.config.cjs index 7fb320e..3e7d1a2 100644 --- a/jest.config.cjs +++ b/jest.config.cjs @@ -39,17 +39,31 @@ const shared = { module.exports = { maxWorkers: 2, detectOpenHandles: true, + // Also at the root, not only inside each project. Jest takes the per-test + // timeout from the GLOBAL config, and with `projects` a value set only on a + // project reaches `configs[].testTimeout` while `globalConfig.testTimeout` + // stays undefined -- so every test silently fell back to Jest's 5 s default. + // Locally that was masked: the transcribe specs were failing fast for a + // missing model, so nothing ran long enough to hit it. On CI, where the + // model is there, whisper takes tens of seconds and every one timed out. + testTimeout: 600_000, projects: [ { ...shared, displayName: 'no-tools', - testMatch: ['/e2e/tests/mcp-install.spec.ts'], + testMatch: [ + '/e2e/tests/mcp-install.spec.ts', + '/e2e/tests/self-update.spec.ts', + ], }, { ...shared, displayName: 'tools', testMatch: ['/e2e/tests/**/*.spec.ts'], - testPathIgnorePatterns: ['/e2e/tests/mcp-install\\.spec\\.ts'], + testPathIgnorePatterns: [ + '/e2e/tests/mcp-install\\.spec\\.ts', + '/e2e/tests/self-update\\.spec\\.ts', + ], }, ], }; diff --git a/mkdocs.yml b/mkdocs.yml index 8d71779..cd6d3d3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -43,14 +43,15 @@ markdown_extensions: nav: - Home: index.md - Getting Started: getting-started.md + - MCP: mcp.md - Usage: - Recordings: usage/recordings.md - Search: usage/search.md - Summaries: usage/summaries.md - Templates: usage/templates.md - Configuration: usage/configuration.md + - Updating: usage/updating.md - CLI Reference: usage/cli.md - - MCP: mcp.md - Development: - Architecture: development/architecture.md - Development: development/development.md diff --git a/package.json b/package.json index 78a53db..0694cf1 100644 --- a/package.json +++ b/package.json @@ -1,9 +1,10 @@ { "name": "ailoud-workspace", - "version": "0.0.0", + "version": "1.1.0", "private": true, "type": "module", "description": "Multilingual audio-to-text CLI with a recording library and LLM summaries", + "author": "Lorem Dev ", "license": "Apache-2.0", "packageManager": "pnpm@11.9.0", "engines": { @@ -18,11 +19,14 @@ "lint": "eslint .", "lint:fix": "eslint . --fix", "typecheck": "pnpm -r typecheck", - "test": "vitest run", + "test": "NODE_OPTIONS=--disable-warning=ExperimentalWarning vitest run", "test:watch": "vitest", - "test:cov": "vitest run --coverage", + "test:cov": "NODE_OPTIONS=--disable-warning=ExperimentalWarning vitest run --coverage", "test:e2e": "pnpm build && jest --config jest.config.cjs", - "test:e2e:no-tools": "pnpm build && jest --config jest.config.cjs --selectProjects no-tools" + "test:e2e:no-tools": "pnpm build && jest --config jest.config.cjs --selectProjects no-tools", + "retire": "node scripts/retire-prereleases.mjs", + "docs:build": "uv run --with-requirements docs/requirements.txt mkdocs build --strict", + "docs:serve": "uv run --with-requirements docs/requirements.txt mkdocs serve" }, "devDependencies": { "@eslint/js": "10.0.1", diff --git a/packages/core/README.md b/packages/core/README.md new file mode 100644 index 0000000..cec11c7 --- /dev/null +++ b/packages/core/README.md @@ -0,0 +1,16 @@ +# @ailoud/core + +The domain layer of [AILoud](https://github.com/lorem-dev/ailoud): the model, +the ports, and the pure logic that does no I/O. + +Every effect reaches this package as a port -- `Fs`, `Clock`, `AudioTool`, +`TranscriptionProvider`, `RecordingStore`, `Summarizer` -- implemented in +[`@ailoud/providers`](https://www.npmjs.com/package/@ailoud/providers) and wired +together by the [`ailoud`](https://www.npmjs.com/package/ailoud) CLI. A lint +rule fails the build if anything here imports `node:fs`, `node:child_process` +or a network module. + +Published because the CLI depends on it. The interfaces are not stable yet, and +there is no reason to depend on this package directly. + +Documentation: diff --git a/packages/core/package.json b/packages/core/package.json index bd9f524..7149db7 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -1,6 +1,6 @@ { "name": "@ailoud/core", - "version": "0.0.0", + "version": "1.1.0", "type": "module", "main": "./dist/index.js", "types": "./dist/index.d.ts", @@ -19,6 +19,7 @@ "typecheck": "tsc -p tsconfig.json --noEmit" }, "description": "AILoud domain model, ports and pure logic: the layer that does no I/O", + "author": "Lorem Dev ", "license": "Apache-2.0", "homepage": "https://lorem-dev.github.io/ailoud/", "bugs": { diff --git a/packages/core/src/domain/llmHost.test.ts b/packages/core/src/domain/llmHost.test.ts new file mode 100644 index 0000000..61f8050 --- /dev/null +++ b/packages/core/src/domain/llmHost.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from 'vitest'; +import { isHostedLlm, withoutTrailingSlashes } from './llmHost.js'; + +describe('isHostedLlm', () => { + it('recognises the two hosted APIs', () => { + expect(isHostedLlm('https://api.openai.com/v1')).toBe(true); + expect(isHostedLlm('https://api.anthropic.com')).toBe(true); + }); + + it('is not fooled by a host that merely starts with one', () => { + // The defect this replaced: `startsWith('https://api.openai.com')` and + // `/api\.(openai|anthropic)\.com/` both said yes to these, and the part + // of a hostname that decides where a request goes is the end of it. + expect(isHostedLlm('https://api.openai.com.example.net/v1')).toBe(false); + expect(isHostedLlm('https://api.anthropic.com.example.net')).toBe(false); + }); + + it('is not fooled by the name appearing elsewhere in the URL', () => { + expect(isHostedLlm('https://example.net/?upstream=api.openai.com')).toBe(false); + expect(isHostedLlm('https://example.net/api.openai.com')).toBe(false); + }); + + it('treats a local server as not hosted', () => { + expect(isHostedLlm('http://localhost:11434/v1')).toBe(false); + expect(isHostedLlm('http://127.0.0.1:8080')).toBe(false); + }); + + it('ignores case in the hostname, as DNS does', () => { + expect(isHostedLlm('https://API.OpenAI.com/v1')).toBe(true); + }); + + it('says no to something that is not a URL', () => { + // Reported as a configuration error elsewhere. Calling it hosted here + // would demand an API key for a value that cannot address anything. + expect(isHostedLlm('api.openai.com')).toBe(false); + expect(isHostedLlm('')).toBe(false); + }); +}); + +describe('withoutTrailingSlashes', () => { + it('strips one slash and many', () => { + expect(withoutTrailingSlashes('https://x.test/')).toBe('https://x.test'); + expect(withoutTrailingSlashes('https://x.test/v1///')).toBe('https://x.test/v1'); + }); + + it('leaves a value with none alone', () => { + expect(withoutTrailingSlashes('https://x.test/v1')).toBe('https://x.test/v1'); + }); + + it('handles a value that is only slashes without backtracking', () => { + // The reason this is a loop and not `replace(/\/+$/, '')`. + expect(withoutTrailingSlashes('/'.repeat(50_000))).toBe(''); + }); +}); diff --git a/packages/core/src/domain/llmHost.ts b/packages/core/src/domain/llmHost.ts new file mode 100644 index 0000000..358750e --- /dev/null +++ b/packages/core/src/domain/llmHost.ts @@ -0,0 +1,37 @@ +/** + * Whether a language-model endpoint is one of the hosted APIs. + * + * By hostname, compared exactly. Two places used to ask this with a substring + * -- `baseUrl.startsWith('https://api.openai.com')` and + * `/api\.(openai|anthropic)\.com/.test(baseUrl)` -- and both answered yes for + * `https://api.openai.com.example.net/v1`, where the interesting part of the + * name is what follows. A local server, which is the case these checks exist + * to spare, is any other host. + */ +const HOSTED_HOSTS: readonly string[] = ['api.openai.com', 'api.anthropic.com']; + +export function isHostedLlm(baseUrl: string): boolean { + let parsed; + try { + parsed = new URL(baseUrl); + } catch { + // Not a URL at all. Reported elsewhere as a configuration error; treating + // it as hosted here would demand an API key for a value that cannot + // address anything. + return false; + } + return HOSTED_HOSTS.includes(parsed.hostname.toLowerCase()); +} + +/** + * `baseUrl` without trailing slashes, so a path can be appended to it. + * + * Written as a loop rather than `replace(/\/+$/, '')`: on a value ending in + * many slashes that pattern backtracks, which is a denial of service when the + * value comes from anywhere but us. There is no regex worth that here. + */ +export function withoutTrailingSlashes(baseUrl: string): string { + let end = baseUrl.length; + while (end > 0 && baseUrl[end - 1] === '/') end -= 1; + return baseUrl.slice(0, end); +} diff --git a/packages/core/src/domain/ports.ts b/packages/core/src/domain/ports.ts index 1d0fc68..9ed684f 100644 --- a/packages/core/src/domain/ports.ts +++ b/packages/core/src/domain/ports.ts @@ -7,6 +7,7 @@ import type { Summary, Transcript, } from './model.js'; +import type { PublishedVersion } from './version.js'; export interface Clock { nowIso(): string; @@ -61,6 +62,12 @@ export interface Fs { writeTextFile(path: string, content: string): Promise; /** Reads text. Rejects when the file is not there -- callers check `exists` first. */ readTextFile(path: string): Promise; + /** + * Renames within one filesystem, replacing the target. Atomic, which is why + * it exists: callers write a temporary file beside the real one and rename it + * over the top, so a reader never sees half a file. + */ + rename(from: string, to: string): Promise; } export interface AudioTool { @@ -304,3 +311,20 @@ export interface ManagedRecordingStore extends RecordingStore { /** SQLite's own `PRAGMA integrity_check` result; "ok" means the database is healthy. */ integrityCheck(): string; } + +/** + * What versions of a package exist. Implemented over the npm registry in + * packages/providers; a port because packages/core reaches no network. + */ +export interface VersionSource { + /** + * `signal` exists so ONE implementation can serve both callers. Without it, + * the background update check could not abandon a request promptly and so + * grew a second HTTP client of its own -- which then drifted, missing a + * guard the first one had. Measured on Node 24: aborting a `fetch` leaves + * the process alive for about 10.5 SECONDS, while `https.request`'s native + * `signal` releases it in about 60ms. That measurement is why the provider + * uses `https.request`, and why this parameter is not optional cosmetics. + */ + published(packageName: string, signal?: AbortSignal): Promise; +} diff --git a/packages/core/src/domain/version.test.ts b/packages/core/src/domain/version.test.ts new file mode 100644 index 0000000..edb8327 --- /dev/null +++ b/packages/core/src/domain/version.test.ts @@ -0,0 +1,115 @@ +import { describe, expect, it } from 'vitest'; +import { chooseUpdateTarget, compareVersions, parseVersion } from './version.js'; + +const published = (...versions: string[]) => + versions.map((version) => ({ version, deprecated: false })); + +describe('parseVersion', () => { + it('reads a final release', () => { + expect(parseVersion('1.2.3')).toEqual({ major: 1, minor: 2, patch: 3, pre: null }); + }); + + it('reads a dev snapshot', () => { + expect(parseVersion('1.2.3-dev.4')).toEqual({ + major: 1, + minor: 2, + patch: 3, + pre: { kind: 'dev', n: 4 }, + }); + }); + + it('reads a release candidate', () => { + expect(parseVersion('1.2.3-rc.1')?.pre).toEqual({ kind: 'rc', n: 1 }); + }); + + it.each(['', '1.2', '1.2.3.4', 'v1.2.3', '1.2.3-beta.1', '1.2.3-dev', '1.2.3-dev.x'])( + 'refuses %o', + (text) => { + expect(parseVersion(text)).toBeNull(); + }, + ); +}); + +describe('compareVersions', () => { + it('sorts a release above its own pre-releases', () => { + // Semver precedence, and the one people get wrong: 1.0.0 > 1.0.0-dev.5. + const release = parseVersion('1.0.0')!; + const snapshot = parseVersion('1.0.0-dev.5')!; + expect(compareVersions(release, snapshot)).toBeGreaterThan(0); + }); + + it('sorts pre-releases of one base by their index', () => { + expect( + compareVersions(parseVersion('1.0.0-dev.10')!, parseVersion('1.0.0-dev.9')!), + ).toBeGreaterThan(0); + }); + + it('sorts rc above dev of the same base', () => { + expect( + compareVersions(parseVersion('1.0.0-rc.1')!, parseVersion('1.0.0-dev.9')!), + ).toBeGreaterThan(0); + }); + + it('is zero for equal versions', () => { + expect(compareVersions(parseVersion('2.0.1-rc.3')!, parseVersion('2.0.1-rc.3')!)).toBe(0); + }); +}); + +describe('chooseUpdateTarget', () => { + it('offers the newest final to a final', () => { + expect(chooseUpdateTarget('1.0.0', published('1.0.0', '1.0.1', '1.1.0'))).toBe('1.1.0'); + }); + + it('offers a newer dev of the same base to a dev', () => { + expect(chooseUpdateTarget('1.0.0-dev.3', published('1.0.0-dev.3', '1.0.0-dev.4'))).toBe( + '1.0.0-dev.4', + ); + }); + + it('offers the very next dev, so i >= 1', () => { + expect(chooseUpdateTarget('1.0.0-dev.1', published('1.0.0-dev.2'))).toBe('1.0.0-dev.2'); + }); + + it('prefers the final over a newer snapshot of the same base', () => { + expect(chooseUpdateTarget('1.0.0-dev.3', published('1.0.0-dev.4', '1.0.0'))).toBe('1.0.0'); + }); + + it('refuses to move a final onto a pre-release', () => { + expect(chooseUpdateTarget('1.0.0', published('1.1.0-dev.1', '1.1.0-rc.1'))).toBeNull(); + }); + + it('refuses to cross pre-release kind', () => { + expect(chooseUpdateTarget('1.0.0-dev.3', published('1.0.0-rc.1'))).toBeNull(); + }); + + it('refuses a pre-release of another base', () => { + expect(chooseUpdateTarget('1.0.0-dev.3', published('1.1.0-dev.1'))).toBeNull(); + }); + + it('never offers a deprecated version', () => { + // What `pnpm retire` leaves behind: every superseded snapshot deprecated. + const available = [ + { version: '1.0.0-dev.9', deprecated: true }, + { version: '1.0.0', deprecated: false }, + ]; + expect(chooseUpdateTarget('1.0.0-dev.3', available)).toBe('1.0.0'); + }); + + it('answers null when nothing is newer', () => { + expect(chooseUpdateTarget('1.1.0', published('1.0.0', '1.1.0'))).toBeNull(); + }); + + it('works when the running version was never published', () => { + // A locally built snapshot. Targets come from ordering, not from finding + // the current version in the list. + expect(chooseUpdateTarget('1.1.0-dev.0', published('1.1.0-dev.1'))).toBe('1.1.0-dev.1'); + }); + + it('ignores versions it cannot parse', () => { + expect(chooseUpdateTarget('1.0.0', published('1.0.1', 'not-a-version'))).toBe('1.0.1'); + }); + + it('throws when its own version is unparseable', () => { + expect(() => chooseUpdateTarget('nonsense', published('1.0.0'))).toThrow(/nonsense/); + }); +}); diff --git a/packages/core/src/domain/version.ts b/packages/core/src/domain/version.ts new file mode 100644 index 0000000..ba103cb --- /dev/null +++ b/packages/core/src/domain/version.ts @@ -0,0 +1,98 @@ +/** A version this project produces: `X.Y.Z`, `X.Y.Z-dev.N` or `X.Y.Z-rc.N`. */ +export interface Version { + readonly major: number; + readonly minor: number; + readonly patch: number; + /** null for a final release. */ + readonly pre: { readonly kind: PreKind; readonly n: number } | null; +} + +export type PreKind = 'dev' | 'rc'; + +/** One version as the registry reports it. */ +export interface PublishedVersion { + readonly version: string; + readonly deprecated: boolean; +} + +// Deliberately narrow. The only pre-release kinds this project publishes are +// `dev` and `rc` (see the tag table in AGENTS.md), and a parser that accepted +// `beta` would be inventing a policy for a version nobody can produce. +const PATTERN = /^(\d+)\.(\d+)\.(\d+)(?:-(dev|rc)\.(\d+))?$/; + +export function parseVersion(text: string): Version | null { + const match = PATTERN.exec(text); + if (match === null) return null; + const [, major, minor, patch, kind, n] = match; + return { + major: Number(major), + minor: Number(minor), + patch: Number(patch), + pre: kind === undefined ? null : { kind: kind as PreKind, n: Number(n) }, + }; +} + +/** rc outranks dev, matching semver's dictionary order on the identifier. */ +const KIND_RANK: Record = { dev: 0, rc: 1 }; + +export function compareVersions(a: Version, b: Version): number { + if (a.major !== b.major) return a.major - b.major; + if (a.minor !== b.minor) return a.minor - b.minor; + if (a.patch !== b.patch) return a.patch - b.patch; + // Semver precedence: a pre-release sorts BELOW the release it leads to, so + // 1.0.0 > 1.0.0-dev.5. This is what makes "a snapshot can move to its own + // final release" fall out of the ordering instead of needing a special case. + if (a.pre === null && b.pre === null) return 0; + if (a.pre === null) return 1; + if (b.pre === null) return -1; + if (a.pre.kind !== b.pre.kind) return KIND_RANK[a.pre.kind] - KIND_RANK[b.pre.kind]; + return a.pre.n - b.pre.n; +} + +/** + * The newest version `current` is allowed to move to, or null. + * + * The policy, which exists because a snapshot is not a lesser release but a + * different line of them: + * + * - A final release moves only to a newer final release. Offering a snapshot + * to someone on a release would hand them less tested code than they have. + * - A pre-release moves to a newer pre-release OF THE SAME KIND AND BASE, or + * to any newer final. `1.0.0-dev.3` may take `1.0.0-dev.4` or `1.0.0`, and + * may not take `1.1.0-dev.1` -- that is a different version's line of + * snapshots, and stepping sideways into it skips whatever `1.0.0` became. + * - A deprecated version is never a target. `pnpm retire` deprecates every + * superseded snapshot, so this is what keeps a retired `-dev.9` from being + * offered in place of the `1.0.0` that replaced it. + */ +export function chooseUpdateTarget( + current: string, + available: readonly PublishedVersion[], +): string | null { + const from = parseVersion(current); + if (from === null) { + throw new Error( + `ailoud cannot read its own version ${JSON.stringify(current)}, so it cannot tell what to update to.`, + ); + } + + let best: { text: string; version: Version } | null = null; + for (const candidate of available) { + if (candidate.deprecated) continue; + const to = parseVersion(candidate.version); + if (to === null) continue; + if (compareVersions(to, from) <= 0) continue; + if (!isEligible(from, to)) continue; + if (best === null || compareVersions(to, best.version) > 0) { + best = { text: candidate.version, version: to }; + } + } + return best?.text ?? null; +} + +function isEligible(from: Version, to: Version): boolean { + if (to.pre === null) return true; + if (from.pre === null) return false; + if (from.pre.kind !== to.pre.kind) return false; + return to.major === from.major && to.minor === from.minor && to.patch === from.patch; +} diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 2eb2e66..7068a9c 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -25,6 +25,7 @@ export type { TempDir, TempFile, TranscriptionProvider, + VersionSource, } from './domain/ports.js'; export type { Migration } from './db/schema.js'; @@ -62,9 +63,13 @@ export { export { AiloudError, FailureError, UsageError, EnvironmentError } from './domain/errors.js'; export { encodeUlid } from './domain/ulid.js'; +export { isHostedLlm, withoutTrailingSlashes } from './domain/llmHost.js'; export { mimeForPath } from './domain/mime.js'; +export type { PreKind, PublishedVersion, Version } from './domain/version.js'; +export { chooseUpdateTarget, compareVersions, parseVersion } from './domain/version.js'; + export { MIGRATIONS, SCHEMA_VERSION, pendingMigrations } from './db/schema.js'; export { importRecording, importPath } from './pipelines/import.js'; diff --git a/packages/core/src/testing/fakes.ts b/packages/core/src/testing/fakes.ts index 07e5631..221c034 100644 --- a/packages/core/src/testing/fakes.ts +++ b/packages/core/src/testing/fakes.ts @@ -117,6 +117,13 @@ export class MemFs implements Fs { throw Object.assign(new Error(`ENOENT: ${path}`), { code: 'ENOENT' }); return content; } + async rename(from: string, to: string): Promise { + const content = this.files.get(from); + if (content === undefined) + throw Object.assign(new Error(`ENOENT: ${from}`), { code: 'ENOENT' }); + this.files.set(to, content); + this.files.delete(from); + } } export class FakeAudioTool implements AudioTool { diff --git a/packages/providers/README.md b/packages/providers/README.md new file mode 100644 index 0000000..2cf98c5 --- /dev/null +++ b/packages/providers/README.md @@ -0,0 +1,20 @@ +# @ailoud/providers + +The adapters of [AILoud](https://github.com/lorem-dev/ailoud): each port +declared in [`@ailoud/core`](https://www.npmjs.com/package/@ailoud/core), +implemented against something real. + +| Port | Implemented with | +| ----------------------- | ----------------------------------------------------------- | +| `AudioTool` | ffmpeg and ffprobe | +| `TranscriptionProvider` | whisper.cpp | +| `RecordingStore` | SQLite (`node:sqlite`) with FTS5 search | +| `Summarizer` | llama.cpp, the Claude CLI, or the Anthropic and OpenAI APIs | + +These talk to binaries and files on the machine, so they are covered by the +end-to-end suite rather than by mocks. + +Published because the CLI depends on it. The interfaces are not stable yet, and +there is no reason to depend on this package directly. + +Documentation: diff --git a/packages/providers/package.json b/packages/providers/package.json index 246edc5..29024af 100644 --- a/packages/providers/package.json +++ b/packages/providers/package.json @@ -1,6 +1,6 @@ { "name": "@ailoud/providers", - "version": "0.0.0", + "version": "1.1.0", "type": "module", "main": "./dist/index.js", "types": "./dist/index.d.ts", @@ -18,6 +18,7 @@ "typecheck": "tsc -p tsconfig.json --noEmit" }, "description": "AILoud port implementations: ffmpeg, sqlite, whisper.cpp, and the LLM adapters", + "author": "Lorem Dev ", "license": "Apache-2.0", "homepage": "https://lorem-dev.github.io/ailoud/", "bugs": { diff --git a/packages/providers/src/audio/ffmpeg.test.ts b/packages/providers/src/audio/ffmpeg.test.ts index 0b75a77..65e503c 100644 --- a/packages/providers/src/audio/ffmpeg.test.ts +++ b/packages/providers/src/audio/ffmpeg.test.ts @@ -1,6 +1,7 @@ import { mkdtemp, rm } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; import { afterAll, beforeAll, describe, expect, it } from 'vitest'; import { run } from '../process/run.js'; import { FfmpegAudioTool } from './ffmpeg.js'; @@ -28,6 +29,22 @@ afterAll(async () => { await rm(dir, { recursive: true, force: true }); }); +/** The first audio stream's sample rate and channel count, as ffprobe sees it. */ +async function audioStreamOf(path: string): Promise<{ sample_rate: string; channels: number }> { + const probe = await run('ffprobe', [ + '-v', + 'error', + '-select_streams', + 'a:0', + '-show_entries', + 'stream=sample_rate,channels', + '-of', + 'json', + path, + ]); + return JSON.parse(probe.stdout).streams[0]; +} + describe('FfmpegAudioTool', () => { it('probes the duration', async () => { const { durationMs } = await new FfmpegAudioTool().probe(source); @@ -38,18 +55,7 @@ describe('FfmpegAudioTool', () => { it('converts to 16 kHz mono wav', async () => { const output = join(dir, 'out.wav'); await new FfmpegAudioTool().toWav16kMono(source, output); - const probe = await run('ffprobe', [ - '-v', - 'error', - '-select_streams', - 'a:0', - '-show_entries', - 'stream=sample_rate,channels', - '-of', - 'json', - output, - ]); - const stream = JSON.parse(probe.stdout).streams[0]; + const stream = await audioStreamOf(output); expect(stream.sample_rate).toBe('16000'); expect(stream.channels).toBe(1); }); @@ -77,3 +83,29 @@ describe('FfmpegAudioTool', () => { await expect(new FfmpegAudioTool().slice(bad, join(dir, 'o.wav'), 0, 1000)).rejects.toThrow(); }); }); + +/** + * The video containers domain/mime.ts knows. `ailoud audio import` accepts + * video because a meeting recording usually is one, and only the audio track + * matters -- so each fixture is the same clip of speech wrapped in a different + * container by scripts/make-fixtures.mjs, and the assertion is that what comes + * out is the 16 kHz mono WAV whisper.cpp needs, whatever went in. + */ +describe.each(['mp4', 'mov', 'mkv', 'webm'])('a %s recording', (container) => { + const fixture = fileURLToPath( + new URL(`../../../../fixtures/en-short.${container}`, import.meta.url), + ); + + it('probes like audio and converts to 16 kHz mono wav', async () => { + const tool = new FfmpegAudioTool(); + const { durationMs } = await tool.probe(fixture); + expect(durationMs).toBeGreaterThan(2000); + expect(durationMs).toBeLessThan(3000); + + const output = join(dir, `${container}.wav`); + await tool.toWav16kMono(fixture, output); + const stream = await audioStreamOf(output); + expect(stream.sample_rate).toBe('16000'); + expect(stream.channels).toBe(1); + }); +}); diff --git a/packages/providers/src/index.ts b/packages/providers/src/index.ts index 307c42d..3349f47 100644 --- a/packages/providers/src/index.ts +++ b/packages/providers/src/index.ts @@ -51,3 +51,10 @@ export { listOpenAiModels, listAnthropicModels, isChatModel } from './llm/models export type { ModelOption } from './llm/models.js'; export { LLAMA_VERSION, installLlama, llamaTarballUrl } from './provision/llamaInstall.js'; export type { InstallLlamaOptions, InstallLlamaResult } from './provision/llamaInstall.js'; + +export { DEFAULT_REGISTRY, DEFAULT_TIMEOUT_MS, NpmRegistry } from './update/npmRegistry.js'; +export type { RegistryTransport } from './update/npmRegistry.js'; +export type { NpmRegistryOptions } from './update/npmRegistry.js'; + +export { detectInstallMethod, installCommandFor, sweepCommandFor } from './update/installMethod.js'; +export type { InstallMethod, DetectOptions } from './update/installMethod.js'; diff --git a/packages/providers/src/llm/anthropic.ts b/packages/providers/src/llm/anthropic.ts index c097e19..8474029 100644 --- a/packages/providers/src/llm/anthropic.ts +++ b/packages/providers/src/llm/anthropic.ts @@ -1,5 +1,6 @@ import type { Summarizer } from '@ailoud/core'; import { EnvironmentError, FailureError } from '@ailoud/core'; +import { withoutTrailingSlashes } from '@ailoud/core'; /** As for the OpenAI adapter: a hosted call that has not returned in five minutes is not going to. */ const REQUEST_TIMEOUT_MS = 5 * 60_000; @@ -71,7 +72,7 @@ export class AnthropicSummarizer implements Summarizer { } public async complete(prompt: string): Promise { - const url = `${this.options.baseUrl.replace(/\/+$/, '')}/messages`; + const url = `${withoutTrailingSlashes(this.options.baseUrl)}/messages`; const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS); diff --git a/packages/providers/src/llm/models.ts b/packages/providers/src/llm/models.ts index 9971b12..c96f35d 100644 --- a/packages/providers/src/llm/models.ts +++ b/packages/providers/src/llm/models.ts @@ -1,4 +1,5 @@ import { EnvironmentError, FailureError } from '@ailoud/core'; +import { withoutTrailingSlashes } from '@ailoud/core'; /** As elsewhere in this directory: a hosted call that has not answered in a minute is not going to. */ const REQUEST_TIMEOUT_MS = 60_000; @@ -77,7 +78,7 @@ export async function listOpenAiModels( apiKey: string | undefined, fetchImpl: typeof fetch = fetch, ): Promise { - const url = `${baseUrl.replace(/\/+$/, '')}/models`; + const url = `${withoutTrailingSlashes(baseUrl)}/models`; const body = (await getJson( url, apiKey === undefined ? {} : { authorization: `Bearer ${apiKey}` }, @@ -104,7 +105,7 @@ export async function listAnthropicModels( apiKey: string, fetchImpl: typeof fetch = fetch, ): Promise { - const root = `${baseUrl.replace(/\/+$/, '')}/models`; + const root = `${withoutTrailingSlashes(baseUrl)}/models`; const headers = { 'x-api-key': apiKey, 'anthropic-version': '2023-06-01' }; const collected: ModelOption[] = []; let after: string | undefined; diff --git a/packages/providers/src/llm/openAiCompatible.ts b/packages/providers/src/llm/openAiCompatible.ts index 69c547c..ac91d01 100644 --- a/packages/providers/src/llm/openAiCompatible.ts +++ b/packages/providers/src/llm/openAiCompatible.ts @@ -1,5 +1,6 @@ import type { Summarizer } from '@ailoud/core'; import { EnvironmentError, FailureError } from '@ailoud/core'; +import { withoutTrailingSlashes } from '@ailoud/core'; /** * A hosted model does not get the hour a local one does. If a request has not @@ -61,7 +62,7 @@ export class OpenAiCompatibleSummarizer implements Summarizer { } public async complete(prompt: string): Promise { - const url = `${this.options.baseUrl.replace(/\/+$/, '')}/chat/completions`; + const url = `${withoutTrailingSlashes(this.options.baseUrl)}/chat/completions`; const controller = new AbortController(); const timer = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS); diff --git a/packages/providers/src/store/schema.lock.json b/packages/providers/src/store/schema.lock.json new file mode 100644 index 0000000..7b4dbaa --- /dev/null +++ b/packages/providers/src/store/schema.lock.json @@ -0,0 +1,9 @@ +{ + "1": "990b9debc296c324211775c6d34d80942a4ea049d9d6bc981f74f6810d8c0817", + "2": "3e6171bd33a660c4b7ccb686d4204b7d4533cbed2724d6714054023d89e4f4bc", + "3": "bbb212932dd1c81d6c81606e87097af6e2c91fd650f173b71288067d878d8de1", + "4": "04f859176f9c0dafc4051220f404ef93e29397427a9063da02eaa24160b8a5c4", + "5": "abe7451f10eea7efb50ae73f2817970e2a1afff58dc6676a970a9d78f7eda867", + "6": "e6bd7848b47e358a03ebc4f3153d4cdd6a827d99a44952c5420883fa5c9df90c", + "7": "65c9b56640066a89ec6f6c35a00c8aaf693cba6c838f031496a7a489d3a04e50" +} diff --git a/packages/providers/src/store/schema.snapshot.sql b/packages/providers/src/store/schema.snapshot.sql new file mode 100644 index 0000000..86e4faf --- /dev/null +++ b/packages/providers/src/store/schema.snapshot.sql @@ -0,0 +1,86 @@ +CREATE INDEX summary_by_recording ON summary_recording(recording_id); + +CREATE INDEX tag_by_name ON tag(tag); + +CREATE INDEX transcript_by_recording ON transcript(recording_id, created_at DESC); + +CREATE TABLE recording ( + id TEXT PRIMARY KEY, + sha256 TEXT NOT NULL UNIQUE, + source_path TEXT NOT NULL, + media_path TEXT NOT NULL, + duration_ms INTEGER NOT NULL, + mime TEXT NOT NULL, + title TEXT, + notes TEXT, + imported_at TEXT NOT NULL + , recorded_at TEXT); + +CREATE TABLE segment ( + id TEXT PRIMARY KEY, + transcript_id TEXT NOT NULL REFERENCES transcript(id) ON DELETE CASCADE, + idx INTEGER NOT NULL, + start_ms INTEGER NOT NULL, + end_ms INTEGER NOT NULL, + text TEXT NOT NULL, + speaker TEXT, + language TEXT, + UNIQUE (transcript_id, idx) + ); + +CREATE VIRTUAL TABLE segment_fts USING fts5( + text, content='segment', content_rowid='rowid' + ); + +CREATE TABLE speaker ( + recording_id TEXT NOT NULL REFERENCES recording(id) ON DELETE CASCADE, + label TEXT NOT NULL, + name TEXT NOT NULL, + PRIMARY KEY (recording_id, label) + ); + +CREATE TABLE summary ( + id TEXT PRIMARY KEY, + created_at TEXT NOT NULL, + language TEXT NOT NULL, + provider TEXT NOT NULL, + model TEXT NOT NULL, + body TEXT NOT NULL + , template TEXT NOT NULL DEFAULT 'meeting', context TEXT NOT NULL DEFAULT ''); + +CREATE TABLE summary_recording ( + summary_id TEXT NOT NULL REFERENCES summary(id) ON DELETE CASCADE, + recording_id TEXT NOT NULL REFERENCES recording(id) ON DELETE CASCADE, + PRIMARY KEY (summary_id, recording_id) + ); + +CREATE TABLE tag ( + recording_id TEXT NOT NULL REFERENCES recording(id) ON DELETE CASCADE, + tag TEXT NOT NULL, + PRIMARY KEY (recording_id, tag) + ); + +CREATE TABLE transcript ( + id TEXT PRIMARY KEY, + recording_id TEXT NOT NULL REFERENCES recording(id) ON DELETE CASCADE, + provider TEXT NOT NULL, + model TEXT NOT NULL, + language TEXT NOT NULL, + text TEXT NOT NULL, + created_at TEXT NOT NULL + ); + +CREATE TRIGGER segment_ad AFTER DELETE ON segment BEGIN + INSERT INTO segment_fts(segment_fts, rowid, text) + VALUES ('delete', old.rowid, old.text); + END; + +CREATE TRIGGER segment_ai AFTER INSERT ON segment BEGIN + INSERT INTO segment_fts(rowid, text) VALUES (new.rowid, new.text); + END; + +CREATE TRIGGER segment_fts_update AFTER UPDATE ON segment BEGIN + INSERT INTO segment_fts(segment_fts, rowid, text) + VALUES ('delete', old.rowid, old.text); + INSERT INTO segment_fts(rowid, text) VALUES (new.rowid, new.text); + END; diff --git a/packages/providers/src/store/schemaGuard.test.ts b/packages/providers/src/store/schemaGuard.test.ts new file mode 100644 index 0000000..5951dec --- /dev/null +++ b/packages/providers/src/store/schemaGuard.test.ts @@ -0,0 +1,204 @@ +import { createHash } from 'node:crypto'; +import { readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { DatabaseSync } from 'node:sqlite'; +import { fileURLToPath } from 'node:url'; +import { describe, expect, it } from 'vitest'; +import type { Migration } from '@ailoud/core'; +import { MIGRATIONS } from '@ailoud/core'; + +/** + * Two guards against the same failure mode: a schema change that ships + * without going through the migration list, or a migration edited after it + * has already run on someone's machine. + * + * A database that already ran migration 4 will never run it again -- SQLite + * only replays the migrations above its stored `user_version`. So an edited + * migration 4 does not fail loudly; it means two machines sitting at the + * same `user_version` with DIFFERENT schemas, which nothing here or in + * SqliteStore would ever notice on its own. The lock file is the only thing + * that would. + * + * schema.lock.json and schema.snapshot.sql are generated by + * scripts/write-schema-snapshot.mjs (run it after adding a migration, never + * to silence a failure here about an already-shipped one -- see + * .agents/skills/check-migrations/SKILL.md). + */ + +const here = dirname(fileURLToPath(import.meta.url)); +const LOCK = join(here, 'schema.lock.json'); +const SNAPSHOT = join(here, 'schema.snapshot.sql'); + +const lock = JSON.parse(readFileSync(LOCK, 'utf8')) as Record; + +/** + * Hashes a migration's statements only, never the comments around them in + * schema.ts. That is not a parsing choice -- the comments are TypeScript + * comments outside the template-literal strings, so they were never part of + * `migration.statements` at runtime, and editing one cannot change this + * value. The comments explaining why a column or table exists are the house + * style and must stay freely editable without touching the lock. + * + * A record-separator character (0x1E), not ';', goes between statements. + * A semicolon can appear INSIDE a statement -- every trigger body in + * schema.ts (segment_ai, segment_ad, segment_fts_update) contains several, + * between its BEGIN...END sub-statements -- so joining with ';' lets two + * different splits of the same bytes collide on the same hash (e.g. + * ["a;b", "c"] and ["a", "b;c"] both concatenate to "a;b;c;"). 0x1E cannot + * appear in any SQL statement written in schema.ts, so this join has no such + * collision. Keep this in exact step with the same-named function in + * scripts/write-schema-snapshot.mjs -- both must compute the same thing + * from the same MIGRATIONS, or a passing test here proves nothing. + */ +const FINGERPRINT_SEPARATOR = '\x1e'; + +function fingerprint(migration: Migration): string { + const hash = createHash('sha256'); + for (const statement of migration.statements) { + hash.update(statement); + hash.update(FINGERPRINT_SEPARATOR); + } + return hash.digest('hex'); +} + +/** + * True only if `versions` is exactly [1, 2, ..., versions.length], in that + * order -- no gap, no duplicate, and no entry whose value disagrees with its + * position. SCHEMA_VERSION (packages/core/src/db/schema.ts) is defined as + * MIGRATIONS.length, which silently assumes exactly this. A gap -- say + * versions [1, 2, 4], version 3 skipped by a typo when adding a new one -- + * lets SqliteStore.migrate write PRAGMA user_version = 4 while + * SCHEMA_VERSION stays 3 (MIGRATIONS.length is still only 3 entries), so + * pendingMigrations then refuses to reopen the very database that build just + * created, as "newer than this build understands". This is asserted here, + * as a loud development-time failure, rather than by redefining + * SCHEMA_VERSION: that constant is what `user_version` MEANS in every + * database that already exists, and changing its definition would be a + * migration hazard of its own. + */ +function isContiguousFromOne(versions: readonly number[]): boolean { + return versions.every((version, index) => version === index + 1); +} + +/** + * A fresh in-memory database with every migration applied, in order. Closes + * `db` before rethrowing if any statement fails partway through: otherwise a + * bad migration would leak the handle before it even reaches the caller's + * own try/finally, since the caller never receives a reference to close. + */ +function migratedDatabase(): DatabaseSync { + const db = new DatabaseSync(':memory:'); + try { + for (const migration of MIGRATIONS) { + for (const statement of migration.statements) db.exec(statement); + } + } catch (error) { + db.close(); + throw error; + } + return db; +} + +/** + * A deterministic text dump of the schema a database ends up with: every + * table, index, and trigger that carries its own SQL, sorted by type then + * name so the output does not depend on creation order. `sql IS NOT NULL` + * excludes the automatic indexes SQLite creates for a composite PRIMARY KEY, + * which store no SQL of their own and are implied by the table that owns + * them. + * + * Also excludes FTS5's shadow tables (segment_fts_config, _data, _docsize, + * _idx: `type = 'table' AND name LIKE 'segment\_fts\_%'`). Those are not + * written by any statement in schema.ts -- they are generated internally by + * the FTS5 extension bundled with whichever SQLite build node:sqlite links, + * which is still experimental in Node 24 with CI pinning only the major + * version. Their exact DDL text (column names, WITHOUT ROWID, quoting + * style) can change on a Node/SQLite upgrade with zero change to any + * migration, which would fail this guard for a reason unrelated to its + * subject -- and a guard that fails for reasons unrelated to its subject + * teaches people to ignore it. `CREATE VIRTUAL TABLE segment_fts` itself is + * NOT excluded (it doesn't match the shadow-table name pattern): the + * declaration stays guarded, only its generated implementation detail does + * not. + * + * This is generated BY the migrations, never hand-edited, so it cannot drift + * from them by construction -- only regeneration can change it, and + * regeneration is scripts/write-schema-snapshot.mjs, which a reviewer sees + * as a diff. + */ +function dumpSchema(db: DatabaseSync): string { + const rows = db + .prepare( + `SELECT sql FROM sqlite_master + WHERE sql IS NOT NULL + AND NOT (type = 'table' AND name LIKE 'segment\\_fts\\_%' ESCAPE '\\') + ORDER BY type, name`, + ) + .all() as unknown as { sql: string }[]; + return rows.map((row) => `${row.sql};`).join('\n\n') + '\n'; +} + +describe('schema guard', () => { + it('has a lock entry for every migration, and no extras', () => { + expect( + Object.keys(lock) + .map(Number) + .sort((a, b) => a - b), + ).toEqual(MIGRATIONS.map((m) => m.version)); + }); + + it('still matches the fingerprint of every shipped migration', () => { + // A database that already ran version 4 will never run it again, so an + // edited version 4 means two machines at the same user_version with + // different schemas. Adding a migration means adding a lock line, which + // is a visible diff in review. + for (const migration of MIGRATIONS) { + expect(fingerprint(migration)).toBe(lock[String(migration.version)]); + } + }); + + it('produces exactly the snapshotted schema', () => { + // Generated BY the migrations, so it cannot drift from them. Any + // structural change fails here until the snapshot is regenerated, which + // puts every schema change in front of a reviewer. + const db = migratedDatabase(); + try { + expect(dumpSchema(db)).toBe(readFileSync(SNAPSHOT, 'utf8')); + } finally { + db.close(); + } + }); + + it('has version numbers contiguous from 1, with no gap and no duplicate', () => { + expect(isContiguousFromOne(MIGRATIONS.map((m) => m.version))).toBe(true); + }); + + it('rejects a gap in version numbers', () => { + expect(isContiguousFromOne([1, 2, 4])).toBe(false); + }); + + it('rejects a duplicate version number', () => { + expect(isContiguousFromOne([1, 2, 2])).toBe(false); + }); + + it('rejects a version that disagrees with its position in the array', () => { + expect(isContiguousFromOne([1, 3, 2])).toBe(false); + }); + + it('does not let statement content collide across the separator', () => { + // Every trigger body in schema.ts contains internal semicolons, so if + // the separator between statements were ';', two differently-split + // statement lists could concatenate to the identical byte stream and + // hash the same. This is not hypothetical: ['a;b', 'c'] and + // ['a', 'b;c'] both produced 'a;b;c;' under the old ';' separator. + const a: Migration = { version: 1, statements: ['a;b', 'c'] }; + const b: Migration = { version: 1, statements: ['a', 'b;c'] }; + expect(fingerprint(a)).not.toBe(fingerprint(b)); + }); + + it('excludes FTS5 shadow tables from the snapshot but keeps segment_fts itself', () => { + const snapshot = readFileSync(SNAPSHOT, 'utf8'); + expect(snapshot).toContain('CREATE VIRTUAL TABLE segment_fts'); + expect(snapshot).not.toMatch(/segment_fts_(config|data|docsize|idx)/); + }); +}); diff --git a/packages/providers/src/system/nodeFs.test.ts b/packages/providers/src/system/nodeFs.test.ts index 164758b..911d6d5 100644 --- a/packages/providers/src/system/nodeFs.test.ts +++ b/packages/providers/src/system/nodeFs.test.ts @@ -64,3 +64,33 @@ describe('NodeFs.exists / isDirectory', () => { } }); }); + +describe('NodeFs.rename', () => { + it('replaces the target and removes the source, atomically', async () => { + const fs = new NodeFs(); + const dir = mkdtempSync(join(tmpdir(), 'ailoud-test-')); + try { + const from = join(dir, 'source.tmp'); + const to = join(dir, 'target.json'); + writeFileSync(from, 'new content'); + writeFileSync(to, 'old content'); + + await fs.rename(from, to); + + expect(existsSync(from)).toBe(false); + expect(await fs.readTextFile(to)).toBe('new content'); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); + + it('rejects when the source does not exist', async () => { + const fs = new NodeFs(); + const dir = mkdtempSync(join(tmpdir(), 'ailoud-test-')); + try { + await expect(fs.rename(join(dir, 'missing'), join(dir, 'target'))).rejects.toBeTruthy(); + } finally { + await rm(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/packages/providers/src/system/nodeFs.ts b/packages/providers/src/system/nodeFs.ts index 5b379f2..7174dfc 100644 --- a/packages/providers/src/system/nodeFs.ts +++ b/packages/providers/src/system/nodeFs.ts @@ -1,6 +1,16 @@ import { createHash } from 'node:crypto'; import { createReadStream } from 'node:fs'; -import { copyFile, mkdir, mkdtemp, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises'; +import { + copyFile, + mkdir, + mkdtemp, + readFile, + readdir, + rename, + rm, + stat, + writeFile, +} from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; import type { Fs, TempDir, TempFile } from '@ailoud/core'; @@ -76,4 +86,7 @@ export class NodeFs implements Fs { async readTextFile(path: string): Promise { return readFile(path, 'utf8'); } + async rename(from: string, to: string): Promise { + await rename(from, to); + } } diff --git a/packages/providers/src/update/installMethod.test.ts b/packages/providers/src/update/installMethod.test.ts new file mode 100644 index 0000000..751c9a5 --- /dev/null +++ b/packages/providers/src/update/installMethod.test.ts @@ -0,0 +1,240 @@ +import { describe, expect, it } from 'vitest'; +import { detectInstallMethod, installCommandFor, sweepCommandFor } from './installMethod.js'; +import type { DetectOptions } from './installMethod.js'; + +function fakeRoots(roots: { npm?: string; pnpm?: string }): DetectOptions['run'] { + return async (command, _args) => { + const root = command === 'npm' ? roots.npm : command === 'pnpm' ? roots.pnpm : undefined; + if (root === undefined) return { code: 1, stdout: '', stderr: `${command}: command not found` }; + return { code: 0, stdout: root, stderr: '' }; + }; +} + +describe('detectInstallMethod', () => { + it('detects a global pnpm install through a symlinked root', async () => { + // pnpm's global bin is a symlink farm, so an unresolved string compare + // misses every time. This test fails on any implementation that skips + // realpath. + const method = await detectInstallMethod({ + packageRoot: '/Users/x/.local/share/pnpm/global/5/node_modules/ailoud', + execPath: '/usr/local/bin/node', + realpath: async (p) => p.replace('/link/', '/real/'), + run: fakeRoots({ + npm: '/opt/homebrew/lib/node_modules', + pnpm: '/Users/x/.local/share/pnpm/global/5/node_modules', + }), + }); + expect(method.kind).toBe('pnpm-global'); + }); + + it('refuses an npx cache entry', async () => { + // An npx cache entry also sits inside a node_modules directory, so this + // packageRoot would be mistaken for a project dependency if the + // /_npx/ check were not evaluated first. + const method = await detectInstallMethod({ + packageRoot: '/Users/x/.npm/_npx/abcd1234/node_modules/ailoud', + execPath: '/usr/local/bin/node', + realpath: async (p) => p, + run: fakeRoots({ + npm: '/opt/homebrew/lib/node_modules', + pnpm: '/Users/x/.local/share/pnpm/global/5/node_modules', + }), + }); + expect(method).toEqual({ kind: 'npx', hint: 'npx ailoud@' }); + }); + + it('refuses a project dependency and names the project', async () => { + const method = await detectInstallMethod({ + packageRoot: '/Users/x/code/some-app/node_modules/ailoud', + execPath: '/usr/local/bin/node', + realpath: async (p) => p, + run: fakeRoots({ + npm: '/opt/homebrew/lib/node_modules', + pnpm: '/Users/x/.local/share/pnpm/global/5/node_modules', + }), + }); + expect(method).toEqual({ + kind: 'project', + projectDir: '/Users/x/code/some-app', + hint: "run your package manager's add command in /Users/x/code/some-app", + }); + }); + + it('is unknown when neither manager claims the root', async () => { + const method = await detectInstallMethod({ + packageRoot: '/opt/custom/ailoud', + execPath: '/usr/local/bin/node', + realpath: async (p) => p, + run: fakeRoots({ + npm: '/opt/homebrew/lib/node_modules', + pnpm: '/Users/x/.local/share/pnpm/global/5/node_modules', + }), + }); + expect(method).toEqual({ + kind: 'unknown', + hint: 'npm install -g ailoud@, or pnpm add -g ailoud@', + }); + }); + + it('survives a package manager that is not installed', async () => { + // pnpm is not on this machine: its "root -g" rejects instead of + // answering. Detection must still resolve via npm rather than throwing. + const method = await detectInstallMethod({ + packageRoot: '/opt/homebrew/lib/node_modules/ailoud', + execPath: '/usr/local/bin/node', + realpath: async (p) => p, + run: async (command) => { + if (command === 'pnpm') throw new Error('spawn pnpm ENOENT'); + return { code: 0, stdout: '/opt/homebrew/lib/node_modules', stderr: '' }; + }, + }); + expect(method.kind).toBe('npm-global'); + }); +}); + +describe('installCommandFor', () => { + it('anchors the npm-global install beside the running node, not to a bare "npm"', () => { + // A bare 'npm' resolves off PATH, which can belong to an entirely + // different Node than the one running us (nvm/fnm/asdf/volta) -- see + // this module's own doc comment on DetectOptions.execPath for the + // machine layout that breaks under a bare name. + const command = installCommandFor( + { kind: 'npm-global' }, + '1.0.1', + '/home/x/.nvm/versions/node/v20.11.0/bin/node', + ); + expect(command).toEqual([ + '/home/x/.nvm/versions/node/v20.11.0/bin/npm', + 'install', + '-g', + 'ailoud@1.0.1', + ]); + }); + + it('keeps the pnpm-global install as a bare "pnpm", deliberately', () => { + // pnpm's global bin comes from PNPM_HOME/corepack, not from any one + // Node's install tree, so there is no execPath-equivalent anchor for it + // -- bare 'pnpm' IS the correct resolution. Asserted explicitly so + // nobody "fixes" this into a broken anchor later. + const command = installCommandFor( + { kind: 'pnpm-global' }, + '1.0.1', + '/home/x/.nvm/versions/node/v20.11.0/bin/node', + ); + expect(command).toEqual(['pnpm', 'add', '-g', 'ailoud@1.0.1']); + }); + + it('refuses to build a command for a method that cannot be updated', () => { + expect( + installCommandFor({ kind: 'npx', hint: 'npx ailoud@1.0.1' }, '1.0.1', '/usr/local/bin/node'), + ).toBeNull(); + }); + + it('names the project, not the pnpm store entry, for a pnpm-installed dependency', async () => { + // pnpm puts a project dependency at + // /node_modules/.pnpm/@/node_modules/. + // Taking the LAST `/node_modules/` named the store entry, so the hint told + // the user to run their add command in `.../.pnpm/ailoud@1.0.0`. + const method = await detectInstallMethod({ + packageRoot: '/Users/x/repo/node_modules/.pnpm/ailoud@1.0.0/node_modules/ailoud', + execPath: '/usr/local/bin/node', + realpath: async (path: string) => path, + run: async () => ({ code: 0, stdout: '/nowhere\n', stderr: '' }), + }); + expect(method).toEqual({ + kind: 'project', + projectDir: '/Users/x/repo', + hint: "run your package manager's add command in /Users/x/repo", + }); + }); + + it('detects a global install under a version manager, where PATH npm is a different Node', async () => { + // nvm, fnm, asdf and volta all do this: the npm on PATH belongs to another + // Node version, so `npm root -g` reports THAT version's root, ours never + // matches, and an ordinary global install used to read as a project + // dependency -- telling the user to run an add command inside + // `~/.nvm/versions/node/v18.20.4/lib`. + const method = await detectInstallMethod({ + execPath: '/home/x/.nvm/versions/node/v18.20.4/bin/node', + packageRoot: '/home/x/.nvm/versions/node/v18.20.4/lib/node_modules/ailoud', + realpath: async (path: string) => path, + run: async () => ({ + code: 0, + stdout: '/home/x/.nvm/versions/node/v20.11.0/lib/node_modules\n', + stderr: '', + }), + }); + expect(method).toEqual({ kind: 'npm-global' }); + }); + + it('does not mistake a sibling directory for the global root', async () => { + // `startsWith` compares characters, not path components, so + // `/usr/lib/node_modules-other` used to read as living under + // `/usr/lib/node_modules`. + const method = await detectInstallMethod({ + execPath: '/usr/bin/node', + packageRoot: '/usr/lib/node_modules-other/ailoud', + realpath: async (path: string) => path, + run: async () => ({ code: 0, stdout: '/usr/lib/node_modules\n', stderr: '' }), + }); + expect(method.kind).not.toBe('npm-global'); + }); +}); + +describe('sweepCommandFor', () => { + it('anchors the npm-global sweep beside the running node, not to a bare "ailoud"', async () => { + // Same reasoning as the install: a bare 'ailoud' resolves off PATH, which + // can be a completely different install than the one the package manager + // just wrote -- the exact staleness bug the subprocess sweep exists to + // prevent. + const command = await sweepCommandFor( + { kind: 'npm-global' }, + '/home/x/.nvm/versions/node/v20.11.0/bin/node', + async () => ({ code: 1, stdout: '', stderr: 'unused for npm-global' }), + ); + expect(command).toEqual(['/home/x/.nvm/versions/node/v20.11.0/bin/ailoud', 'self', 'sync']); + }); + + it('anchors the pnpm-global sweep to the path "pnpm bin -g" reports', async () => { + const seen: Array<{ command: string; args: readonly string[] }> = []; + const command = await sweepCommandFor( + { kind: 'pnpm-global' }, + '/usr/local/bin/node', + async (cmd, args) => { + seen.push({ command: cmd, args }); + return { code: 0, stdout: '/home/x/.local/share/pnpm\n', stderr: '' }; + }, + ); + expect(seen).toEqual([{ command: 'pnpm', args: ['bin', '-g'] }]); + expect(command).toEqual(['/home/x/.local/share/pnpm/ailoud', 'self', 'sync']); + }); + + it('answers null, rather than a guess, when "pnpm bin -g" fails', async () => { + const command = await sweepCommandFor( + { kind: 'pnpm-global' }, + '/usr/local/bin/node', + async () => ({ code: 1, stdout: '', stderr: 'pnpm: command not found' }), + ); + expect(command).toBeNull(); + }); + + it('answers null when "pnpm bin -g" throws rather than exiting non-zero', async () => { + const command = await sweepCommandFor( + { kind: 'pnpm-global' }, + '/usr/local/bin/node', + async () => { + throw new Error('spawn pnpm ENOENT'); + }, + ); + expect(command).toBeNull(); + }); + + it('answers null for a method that cannot be updated', async () => { + const command = await sweepCommandFor( + { kind: 'unknown', hint: 'npm install -g ailoud@, or pnpm add -g ailoud@' }, + '/usr/local/bin/node', + async () => ({ code: 0, stdout: '/nowhere', stderr: '' }), + ); + expect(command).toBeNull(); + }); +}); diff --git a/packages/providers/src/update/installMethod.ts b/packages/providers/src/update/installMethod.ts new file mode 100644 index 0000000..651cada --- /dev/null +++ b/packages/providers/src/update/installMethod.ts @@ -0,0 +1,195 @@ +import { dirname, join } from 'node:path'; +import type { RunResult } from '../process/run.js'; + +/** + * `hint` is the command to run by hand, and every refusing variant carries + * one: a refusal that does not say what to do instead is just a failure. + */ +export type InstallMethod = + | { readonly kind: 'npm-global' } + | { readonly kind: 'pnpm-global' } + | { readonly kind: 'npx'; readonly hint: string } + | { readonly kind: 'project'; readonly projectDir: string; readonly hint: string } + | { readonly kind: 'unknown'; readonly hint: string }; + +export interface DetectOptions { + /** + * The Node binary running this process, i.e. `process.execPath`. + * + * Used to locate the global `node_modules` of the Node that is RUNNING us, + * which is the one ailoud was installed into. `npm root -g` cannot answer + * that on its own: under nvm, fnm, asdf and volta the npm on PATH often + * belongs to a different Node version, so it reports that version's root, + * our own root never matches it, and an ordinary global install is then + * misread as a project dependency -- with a hint telling the user to run an + * add command inside `~/.nvm/versions/node/v18.20.4/lib`. + */ + readonly execPath: string; + /** Where the installed package sits, resolved from `import.meta.url`. */ + readonly packageRoot: string; + readonly realpath: (path: string) => Promise; + readonly run: (command: string, args: readonly string[]) => Promise; +} + +export async function detectInstallMethod(options: DetectOptions): Promise { + const root = await options.realpath(options.packageRoot); + + // FIRST. An npx cache entry also sits inside a `node_modules`, so a + // node_modules check placed ahead of this one reports it as a project + // dependency and prints the wrong command. + if (root.includes('/_npx/')) return { kind: 'npx', hint: 'npx ailoud@' }; + + // Both managers, in parallel, each tolerating "not installed". `npm root -g` + // and `pnpm root -g` print the global node_modules. + const [npmRoot, pnpmRoot] = await Promise.all([ + globalRoot(options, 'npm'), + globalRoot(options, 'pnpm'), + ]); + if (npmRoot !== null && isUnder(root, npmRoot)) return { kind: 'npm-global' }; + if (pnpmRoot !== null && isUnder(root, pnpmRoot)) return { kind: 'pnpm-global' }; + + // The global root of the Node running us, checked after the managers and + + // before the project fallback: it is the layout npm itself uses + + // (`/lib/node_modules`), and it is version-correct by + + // construction because it comes from our own interpreter. + + const ownRoot = await options + .realpath(`${dirname(dirname(options.execPath))}/lib/node_modules`) + .catch(() => null); + + if (ownRoot !== null && isUnder(root, ownRoot)) return { kind: 'npm-global' }; + + // Whatever is left inside a node_modules is somebody's dependency. The + // FIRST `/node_modules/` is the project boundary, not the last: pnpm installs + // a project dependency at + // `/node_modules/.pnpm/ailoud@1.0.0/node_modules/ailoud`, so taking + // the last one named the store entry and told the user to run their add + // command in `.../node_modules/.pnpm/ailoud@1.0.0`. A nested dependency of a + // dependency lands on the project for the same reason, which is also right. + const marker = root.indexOf('/node_modules/'); + if (marker !== -1) { + const projectDir = root.slice(0, marker); + return { + kind: 'project', + projectDir, + hint: `run your package manager's add command in ${projectDir}`, + }; + } + return { + kind: 'unknown', + hint: 'npm install -g ailoud@, or pnpm add -g ailoud@', + }; +} + +async function globalRoot(options: DetectOptions, manager: string): Promise { + try { + const result = await options.run(manager, ['root', '-g']); + if (result.code !== 0) return null; + // realpath both sides: pnpm's global tree is a symlink farm, and an + // unresolved string compare misses every time. + return await options.realpath(result.stdout.trim()); + } catch { + return null; // that manager is not on this machine + } +} + +/** + * The argv for an update, or null when this install method cannot be updated. + * + * `npm-global` is anchored to `join(dirname(execPath), 'npm')` rather than + * bare `'npm'`: every npm global install lays its bins out beside `node` + * itself, so this is the one npm that is GUARANTEED to belong to the Node + * that is running us, unlike a bare name, which PATH resolves and which can + * belong to an entirely different Node under nvm/fnm/asdf/volta (see + * `DetectOptions.execPath`'s own doc comment, and `self.ts`'s doc comment on + * `updateSelf`, for the concrete machine layout that breaks under a bare + * name). + * + * `pnpm-global` deliberately stays bare `'pnpm'`: pnpm's global bin comes + * from `PNPM_HOME`/corepack, not from any one Node's install tree, so there + * is no execPath-equivalent anchor for it, and bare `pnpm` IS the correct + * resolution here. Do not "fix" this into an anchored path -- there is + * nothing to anchor it to. + */ +export function installCommandFor( + method: InstallMethod, + target: string, + execPath: string, +): readonly string[] | null { + switch (method.kind) { + case 'npm-global': + return [join(dirname(execPath), 'npm'), 'install', '-g', `ailoud@${target}`]; + case 'pnpm-global': + return ['pnpm', 'add', '-g', `ailoud@${target}`]; + case 'npx': + case 'project': + case 'unknown': + return null; + } +} + +/** + * The argv for the subprocess that re-syncs rules after a successful + * install, or null when it cannot be determined -- the caller then prints + * the command for the user to run by hand rather than guessing. + * + * Anchored the same way `installCommandFor` anchors the install itself, and + * for the same reason: a bare `ailoud` resolved off PATH can be an entirely + * different install than the one the package manager just wrote. + * + * `npm-global`: the new binary sits beside `npm` and `node` in the same bin + * directory by construction, so `dirname(execPath)` anchors it exactly like + * the install command above. + * + * `pnpm-global`: there is no execPath-equivalent anchor, so this asks pnpm + * itself where its global bin lives (`pnpm bin -g`), through the same `run` + * detection uses -- bounded to the same 10 second timeout in production + * (`boundedDetectRun` in `self.ts`). Null when that query fails or answers + * nothing, rather than guessing at a bare `ailoud`. + */ +export async function sweepCommandFor( + method: InstallMethod, + execPath: string, + run: (command: string, args: readonly string[]) => Promise, +): Promise { + switch (method.kind) { + case 'npm-global': + return [join(dirname(execPath), 'ailoud'), 'self', 'sync']; + case 'pnpm-global': { + const bin = await pnpmGlobalBin(run); + return bin === null ? null : [join(bin, 'ailoud'), 'self', 'sync']; + } + case 'npx': + case 'project': + case 'unknown': + return null; + } +} + +async function pnpmGlobalBin( + run: (command: string, args: readonly string[]) => Promise, +): Promise { + try { + const result = await run('pnpm', ['bin', '-g']); + if (result.code !== 0) return null; + const bin = result.stdout.trim(); + return bin === '' ? null : bin; + } catch { + return null; + } +} + +/** + * Whether `path` sits inside `directory`. + * + * The separator is required, so `/usr/lib/node_modules-other/ailoud` does not + * read as living under `/usr/lib/node_modules`. A bare `startsWith` compares + * characters, not path components. + */ +function isUnder(path: string, directory: string): boolean { + const base = directory.endsWith('/') ? directory : `${directory}/`; + return path.startsWith(base); +} diff --git a/packages/providers/src/update/npmRegistry.test.ts b/packages/providers/src/update/npmRegistry.test.ts new file mode 100644 index 0000000..dba18f0 --- /dev/null +++ b/packages/providers/src/update/npmRegistry.test.ts @@ -0,0 +1,107 @@ +import { describe, expect, it, vi } from 'vitest'; +import { NpmRegistry, isDeprecated } from './npmRegistry.js'; +import type { RegistryTransport } from './npmRegistry.js'; + +/** A transport that answers with a fixed status and body, recording its calls. */ +const answering = (status: number, body: string) => + vi.fn(async () => ({ status, body })); + +const packument = { + name: 'ailoud', + versions: { + '1.0.0-dev.1': { name: 'ailoud', version: '1.0.0-dev.1', deprecated: 'superseded by 1.0.0' }, + '1.0.0': { name: 'ailoud', version: '1.0.0' }, + }, +}; + +describe('NpmRegistry', () => { + it('reports every version, marking the deprecated ones', async () => { + const transport = answering(200, JSON.stringify(packument)); + const registry = new NpmRegistry({ transport }); + expect(await registry.published('ailoud')).toEqual([ + { version: '1.0.0-dev.1', deprecated: true }, + { version: '1.0.0', deprecated: false }, + ]); + }); + + it('asks for the abbreviated packument', async () => { + const transport = answering(200, JSON.stringify(packument)); + await new NpmRegistry({ transport }).published('ailoud'); + const [, headers] = transport.mock.calls[0]!; + expect(headers).toMatchObject({ accept: 'application/vnd.npm.install-v1+json' }); + }); + + it('escapes a scoped name', async () => { + const transport = answering(200, JSON.stringify(packument)); + await new NpmRegistry({ transport }).published('@ailoud/core'); + expect(transport.mock.calls[0]![0]).toBe('https://registry.npmjs.org/@ailoud%2fcore'); + }); + + it('throws with the status when the registry refuses', async () => { + const transport = answering(503, 'nope'); + await expect(new NpmRegistry({ transport }).published('ailoud')).rejects.toThrow(/503/); + }); + + it('throws when the versions object is present but empty', async () => { + // A 200 with `{"versions": {}}` used to resolve to an empty list, which + // every caller reads as "nothing newer exists". A package with no versions + // cannot be the one we are running. + const transport = answering(200, '{"versions":{}}'); + await expect(new NpmRegistry({ transport }).published('ailoud')).rejects.toThrow(/no versions/); + }); + + it('treats an empty deprecation message as not deprecated', async () => { + // `npm deprecate @ ""` un-deprecates by setting an empty + // string, not by removing the field. Testing for the key rather than the + // value reported a revived version as still deprecated, which would refuse + // a legitimate update. + const body = JSON.stringify({ + versions: { + '1.0.0': { version: '1.0.0', deprecated: '' }, + '1.0.1': { version: '1.0.1', deprecated: 'do not use' }, + }, + }); + const transport = answering(200, body); + expect(await new NpmRegistry({ transport }).published('ailoud')).toEqual([ + { version: '1.0.0', deprecated: false }, + { version: '1.0.1', deprecated: true }, + ]); + }); + + it('throws when the body has no versions', async () => { + // A silent empty answer would read as "you are up to date", which is the + // one wrong thing a version check can say. + const transport = answering(200, '{}'); + await expect(new NpmRegistry({ transport }).published('ailoud')).rejects.toThrow(/versions/); + }); +}); + +describe('isDeprecated', () => { + // The single rule both `packages/providers/src/update/npmRegistry.ts` and + // `apps/cli/src/updateNotice.ts` import from here -- so this file is what + // keeps the two call sites from drifting apart again, the way they did + // before this rule had one home. + it('treats an empty deprecation message as not deprecated', () => { + // `npm deprecate @ ""` un-deprecates by setting an empty + // string rather than removing the field. Testing the key's presence + // rather than the value reports a revived version as still deprecated. + expect(isDeprecated({ deprecated: '' })).toBe(false); + }); + + it('treats a non-empty deprecation message as deprecated', () => { + expect(isDeprecated({ deprecated: 'do not use' })).toBe(true); + }); + + it('treats a boolean true as deprecated', () => { + expect(isDeprecated({ deprecated: true })).toBe(true); + }); + + it('treats a missing field as not deprecated', () => { + expect(isDeprecated({})).toBe(false); + }); + + it('treats a non-object entry as not deprecated', () => { + expect(isDeprecated(null)).toBe(false); + expect(isDeprecated('nope')).toBe(false); + }); +}); diff --git a/packages/providers/src/update/npmRegistry.ts b/packages/providers/src/update/npmRegistry.ts new file mode 100644 index 0000000..55946db --- /dev/null +++ b/packages/providers/src/update/npmRegistry.ts @@ -0,0 +1,143 @@ +import { request as httpsRequest } from 'node:https'; +import type { PublishedVersion, VersionSource } from '@ailoud/core'; +import { FailureError } from '@ailoud/core'; + +/** + * Exported so callers report the same host and wait that this class would use + * by default. Two copies of these numbers drift, and then `self check` names a + * timeout the registry client never applied. + */ +export const DEFAULT_REGISTRY = 'https://registry.npmjs.org'; +export const DEFAULT_TIMEOUT_MS = 10_000; + +const REGISTRY = DEFAULT_REGISTRY; +const TIMEOUT_MS = DEFAULT_TIMEOUT_MS; + +/** + * Fetches one URL and reports its status and body text. + * + * A seam rather than `fetch` itself, for a measured reason: aborting a `fetch` + * does NOT release the socket, so a process that gives up on a request still + * waits about 10.5 seconds to exit (Node 24, measured against an + * unresponsive address). `https.request`'s native `signal` destroys the + * socket at once and the process exits in about 60ms. The background update + * check must abandon a request the instant a command is ready to finish, so + * that difference decides the implementation -- and it is the whole reason + * this class, rather than a second hand-rolled client, can serve both callers. + */ +export type RegistryTransport = ( + url: string, + headers: Record, + signal: AbortSignal, +) => Promise<{ readonly status: number; readonly body: string }>; + +export interface NpmRegistryOptions { + readonly registry?: string; + readonly timeoutMs?: number; + /** Injected in tests, so a unit test never opens a socket. */ + readonly transport?: RegistryTransport; +} + +const httpsTransport: RegistryTransport = (url, headers, signal) => + new Promise((resolve, reject) => { + const request = httpsRequest(url, { headers, signal }, (response) => { + const chunks: Buffer[] = []; + response.on('data', (chunk: Buffer) => chunks.push(chunk)); + response.on('end', () => + resolve({ status: response.statusCode ?? 0, body: Buffer.concat(chunks).toString('utf8') }), + ); + }); + request.on('error', reject); + request.end(); + }); + +/** + * Which versions of a package exist, read from the npm registry. + * + * Only the abbreviated packument is requested (the `accept` header below), + * which still carries every version's `deprecated` flag -- the field the + * whole update policy turns on -- at a fraction of the size of the full + * document. + */ +export class NpmRegistry implements VersionSource { + private readonly registry: string; + private readonly timeoutMs: number; + private readonly transport: RegistryTransport; + + constructor(options: NpmRegistryOptions = {}) { + this.registry = options.registry ?? REGISTRY; + this.timeoutMs = options.timeoutMs ?? TIMEOUT_MS; + this.transport = options.transport ?? httpsTransport; + } + + async published(packageName: string, signal?: AbortSignal): Promise { + // Same escaping npm itself uses: the slash in a scoped name would + // otherwise be a path separator. + const url = `${this.registry}/${packageName.replaceAll('/', '%2f')}`; + // The caller's cancellation AND our own timeout: whichever fires first + // wins. A caller that passes no signal still gets the timeout. + const deadline = AbortSignal.timeout(this.timeoutMs); + const combined = signal === undefined ? deadline : AbortSignal.any([signal, deadline]); + const response = await this.transport( + url, + { accept: 'application/vnd.npm.install-v1+json' }, + combined, + ); + if (response.status < 200 || response.status >= 300) { + throw new FailureError( + `the npm registry answered ${response.status} for ${packageName}, so ailoud cannot tell which versions exist.`, + ); + } + const body: unknown = JSON.parse(response.body); + const versions = + typeof body === 'object' && body !== null + ? (body as { versions?: unknown }).versions + : undefined; + if (typeof versions !== 'object' || versions === null) { + throw new FailureError( + `the npm registry returned no versions for ${packageName}, so ailoud cannot tell which versions exist.`, + ); + } + const published = Object.entries(versions as Record).map( + ([version, entry]) => ({ version, deprecated: isDeprecated(entry) }), + ); + // An empty list is not an answer. `{"versions": {}}` with a 200 would + // otherwise resolve to [], which every caller reads as "nothing newer + // exists" -- the one wrong thing a version check can say. A package that + // really has no versions cannot be the one we are running. + if (published.length === 0) { + throw new FailureError( + `the npm registry listed no versions of ${packageName}, so ailoud cannot tell which versions exist.`, + ); + } + return published; + } +} + +/** + * Whether the registry says this version is deprecated. + * + * The value matters, not the key. npm stores the deprecation MESSAGE here, and + * `npm deprecate @ ""` un-deprecates by setting an empty string + * rather than removing the field. Testing `'deprecated' in entry` therefore + * reports a revived version as still deprecated, which would refuse a + * legitimate update and, if a registry emitted the empty form widely, refuse + * every update. + * + * It lives HERE, in the provider, rather than in the domain: it decodes one + * field of npm's packument wire format, which is a provider's business and + * not a rule about versions. `apps/cli` reaches it through this package, so + * nothing needs a copy. + * + * `scripts/retire-prereleases.mjs` answers a related question with its own + * truthiness test, deliberately, and says why there. It is NOT importing this + * function, so do not describe this as the only copy -- an earlier version of + * this comment did, which made it false. + */ +export function isDeprecated(entry: unknown): boolean { + if (typeof entry !== 'object' || entry === null) return false; + const flag: unknown = (entry as { deprecated?: unknown }).deprecated; + if (typeof flag === 'string') return flag.length > 0; + // Not a shape npm documents, but a boolean true is unambiguous if it appears. + return flag === true; +} diff --git a/scripts/check-changelog.mjs b/scripts/check-changelog.mjs new file mode 100755 index 0000000..c452029 --- /dev/null +++ b/scripts/check-changelog.mjs @@ -0,0 +1,89 @@ +#!/usr/bin/env node +// Verify CHANGES.md is fit to release a given tag. +// +// Usage: node scripts/check-changelog.mjs +// The tag falls back to $GITHUB_REF_NAME; a single leading `v` is stripped. +// +// Run on every tag before anything is published. A version number can never +// be reused and the unpublish window is 72 hours, so the changelog is worth +// checking while refusing is still free. +import { + HARD_LIMIT, + SOFT_LIMIT, + baseVersion, + countBullets, + escapeForRegExp, + fail, + isPrerelease, + readChanges, + splitSections, + versionFromTag, + versionHeading, + warn, +} from './lib/changelog.mjs'; + +const SCOPE = 'check-changelog'; + +const rawTag = process.argv[2] ?? process.env.GITHUB_REF_NAME; +if (!rawTag) fail(SCOPE, 'no tag given (pass one, or set $GITHUB_REF_NAME)'); + +const version = versionFromTag(rawTag); +const { sections } = splitSections(readChanges()); + +// Collected rather than thrown one at a time: someone fixing a changelog +// before a release wants the whole list, not one round trip per problem. +const problems = []; + +const own = sections.find((section) => versionHeading(version).test(section.heading)); +if (own === undefined) { + problems.push(`no "## Version ${version}" section. Promote it with bump-version first.`); +} + +const entries = own === undefined ? 0 : countBullets(own.body); +if (own !== undefined && entries === 0) { + problems.push(`the "## Version ${version}" section has no entries.`); +} +if (entries > HARD_LIMIT) { + problems.push( + `${entries} entries in this version; the hard limit is ${HARD_LIMIT}. Merge or cut some.`, + ); +} + +// A final tag must not leave its own pre-release sections behind: they +// describe the same release, and a reader of 1.0.0's notes should not have to +// read 1.0.0-dev.1's as well. fold-prereleases exists to merge them. +if (!isPrerelease(version)) { + const base = baseVersion(version); + const leftovers = sections + .map((section) => section.heading) + .filter((heading) => new RegExp(`^## Version ${escapeForRegExp(base)}-`).test(heading)); + if (leftovers.length > 0) { + problems.push( + `${leftovers.length} pre-release section(s) for ${base} are still present ` + + `(${leftovers.map((h) => h.replace('## Version ', '')).join(', ')}). ` + + `Run: node scripts/fold-prereleases.mjs ${base}`, + ); + } +} + +// Nothing may be stranded in Development at release time: an entry left there +// is a change that shipped and went unmentioned. +const development = sections.find((section) => /^## Development$/.test(section.heading)); +if (development !== undefined) { + const stranded = countBullets(development.body); + if (stranded > 0) { + problems.push( + `${stranded} entr${stranded === 1 ? 'y is' : 'ies are'} still under "## Development".`, + ); + } +} + +if (problems.length > 0) { + for (const problem of problems) console.error(`${SCOPE}: ${problem}`); + process.exit(1); +} + +if (entries > SOFT_LIMIT) { + warn(`${SCOPE}: ${entries} entries, over the soft limit of ${SOFT_LIMIT}.`); +} +console.log(`${SCOPE}: ${version} is ready (${entries} entries).`); diff --git a/scripts/check-changelog.test.mjs b/scripts/check-changelog.test.mjs new file mode 100644 index 0000000..83e418e --- /dev/null +++ b/scripts/check-changelog.test.mjs @@ -0,0 +1,104 @@ +import { describe, expect, it } from 'vitest'; +import { changes, entries, makeSandbox, run, useSandboxes } from './testing/harness.mjs'; + +useSandboxes(); + +describe('check-changelog', () => { + it('passes a version with entries and nothing stranded', () => { + const dir = makeSandbox(changes(`## Development\n\n## Version 1.0.0\n\n### Added\n\n- One.\n`)); + const result = run(dir, 'check-changelog.mjs', ['v1.0.0']); + expect(result.code).toBe(0); + expect(result.stdout).toMatch(/1\.0\.0 is ready \(1 entries\)/); + }); + + it('refuses a version with no section', () => { + const dir = makeSandbox(changes('## Development\n\n- Something.\n')); + const result = run(dir, 'check-changelog.mjs', ['v9.9.9']); + expect(result.code).toBe(1); + expect(result.stderr).toMatch(/no "## Version 9\.9\.9" section/); + }); + + it('refuses a section with no entries', () => { + const dir = makeSandbox(changes('## Development\n\n## Version 1.0.0\n\n### Added\n')); + expect(run(dir, 'check-changelog.mjs', ['1.0.0']).stderr).toMatch(/has no entries/); + }); + + it('refuses entries left stranded under Development', () => { + // An entry left there is a change that shipped and went unmentioned. + const dir = makeSandbox( + changes('## Development\n\n- Forgotten.\n\n## Version 1.0.0\n\n- One.\n'), + ); + const result = run(dir, 'check-changelog.mjs', ['1.0.0']); + expect(result.code).toBe(1); + expect(result.stderr).toMatch(/still under "## Development"/); + }); + + it('refuses a final tag whose pre-release sections were never folded', () => { + const dir = makeSandbox( + changes( + '## Development\n\n## Version 1.0.0\n\n- One.\n\n## Version 1.0.0-dev.1\n\n- Older.\n', + ), + ); + const result = run(dir, 'check-changelog.mjs', ['1.0.0']); + expect(result.code).toBe(1); + expect(result.stderr).toMatch(/pre-release section\(s\) for 1\.0\.0 are still present/); + expect(result.stderr).toMatch(/fold-prereleases\.mjs 1\.0\.0/); + }); + + it('allows a PRE-RELEASE tag to coexist with its siblings', () => { + // Only a final tag has to have folded them. + const dir = makeSandbox( + changes( + '## Development\n\n## Version 1.0.0-dev.2\n\n- Two.\n\n## Version 1.0.0-dev.1\n\n- One.\n', + ), + ); + expect(run(dir, 'check-changelog.mjs', ['v1.0.0-dev.2']).code).toBe(0); + }); + + it('reports every problem at once, not one per run', () => { + const dir = makeSandbox(changes('## Development\n\n- Stranded.\n')); + const result = run(dir, 'check-changelog.mjs', ['1.0.0']); + expect(result.stderr).toMatch(/no "## Version 1\.0\.0" section/); + expect(result.stderr).toMatch(/still under "## Development"/); + }); + + it('refuses past the hard limit and only warns past the soft one', () => { + const over = makeSandbox(changes(`## Development\n\n## Version 1.0.0\n\n${entries(51)}\n`)); + const hard = run(over, 'check-changelog.mjs', ['1.0.0']); + expect(hard.code).toBe(1); + expect(hard.stderr).toMatch(/hard limit is 50/); + + const soft = makeSandbox(changes(`## Development\n\n## Version 1.0.0\n\n${entries(12)}\n`)); + const result = run(soft, 'check-changelog.mjs', ['1.0.0']); + expect(result.code).toBe(0); + expect(`${result.stdout}${result.stderr}`).toMatch(/soft limit of 10/); + }); + + it('refuses with no tag at all', () => { + const dir = makeSandbox(changes('## Development\n')); + expect(run(dir, 'check-changelog.mjs').stderr).toMatch(/no tag given/); + }); + + it('falls back to $GITHUB_REF_NAME, which is how the workflow calls it', () => { + const dir = makeSandbox(changes('## Development\n\n## Version 1.0.0\n\n- One.\n')); + const result = run(dir, 'check-changelog.mjs', [], { env: { GITHUB_REF_NAME: 'v1.0.0' } }); + expect(result.code).toBe(0); + expect(result.stdout).toMatch(/1\.0\.0 is ready/); + }); + + it('warns as a GitHub annotation when it runs on a runner', () => { + // ::warning:: goes to stdout and shows up on the run summary; the plain + // warning goes to stderr. Both paths are exercised because CI once saw + // one of them and no test ever had. + const body = `## Development\n\n## Version 1.0.0\n\n${entries(12)}\n`; + const annotated = run(makeSandbox(body), 'check-changelog.mjs', ['1.0.0'], { + env: { GITHUB_ACTIONS: 'true' }, + }); + expect(annotated.stdout).toMatch(/::warning::.*soft limit of 10/); + expect(annotated.stderr).toBe(''); + + const plain = run(makeSandbox(body), 'check-changelog.mjs', ['1.0.0']); + expect(plain.stderr).toMatch(/soft limit of 10/); + expect(plain.stdout).not.toContain('::warning::'); + }); +}); diff --git a/scripts/check-dependency-age.mjs b/scripts/check-dependency-age.mjs new file mode 100644 index 0000000..d02f43a --- /dev/null +++ b/scripts/check-dependency-age.mjs @@ -0,0 +1,110 @@ +#!/usr/bin/env node +// Refuse dependency versions that are too fresh to trust. +// +// Usage: node scripts/check-dependency-age.mjs [--days 14] +// +// WHY +// +// A compromised release is discovered by other people, and that takes days: +// the npm supply-chain incidents of recent years were all caught within a week +// or two of publication, after the malicious version had already been installed +// by everyone who upgraded immediately. Waiting is the whole mitigation, and it +// costs nothing -- there is no urgency in a patch that has been out for two +// weeks that was not there on day one. +// +// Checked against the version each manifest PINS, not what is installed: this +// project pins exact versions, so the manifest is the decision and the lockfile +// only records it. A spec that is not an exact version is reported rather than +// aged, because a range means the decision was left to whatever resolved last. +import { existsSync, readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { escapePackageName } from './lib/npmOidc.mjs'; +import { + DEFAULT_DAYS, + EXACT, + classify, + collectDependencies, + staleExceptions, +} from './lib/dependencyAge.mjs'; + +const SCOPE = 'check-dependency-age'; +const REGISTRY = 'https://registry.npmjs.org'; +const EXCEPTIONS = 'scripts/dependency-age-exceptions.json'; + +const root = join(dirname(fileURLToPath(import.meta.url)), '..'); +const read = (path) => readFileSync(join(root, path), 'utf8'); + +/** + * When `version` was published, or null if the registry does not know. + * + * A 404 means the registry has no such package, which is a real answer. Any + * other refusal -- a 429, a 502 -- is the registry declining to answer, and + * returning null for it turned this whole check into a no-op that reported + * success: every entry became "unknown", nothing was young, exit 0. So those + * throw, and the rule fails closed. + */ +async function publishedAt(name, version) { + const response = await fetch(`${REGISTRY}/${escapePackageName(name)}`); + if (response.status === 404) return null; + if (!response.ok) { + throw new Error(`the registry answered ${response.status} for ${name}; cannot judge its age`); + } + const { time } = await response.json(); + const stamp = time?.[version]; + if (typeof stamp !== 'string') return null; + const parsed = Date.parse(stamp); + // An unparseable timestamp is not an age. Reported as unknown rather than + // compared as NaN, which produced "published NaN days ago". + return Number.isFinite(parsed) ? parsed : null; +} + +const daysFlag = process.argv.indexOf('--days'); +const days = daysFlag === -1 ? DEFAULT_DAYS : Number(process.argv[daysFlag + 1]); +if (!Number.isFinite(days) || days < 0) { + console.error(`${SCOPE}: --days needs a non-negative number`); + process.exit(1); +} + +const exceptions = existsSync(join(root, EXCEPTIONS)) ? JSON.parse(read(EXCEPTIONS)) : {}; +const dependencies = collectDependencies(read); +const unpinned = [...dependencies].filter(([, spec]) => !EXACT.test(spec)); +const pinned = [...dependencies].filter(([, spec]) => EXACT.test(spec)); + +const entries = await Promise.all( + pinned.map(async ([name, spec]) => [name, spec, await publishedAt(name, spec)]), +); +const now = Date.now(); +const { young, exempt, unknown } = classify(entries, now, days, exceptions); + +console.log(`${SCOPE}: ${pinned.length} pinned direct dependencies, minimum age ${days} days`); + +for (const [name, spec] of unpinned) { + console.log(` ${name}@${spec} is not an exact version; its age was not checked`); +} +for (const { name, spec } of unknown) { + console.log(` ${name}@${spec}: the registry reported no publish time`); +} +for (const { name, spec, ageDays, reason } of exempt) { + console.log(` ${name}@${spec} is ${ageDays.toFixed(1)} days old, allowed: ${reason}`); +} +for (const key of staleExceptions(exceptions, entries, now, days)) { + // Not a failure: a hole that has closed is only clutter. Left unreported it + // would still be there the next time something needed exempting. + console.log(` ${EXCEPTIONS} no longer needs its entry for ${key}`); +} + +if (young.length === 0) { + console.log(`${SCOPE}: nothing newer than ${days} days.`); + process.exit(0); +} + +for (const { name, spec, ageDays } of young) { + console.error(`${SCOPE}: ${name}@${spec} was published ${ageDays.toFixed(1)} days ago`); +} +console.error( + `${SCOPE}: wait until each is ${days} days old, or pin the previous version. ` + + 'A compromised release is found by other people, and that takes days. ' + + `If one fixes a critical advisory, add it to ${EXCEPTIONS} with the advisory ID.`, +); +process.exit(1); diff --git a/scripts/check-docs-render.mjs b/scripts/check-docs-render.mjs new file mode 100644 index 0000000..b81d0c4 --- /dev/null +++ b/scripts/check-docs-render.mjs @@ -0,0 +1,56 @@ +#!/usr/bin/env node +// Fail the build on a documentation page that RENDERED wrong, not one that +// failed to build. +// +// Usage: +// pnpm docs:build && node scripts/check-docs-render.mjs +// +// `mkdocs build --strict` checks the nav, internal links and cross-references, +// and stays green as long as every piece of markdown is valid -- it has no +// idea what the page looks like once Material has rendered it. A `!!! note` +// block whose content is not indented four spaces is valid markdown that +// means something other than its author intended: Material renders an empty +// box, and the note's own text falls out below it as an ordinary paragraph. +// That shipped in this repository once. This script reads the built HTML +// under site/ and looks for the textual symptom each of these leaves behind: +// +// - an admonition div holding nothing but its title +// - a table's header row surviving as literal text, its separator missing +// - a stray triple backtick, left by a code fence closed in the wrong place +// - a "#Heading" run that never became a real heading +// +// See scripts/lib/checkDocsRender.mjs for how each is detected and why. +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { checkSite } from './lib/checkDocsRender.mjs'; + +const SCOPE = 'check-docs-render'; +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); + +const { filesChecked, counts, failures } = checkSite(ROOT); + +if (filesChecked === 0) { + console.error(`${SCOPE}: no HTML under site/ -- run \`pnpm docs:build\` first.`); + process.exit(1); +} + +console.log( + `${SCOPE}: checked ${filesChecked} page(s) -- ` + + `${counts.admonitions} admonition(s), ${counts.tables} table(s), ` + + `${counts.codeBlocks} code block(s), ${counts.headings} heading(s).`, +); + +if (failures.length > 0) { + for (const { file, kind, snippet } of failures) { + console.error(`${SCOPE}: [${kind}] ${file}`); + console.error(`${SCOPE}: ${snippet}`); + } + console.error( + `${SCOPE}: ${failures.length} rendering problem(s) found. ` + + `The build stayed green; the page did not render as written.`, + ); + process.exit(1); +} + +console.log(`${SCOPE}: clean -- every check found what it should have.`); diff --git a/scripts/check-docs-render.test.mjs b/scripts/check-docs-render.test.mjs new file mode 100644 index 0000000..5d2ec7e --- /dev/null +++ b/scripts/check-docs-render.test.mjs @@ -0,0 +1,76 @@ +// This is the thin-wrapper test: it only checks that the CLI finds site/, +// reports counts, and exits non-zero with the offending file named. The +// detection rules themselves are covered in scripts/lib/checkDocsRender.test.mjs, +// against the pure functions directly, not by spawning a process per case. +import { cpSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; + +import { REPO, run } from './testing/harness.mjs'; + +const made = []; +afterEach(() => { + for (const dir of made.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +/** + * A throwaway repository holding only the scripts and a fixture site/ tree. + * + * check-docs-render.mjs resolves site/ relative to its own file location, not + * the working directory, so copying scripts/ next to a fixture tree puts it + * somewhere with content we control rather than the real (and constantly + * rebuilt) site/. + */ +function makeSiteSandbox(files) { + const dir = mkdtempSync(join(tmpdir(), 'ailoud-check-docs-render-')); + made.push(dir); + cpSync(join(REPO, 'scripts'), join(dir, 'scripts'), { recursive: true }); + for (const [relativePath, content] of Object.entries(files)) { + const full = join(dir, 'site', relativePath); + mkdirSync(dirname(full), { recursive: true }); + writeFileSync(full, content, 'utf8'); + } + return dir; +} + +const ARTICLE = (inner) => `
${inner}
`; + +describe('check-docs-render', () => { + it('exits clean and reports counts for a healthy site', () => { + const dir = makeSiteSandbox({ + 'index.html': ARTICLE('

Home

'), + 'mcp/index.html': ARTICLE( + '

Note

Fine.

', + ), + }); + + const result = run(dir, 'check-docs-render.mjs', [], { cwd: dir }); + expect(result.code).toBe(0); + expect(result.stdout).toMatch(/checked 2 page\(s\)/); + expect(result.stdout).toMatch(/clean/); + }); + + it('exits non-zero and names the file holding a broken admonition', () => { + const dir = makeSiteSandbox({ + 'mcp/index.html': ARTICLE( + '

Note

Fell out.

', + ), + }); + + const result = run(dir, 'check-docs-render.mjs', [], { cwd: dir }); + expect(result.code).toBe(1); + expect(result.stderr).toMatch(/\[admonition\]/); + expect(result.stderr).toMatch(/mcp[/\\]index\.html/); + }); + + it('refuses to run with no site/ built yet', () => { + const dir = mkdtempSync(join(tmpdir(), 'ailoud-check-docs-render-empty-')); + made.push(dir); + cpSync(join(REPO, 'scripts'), join(dir, 'scripts'), { recursive: true }); + + const result = run(dir, 'check-docs-render.mjs', [], { cwd: dir }); + expect(result.code).toBe(1); + expect(result.stderr).toMatch(/pnpm docs:build/); + }); +}); diff --git a/scripts/docs-surface.mjs b/scripts/docs-surface.mjs new file mode 100644 index 0000000..e589309 --- /dev/null +++ b/scripts/docs-surface.mjs @@ -0,0 +1,19 @@ +#!/usr/bin/env node +// Print one sorted, deduplicated line per command, subcommand or flag +// documented anywhere under docs/ or in README.md. +// +// Usage: node scripts/docs-surface.mjs > surface.txt +// +// This is the safety net for compressing the documentation. Capture this +// output before a prose-cutting pass and after; comm -23 before.txt after.txt +// must print nothing. Anything it does print was documented and is not +// anymore -- only prose may be cut, never a command, subcommand, flag or +// option. See .superpowers/plans/2026-09-06-documentation-mcp-first.md. +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +import { buildSurface } from './lib/docsSurface.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); + +for (const line of buildSurface(ROOT)) console.log(line); diff --git a/scripts/docs-surface.test.mjs b/scripts/docs-surface.test.mjs new file mode 100644 index 0000000..3ad681e --- /dev/null +++ b/scripts/docs-surface.test.mjs @@ -0,0 +1,68 @@ +// This is the thin-wrapper test: it only checks that the CLI finds the right +// files and prints what the library builds. The extraction rules themselves +// are covered in scripts/lib/docsSurface.test.mjs, against the pure function +// directly, not by spawning a process per case. +import { cpSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { afterEach, describe, expect, it } from 'vitest'; + +import { REPO, run } from './testing/harness.mjs'; + +const made = []; +afterEach(() => { + for (const dir of made.splice(0)) rmSync(dir, { recursive: true, force: true }); +}); + +/** + * A throwaway repository holding only the scripts and a fixture docs/ tree. + * + * docs-surface.mjs resolves docs/ and README.md relative to its own file + * location, not the working directory, so copying scripts/ next to a fixture + * tree is what puts it somewhere with content we control rather than the real + * (and constantly changing) documentation. + */ +function makeDocsSandbox(files) { + const dir = mkdtempSync(join(tmpdir(), 'ailoud-docs-surface-')); + made.push(dir); + cpSync(join(REPO, 'scripts'), join(dir, 'scripts'), { recursive: true }); + mkdirSync(join(dir, 'docs'), { recursive: true }); + for (const [relativePath, content] of Object.entries(files)) { + const full = join(dir, relativePath); + mkdirSync(dirname(full), { recursive: true }); + writeFileSync(full, content, 'utf8'); + } + return dir; +} + +describe('docs-surface', () => { + it('reports the sorted, deduplicated surface across README.md and docs/', () => { + const dir = makeDocsSandbox({ + 'README.md': '```shell\nailoud audio ls --json\n```\n', + 'docs/usage/recordings.md': 'Run `ailoud audio ls` again for the same thing.\n', + 'docs/mcp.md': '`ailoud mcp` serves the library. See `ailoud mcp install`.\n', + }); + + const result = run(dir, 'docs-surface.mjs', [], { cwd: dir }); + expect(result.code).toBe(0); + expect(result.stderr).toBe(''); + + const lines = result.stdout.trim().split('\n'); + expect(lines).toEqual([...lines].sort()); + expect(new Set(lines).size).toBe(lines.length); + expect(lines).toEqual( + expect.arrayContaining(['--json', 'ailoud audio ls', 'ailoud mcp', 'ailoud mcp install']), + ); + }); + + it('reads only README.md and docs/, not a sibling file like CONTRIBUTING.md', () => { + const dir = makeDocsSandbox({ + 'README.md': 'Nothing documented here.\n', + 'CONTRIBUTING.md': '`ailoud audio ls --secret`\n', + }); + + const result = run(dir, 'docs-surface.mjs', [], { cwd: dir }); + expect(result.code).toBe(0); + expect(result.stdout.trim()).toBe(''); + }); +}); diff --git a/scripts/fold-prereleases.mjs b/scripts/fold-prereleases.mjs new file mode 100755 index 0000000..2cce9a4 --- /dev/null +++ b/scripts/fold-prereleases.mjs @@ -0,0 +1,103 @@ +#!/usr/bin/env node +// Fold the pre-release sections of a version back into one released section. +// +// Usage: node scripts/fold-prereleases.mjs +// +// Cutting 1.0.0 after 1.0.0-dev.1, -dev.2 and -rc.1 leaves four changelog +// sections describing one release: three pre-release ones plus whatever +// accumulated in `## Development`. A reader of the released notes wants one. +// This merges them into `## Version `, keeps the subsection grouping, +// drops duplicates, and removes the pre-release sections. +// +// Only the SAME version's pre-releases are folded: 1.0.0-dev.1 goes into 1.0.0 +// and never into 1.1.0. That is what stops an entry from an abandoned line +// reappearing under a release it was never part of. +import { + HARD_LIMIT, + SOFT_LIMIT, + escapeForRegExp, + fail, + fingerprint, + groupBullets, + readChanges, + splitSections, + versionFromTag, + versionHeading, + warn, + writeChanges, +} from './lib/changelog.mjs'; + +const SCOPE = 'fold-prereleases'; + +const version = versionFromTag(process.argv[2] ?? ''); +if (!/^\d+\.\d+\.\d+$/.test(version)) { + fail(SCOPE, `expected a released version like 1.0.0, got "${process.argv[2] ?? ''}"`); +} + +const { head, sections } = splitSections(readChanges()); + +const isOwnPrerelease = (heading) => + new RegExp(`^## Version ${escapeForRegExp(version)}-`).test(heading); +const isDevelopment = (heading) => /^## Development$/.test(heading); +const isTarget = (heading) => versionHeading(version).test(heading); +const isFolded = (heading) => + isOwnPrerelease(heading) || isDevelopment(heading) || isTarget(heading); + +const folded = sections.filter((section) => isFolded(section.heading)); +if (folded.length === 0) fail(SCOPE, 'nothing to fold: no Development and no matching sections'); +const prereleaseCount = folded.filter((section) => isOwnPrerelease(section.heading)).length; + +// Sections are read in document order, which puts Development first and the +// pre-releases below it, newest to oldest. First occurrence of a duplicate +// wins, so the newest wording of an entry is the one kept. +const merged = new Map(); +const seen = new Set(); +for (const section of folded) { + for (const [name, entries] of groupBullets(section.body)) { + for (const entry of entries) { + const key = fingerprint(entry); + if (key === '' || seen.has(key)) continue; + seen.add(key); + if (!merged.has(name)) merged.set(name, []); + merged.get(name).push(entry); + } + } +} + +const total = [...merged.values()].reduce((sum, entries) => sum + entries.length, 0); +if (total === 0) fail(SCOPE, 'every folded section was empty'); +if (total > HARD_LIMIT) { + fail( + SCOPE, + `the folded section would have ${total} entries; the hard limit is ${HARD_LIMIT}. ` + + 'Merge related entries in CHANGES.md before cutting the release.', + ); +} +if (total > SOFT_LIMIT) { + warn(`${SCOPE}: ${total} entries, over the soft limit of ${SOFT_LIMIT}.`); +} + +const body = [...merged.entries()] + .filter(([, entries]) => entries.length > 0) + .map(([name, entries]) => [`### ${name}`, '', ...entries.map((e) => e.join('\n'))].join('\n')) + .join('\n\n'); + +const kept = sections.filter((section) => !isFolded(section.heading)); + +writeChanges( + [ + head.replace(/\n+$/, ''), + '## Development', + `## Version ${version}`, + body, + ...kept.map((section) => [section.heading, section.body.replace(/\n+$/, '')].join('\n')), + ] + .filter((part) => part !== '') + .join('\n\n') + .replace(/\n{3,}/g, '\n\n'), +); + +console.log( + `${SCOPE}: folded ${prereleaseCount} pre-release section(s) and Development into ` + + `## Version ${version} (${total} entries)`, +); diff --git a/scripts/fold-prereleases.test.mjs b/scripts/fold-prereleases.test.mjs new file mode 100644 index 0000000..8e78ea0 --- /dev/null +++ b/scripts/fold-prereleases.test.mjs @@ -0,0 +1,114 @@ +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { changes, entries, makeSandbox, run, useSandboxes } from './testing/harness.mjs'; + +useSandboxes(); + +describe('fold-prereleases', () => { + const withPrereleases = changes( + [ + '## Development', + '', + '### Added', + '', + '- Newest, from Development.', + '', + '## Version 1.0.0-dev.2', + '', + '### Added', + '', + '- From dev.2.', + '- A duplicate that', + ' wraps across lines.', + '', + '### Fixed', + '', + '- Only dev.2 had this.', + '', + '## Version 1.0.0-dev.1', + '', + '### Added', + '', + '- From dev.1.', + '- A duplicate that wraps across lines.', + '', + '## Version 0.9.0', + '', + '### Added', + '', + '- An older release.', + '', + ].join('\n'), + ); + + it('merges the pre-releases and Development into one released section', () => { + const dir = makeSandbox(withPrereleases); + const result = run(dir, 'fold-prereleases.mjs', ['1.0.0']); + expect(result.code).toBe(0); + const out = readFileSync(join(dir, 'CHANGES.md'), 'utf8'); + expect(out).toContain('## Version 1.0.0'); + expect(out).not.toContain('1.0.0-dev.1'); + expect(out).not.toContain('1.0.0-dev.2'); + }); + + it('drops a duplicate even when one copy was rewrapped', () => { + const dir = makeSandbox(withPrereleases); + run(dir, 'fold-prereleases.mjs', ['1.0.0']); + const out = readFileSync(join(dir, 'CHANGES.md'), 'utf8'); + expect(out.match(/duplicate that/g)).toHaveLength(1); + }); + + it('keeps the subsection grouping', () => { + const dir = makeSandbox(withPrereleases); + run(dir, 'fold-prereleases.mjs', ['1.0.0']); + const out = readFileSync(join(dir, 'CHANGES.md'), 'utf8'); + expect(out).toContain('### Added'); + expect(out).toContain('### Fixed'); + expect(out).toContain('- Only dev.2 had this.'); + }); + + it('leaves an unrelated release alone', () => { + // 1.0.0-dev.1 folds into 1.0.0 and never into another version. + const dir = makeSandbox(withPrereleases); + run(dir, 'fold-prereleases.mjs', ['1.0.0']); + const out = readFileSync(join(dir, 'CHANGES.md'), 'utf8'); + expect(out).toContain('## Version 0.9.0'); + expect(out).toContain('- An older release.'); + }); + + it('leaves an empty Development heading for the next cycle', () => { + const dir = makeSandbox(withPrereleases); + run(dir, 'fold-prereleases.mjs', ['1.0.0']); + const out = readFileSync(join(dir, 'CHANGES.md'), 'utf8'); + const development = out.slice(out.indexOf('## Development'), out.indexOf('## Version 1.0.0')); + expect(development).not.toMatch(/^- /m); + }); + + it('is idempotent: folding twice changes nothing more', () => { + const dir = makeSandbox(withPrereleases); + run(dir, 'fold-prereleases.mjs', ['1.0.0']); + const once = readFileSync(join(dir, 'CHANGES.md'), 'utf8'); + run(dir, 'fold-prereleases.mjs', ['1.0.0']); + expect(readFileSync(join(dir, 'CHANGES.md'), 'utf8')).toBe(once); + }); + + it('refuses anything that is not a released version', () => { + const dir = makeSandbox(withPrereleases); + for (const bad of ['1.0.0-dev.1', 'latest', '1.0', '']) { + const result = run(dir, 'fold-prereleases.mjs', bad === '' ? [] : [bad]); + expect(result.code, bad).toBe(1); + expect(result.stderr, bad).toMatch(/expected a released version/); + } + }); + + it('refuses past the hard limit rather than writing an oversized section', () => { + const dir = makeSandbox(changes(`## Development\n\n${entries(51)}\n`)); + const before = readFileSync(join(dir, 'CHANGES.md'), 'utf8'); + const result = run(dir, 'fold-prereleases.mjs', ['1.0.0']); + expect(result.code).toBe(1); + expect(result.stderr).toMatch(/hard limit is 50/); + // And left the file exactly as it was. + expect(readFileSync(join(dir, 'CHANGES.md'), 'utf8')).toBe(before); + }); +}); diff --git a/scripts/lib/changelog.mjs b/scripts/lib/changelog.mjs new file mode 100644 index 0000000..879b9fa --- /dev/null +++ b/scripts/lib/changelog.mjs @@ -0,0 +1,152 @@ +// Shared vocabulary for the release scripts: the limits, the two ways of +// reporting, and the CHANGES.md parsing all three of them need. +// +// Extracted because the numbers and the parsing were copied into every script +// that touched the changelog. Copied limits drift, and a limit that differs +// between the script that warns and the script that refuses is worse than no +// limit -- one of them is then wrong and nobody knows which. +import { readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +/** + * The published packages, in dependency order. + * + * The CLI is last because the other two are its dependencies: publish and + * deprecate both have to walk them in this order. + */ +export const PACKAGES = ['@ailoud/core', '@ailoud/providers', 'ailoud']; + +/** + * Which pre-release tags belong to a released version, and which may be deleted. + * + * Pure: it takes the tag list and an "is this commit on main" predicate rather + * than running git, so the decision is testable without a repository. + */ +export function planRetirement(version, tags, isOnMain) { + const prefix = `v${version}-`; + const mine = tags.filter((tag) => tag.startsWith(prefix)).sort(); + return { + versions: mine.map((tag) => versionFromTag(tag)), + deletable: mine.filter((tag) => isOnMain(tag)), + kept: mine.filter((tag) => !isOnMain(tag)), + }; +} + +/** Entries per version section. Stated once here, quoted in AGENTS.md and CHANGES.md. */ +export const SOFT_LIMIT = 10; +export const HARD_LIMIT = 50; + +/** Where the repository root is, relative to scripts/lib/. */ +export const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..'); +export const CHANGES_PATH = join(ROOT, 'CHANGES.md'); + +/** Stops with a message the caller owns. */ +export function fail(scope, message) { + console.error(`${scope}: ${message}`); + process.exit(1); +} + +/** + * A warning, not an error. + * + * `console.error` made a soft-limit notice look like a failure in a terminal + * and in a CI log alike. Under GitHub Actions this becomes an annotation, + * which is what a warning should be there. + */ +export function warn(message) { + if (process.env.GITHUB_ACTIONS === 'true') { + console.log(`::warning::${message}`); + return; + } + console.warn(message); +} + +/** `v1.2.3` and `1.2.3` both mean 1.2.3. */ +export function versionFromTag(tag) { + return String(tag).replace(/^v/, ''); +} + +/** The release part of a version: `1.0.0-dev.2` -> `1.0.0`. */ +export function baseVersion(version) { + return version.split('-')[0]; +} + +export function isPrerelease(version) { + return version.includes('-'); +} + +/** Safe inside a RegExp built from a version string. */ +export function escapeForRegExp(value) { + return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +export function readChanges() { + return readFileSync(CHANGES_PATH, 'utf8'); +} + +export function writeChanges(text) { + writeFileSync(CHANGES_PATH, `${text.trimEnd()}\n`); +} + +/** Every `## ` section as {heading, body}, plus whatever precedes the first one. */ +export function splitSections(text) { + const lines = text.split('\n'); + const starts = []; + lines.forEach((line, at) => { + if (line.startsWith('## ')) starts.push(at); + }); + const head = lines.slice(0, starts[0] ?? lines.length).join('\n'); + const sections = starts.map((start, index) => ({ + heading: lines[start], + body: lines.slice(start + 1, starts[index + 1] ?? lines.length).join('\n'), + })); + return { head, sections }; +} + +/** + * Bullets of a section body, grouped by their `### ` subsection. + * + * A continuation line is attached to the bullet above it rather than dropped, + * which is what keeps an 80-column entry whole -- every wrapped entry would + * otherwise lose everything after its first line. + */ +export function groupBullets(body) { + const groups = new Map(); + let current = 'Added'; + for (const line of body.split('\n')) { + if (line.startsWith('### ')) { + current = line.slice(4).trim(); + if (!groups.has(current)) groups.set(current, []); + continue; + } + if (/^\s*- /.test(line)) { + if (!groups.has(current)) groups.set(current, []); + groups.get(current).push([line]); + continue; + } + const entries = groups.get(current); + if (entries !== undefined && entries.length > 0 && line.trim() !== '') { + entries[entries.length - 1].push(line); + } + } + return groups; +} + +export function countBullets(body) { + return body.split('\n').filter((line) => /^\s*- /.test(line)).length; +} + +/** Whitespace-insensitive, so the same entry rewrapped is still the same entry. */ +export function fingerprint(entry) { + return entry.join(' ').replace(/\s+/g, ' ').trim().toLowerCase(); +} + +/** The body of the section whose heading matches, or null. */ +export function findSection(sections, pattern) { + return sections.find((section) => pattern.test(section.heading)) ?? null; +} + +export function versionHeading(version) { + return new RegExp(`^## Version ${escapeForRegExp(version)}(\\s|$)`); +} diff --git a/scripts/lib/changelog.test.mjs b/scripts/lib/changelog.test.mjs new file mode 100644 index 0000000..a65495b --- /dev/null +++ b/scripts/lib/changelog.test.mjs @@ -0,0 +1,196 @@ +import { describe, expect, it } from 'vitest'; +import { + HARD_LIMIT, + SOFT_LIMIT, + baseVersion, + countBullets, + escapeForRegExp, + fingerprint, + groupBullets, + isPrerelease, + planRetirement, + splitSections, + versionFromTag, + versionHeading, +} from './changelog.mjs'; + +describe('the limits', () => { + it('are stated once, and the soft one is below the hard one', () => { + // Copied limits drift, and a limit that differs between the script that + // warns and the script that refuses is worse than no limit at all. + expect(SOFT_LIMIT).toBe(10); + expect(HARD_LIMIT).toBe(50); + expect(SOFT_LIMIT).toBeLessThan(HARD_LIMIT); + }); +}); + +describe('versionFromTag', () => { + it('strips one leading v and nothing else', () => { + expect(versionFromTag('v1.2.3')).toBe('1.2.3'); + expect(versionFromTag('1.2.3')).toBe('1.2.3'); + expect(versionFromTag('v1.2.3-dev.1')).toBe('1.2.3-dev.1'); + // Not a recursive strip: a version does not start with v twice. + expect(versionFromTag('vv1.2.3')).toBe('v1.2.3'); + }); +}); + +describe('baseVersion and isPrerelease', () => { + it('splits a pre-release from its release', () => { + expect(baseVersion('1.0.0-dev.2')).toBe('1.0.0'); + expect(baseVersion('1.0.0')).toBe('1.0.0'); + expect(isPrerelease('1.0.0-dev.2')).toBe(true); + expect(isPrerelease('1.0.0-rc.1')).toBe(true); + expect(isPrerelease('1.0.0')).toBe(false); + }); +}); + +describe('escapeForRegExp and versionHeading', () => { + it('does not let a dot in a version match any character', () => { + // Unescaped, `1.0.0` matches `1x0y0` -- and a heading for another version. + expect(escapeForRegExp('1.0.0')).toBe('1\\.0\\.0'); + expect(versionHeading('1.0.0').test('## Version 1.0.0')).toBe(true); + expect(versionHeading('1.0.0').test('## Version 1x0y0')).toBe(false); + }); + + it('matches a heading with a trailing date but not a longer version', () => { + expect(versionHeading('1.0.0').test('## Version 1.0.0 -- 2026-09-05')).toBe(true); + expect(versionHeading('1.0.0').test('## Version 1.0.0-dev.1')).toBe(false); + expect(versionHeading('1.0.0').test('## Version 1.0.10')).toBe(false); + }); +}); + +describe('splitSections', () => { + const text = [ + '# Title', + '', + 'preamble', + '', + '## Development', + '', + '- one', + '', + '## Version 1.0.0', + '', + '- two', + ].join('\n'); + + it('keeps whatever precedes the first section', () => { + expect(splitSections(text).head).toContain('preamble'); + }); + + it('returns each section with its heading and body', () => { + const { sections } = splitSections(text); + expect(sections.map((s) => s.heading)).toEqual(['## Development', '## Version 1.0.0']); + expect(sections[0].body).toContain('- one'); + expect(sections[1].body).toContain('- two'); + }); + + it('handles a file with no sections at all', () => { + const { head, sections } = splitSections('# Just a title\n'); + expect(sections).toEqual([]); + expect(head).toContain('Just a title'); + }); + + it('does not treat a ### subsection as a section', () => { + const { sections } = splitSections('## Version 1.0.0\n\n### Added\n\n- x\n'); + expect(sections).toHaveLength(1); + expect(sections[0].body).toContain('### Added'); + }); +}); + +describe('groupBullets', () => { + it('groups by subsection, in first-seen order', () => { + const groups = groupBullets('### Added\n\n- a\n\n### Fixed\n\n- b\n'); + expect([...groups.keys()]).toEqual(['Added', 'Fixed']); + expect(groups.get('Added')).toEqual([['- a']]); + }); + + it('keeps a wrapped entry whole', () => { + // Every 80-column entry wraps; dropping the continuation would lose + // everything after the first line. + const groups = groupBullets('### Added\n\n- first line\n second line\n'); + expect(groups.get('Added')).toEqual([['- first line', ' second line']]); + }); + + it('defaults to Added when a body has no subsection heading', () => { + expect([...groupBullets('- loose entry\n').keys()]).toEqual(['Added']); + }); + + it('ignores a blank line between entries', () => { + expect(groupBullets('- a\n\n- b\n').get('Added')).toEqual([['- a'], ['- b']]); + }); + + it('records a subsection that has no entries', () => { + const groups = groupBullets('### Added\n\n### Fixed\n\n- b\n'); + expect(groups.get('Added')).toEqual([]); + }); +}); + +describe('fingerprint', () => { + it('sees the same entry rewrapped as the same entry', () => { + // The reason a duplicate across two dev tags is caught even after one of + // them was reflowed. + expect(fingerprint(['- a duplicated entry that', ' wraps across two lines.'])).toBe( + fingerprint(['- a duplicated entry that wraps across two lines.']), + ); + }); + + it('is case-insensitive', () => { + expect(fingerprint(['- Same Thing'])).toBe(fingerprint(['- same thing'])); + }); + + it('keeps genuinely different entries apart', () => { + expect(fingerprint(['- one'])).not.toBe(fingerprint(['- two'])); + }); +}); + +describe('countBullets', () => { + it('counts entries, not their continuation lines', () => { + expect(countBullets('- a\n continued\n- b\n')).toBe(2); + }); + + it('counts an indented entry', () => { + expect(countBullets(' - nested\n')).toBe(1); + }); + + it('counts nothing in prose', () => { + expect(countBullets('### Added\n\nsome prose\n')).toBe(0); + }); +}); + +describe('planRetirement', () => { + const tags = [ + 'v1.0.0-dev.2', + 'v1.0.0-dev.1', + 'v1.0.0-rc.1', + 'v1.1.0-dev.1', + 'v1.0.0', + 'v10.0.0-dev.1', + ]; + + it('takes the pre-releases of one version and no others', () => { + // v1.1.0-dev.1 belongs to a version that has not been released, and + // v10.0.0-dev.1 shares a prefix with "v1" only as text. + const { versions } = planRetirement('1.0.0', tags, () => true); + expect(versions).toEqual(['1.0.0-dev.1', '1.0.0-dev.2', '1.0.0-rc.1']); + }); + + it('leaves the final release itself alone', () => { + const { versions } = planRetirement('1.0.0', tags, () => true); + expect(versions).not.toContain('1.0.0'); + }); + + it('splits the tags by whether their commit is on main', () => { + // A tag whose commit is not reachable from main cannot be deleted: the + // commit could then be collected, and the published provenance attests it. + const onMain = (tag) => tag !== 'v1.0.0-dev.2'; + const { deletable, kept } = planRetirement('1.0.0', tags, onMain); + expect(deletable).toEqual(['v1.0.0-dev.1', 'v1.0.0-rc.1']); + expect(kept).toEqual(['v1.0.0-dev.2']); + }); + + it('plans nothing for a version that never had a pre-release', () => { + const plan = planRetirement('2.0.0', tags, () => true); + expect(plan).toEqual({ versions: [], deletable: [], kept: [] }); + }); +}); diff --git a/scripts/lib/checkDocsRender.mjs b/scripts/lib/checkDocsRender.mjs new file mode 100644 index 0000000..784cc1b --- /dev/null +++ b/scripts/lib/checkDocsRender.mjs @@ -0,0 +1,293 @@ +// Pure logic behind scripts/check-docs-render.mjs. +// +// Kept out of the CLI file on purpose: this repository has twice had to move +// logic out of a `scripts/*.mjs` entry point into `scripts/lib/` because a +// test that imported the CLI file ran the CLI. Nothing here touches the +// filesystem except `collectSiteFiles` and `checkSite`, which only read. +// +// Why this exists: `mkdocs build --strict` checks the nav, internal links and +// references, but it has no idea what the page LOOKS like once Material has +// rendered it. A `!!! note` block whose content is not indented four spaces +// builds green and renders an empty box, with the note's own text falling out +// as an ordinary paragraph below it -- that shipped in this repository once. +// The strict build cannot see it because nothing about the markdown was +// invalid; it rendered exactly as written, just not as intended. The four +// checks here all share that shape: each looks for the literal, textual +// symptom left behind in the HTML when a piece of markdown parsed as +// something other than what its author meant, rather than for a markdown +// error the build would already have caught. +import { existsSync, readdirSync, readFileSync } from 'node:fs'; +import { join } from 'node:path'; + +/** + * The main content region of one rendered page -- between `
` + * and its matching `
` -- or the whole document if a page has no + * article wrapper (there is exactly one per page in this theme, but a + * missing wrapper should not crash the check, only widen it). + * + * Restricting every check to this region is what keeps the nav, the search + * modal markup and the generated table of contents from ever being mistaken + * for page content: none of those are what an author wrote. + */ +export function extractArticle(html) { + const open = /]*>/.exec(html); + if (!open) return html; + const start = open.index + open[0].length; + const end = html.indexOf('', start); + return end === -1 ? html.slice(start) : html.slice(start, end); +} + +/** + * `articleHtml` with every `
...
` block removed. + * + * The table, fence and heading checks all look for a piece of markdown + * syntax leaking into the page as literal text -- a real, correctly + * highlighted code sample is exactly the one place that kind of text is + * supposed to appear (a shell comment starting with `#`, a shown-not-run + * command piped with `|`), so it is cut before any of those checks run. + */ +function withoutCodeBlocks(articleHtml) { + return articleHtml.replace(/]*>[\s\S]*?<\/pre>/g, ''); +} + +/** + * `html` with every tag removed, leaving only the text a reader would see. + * + * Two things a one-pass `<[^>]+>` gets wrong, and both are why this looks + * heavier than it should: + * + * A quoted attribute may CONTAIN `>`. `
` stops the naive + * match early and leaves `">` behind as stray text -- which matters because + * the callers hunt for markdown syntax that leaked into the page as literal + * text, and a leftover fragment is exactly what they would report. A checker + * that invents findings gets ignored, which costs more than it saves. So the + * pattern below steps over quoted runs. + * + * And stripping once is not enough: on `<