feat(release): gate every CLI change on an integration tier, report each next release

.agents/skills/cli-tests/SKILL.md

@@ -0,0 +1,114 @@
+---
+name: cli-tests
+description: How to test a change to the wego CLI in this repository. Use when adding or changing a command, a flag, output, an error path or an API call, when writing or moving a unit test or an integration scenario, or when a fixture or the contract check fails.
+---
+
+# Testing a CLI change
+
+The rules are in `AGENTS.md`; this is the procedure. The short version: parsers and
+logic get unit tests, everything a caller sees gets an integration scenario, and
+nothing here talks to a real API.
+
+## 1. Choose the tier
+
+| You are testing | Tier | Where |
+|---|---|---|
+| How argv becomes API-call arguments (a flag, a bound, a default) | Unit, on the `parse*Args` function or a plain function | `src/<module>.test.ts` |
+| HTTP client behaviour: query mapping, headers, retries, timeouts, tolerant parsing | Unit | `src/api.test.ts` |
+| Pure logic: settle loops, formatting, precedence, PKCE, storage | Unit | beside the module |
+| Exit code, stdout, stderr, what reaches the wire, files written | Integration scenario | `integration/<area>.test.ts` |
+| Install, update, uninstall, signing | Artifact checks | release workflows and `scripts/*.sh` |
+| Behaviour against staging, skill quality | Not here | wego-ai's next smoke and evals |
+
+If a unit test needs a command's output, it is an integration scenario. Importing
+`run` or a command handler (`login`, `whoami`, `places`, `info`, `feedback`,
+`flights`, `hotels`, `logout`, `config`, `telemetry`) into a unit test fails
+`scripts/unit-tier-guard.test.ts`. If the logic you need is inside a handler,
+export it as a plain function and unit-test that.
+
+## 2. Write the scenario
+
+Read `integration/README.md` once, then copy the shape of a neighbouring file.
+
+```ts
+import { expect, it } from "bun:test";
+import { readFixture, route } from "./harness/fixtures";
+import { useScenario } from "./harness/scenario";
+import { json, signIn } from "./harness/wego";
+
+const s = useScenario();
+
+it("prints the caller's identity", async () => {
+  signIn(s.home); // as a previous login left it
+  const fake = s.fake({ routes: [route("user")] });
+  const result = await s.run(["whoami"]);
+
+  expect(result.code).toBe(0);
+  expect(json(result).sub).toBe(readFixture("user").body.sub);
+  expect(fake.requests("getCurrentUser")[0]?.token).toBe("access-1");
+});
+```
+
+- Assert on `result.code`, `result.out`, `result.err`, `fake.seen` /
+  `fake.requests(op)` (path, query, body, token), and files in `s.home`.
+- Values from the API are read from the fixture; values the scenario sets (argv,
+  settings) may be written out.
+- A route's answers are served in order and the last repeats: a settling search
+  needs `route(first, then)`, not a read count.
+- Logged-out, expired and refreshed sessions: `signIn(s.home, {...})` plus the
+  fake's `accept` and `refresh` options (see `integration/auth.test.ts`).
+- Faults: `{ fault: "non-json" }`, `startDropper()` (a reset) and `startDropper({ partial: true })` (a body cut off).
+- The test fails by itself if any request or answer breaks the contract, or a
+  request reaches a route nobody declared. Do not assert that separately.
+- Keep a scenario under a few seconds. A path that needs a long wait is tested
+  as a plain function; keep one fast scenario for it end to end.
+
+### When a command or flag changes
+
+`skills/wego/SKILL.md` tells the user's agent which commands to run, and it ships
+inside the binary. `integration/skill-matches-cli.test.ts` fails when the skill
+names a command that no longer answers `--help`, or a `--flag` its help does not
+list. Rename a flag, and update the skill in the same change. Whether the skill
+still leads the agent well is the evals' question, answered per release in wego-ai.
+
+## 3. Get the fixture
+
+There is no recorder or generator. Take the first that fits:
+
+1. **Reuse** an existing file in `integration/fixtures/`.
+2. **Edit a copy** in the scenario: `answer("flights-results", (b) => ({ ...b, results: [] }))`.
+3. **Errors inline**: `problem(404, "not_found")`. Codes are the contract's closed
+   set: `validation_failed`, `invalid_token`, `insufficient_scope`, `not_found`,
+   `rates_require_hotel_search`, `rate_limited`, `bad_gateway`,
+   `upstream_unavailable`, `upstream_rate_limited`, `internal_error`. A status the
+   operation does not declare is rejected: the contract declares no 410 and no
+   500, so use 502 `bad_gateway` for a server failure.
+4. **New file**, only for an operation nothing answers yet:
+   `{ "op": "<operationId>", "status": 200, "body": { … } }`. Then run
+   `bun test --preload ./integration/harness/preload.ts ./integration/fixtures.test.ts`
+   and fix each path it names until it passes. A few list items, not a page.
+   Optionally base the body on a real staging answer with `curl` (the command is
+   in `integration/README.md`), never on `wego` output, and replace the account's
+   identity with `integration@example.com`, `Integration Test`, `1001`.
+
+If the contract rejects a fixture, fix the fixture. Never loosen
+`integration/harness/contract.ts` or `fake.ts` to make a scenario pass.
+
+## 4. When the API changed
+
+1. Refresh the contract: `bun run api-contract:refresh` (see `CONTRIBUTING.md`,
+   "When the API changes"), committed on its own.
+2. `bun run typecheck`: the static contract checks name what the CLI's types no
+   longer match.
+3. `bun run test:integration`: the fixtures the new contract rejects fail by name
+   in `fixtures.test.ts`; fix them, then the scenarios.
+
+## 5. Before you finish
+
+```sh
+bun run check
+bun run test:integration
+```
+
+Both clean. In the pull request, say which tier each new test is in, and for any
+test you removed, where its coverage went.

.claude/skills

@@ -0,0 +1 @@
+../.agents/skills

.github/CODEOWNERS

@@ -175,6 +175,19 @@ src/skill-embed.ts @sunny-wego @yeouchien-wego @chuyeowego
 # redirected token.
 contract/openapi.json @sunny-wego @yeouchien-wego @chuyeowego
 
+# THE INTEGRATION TIER. Unlike the unit tests below, it is a release gate:
+# `release` needs `integration (<target>)` on all five targets, so what this
+# directory accepts is what may be published. Two things in it are a small diff
+# away from making that gate vacuous or unsafe:
+#
+#   - `harness/contract.ts` and `harness/fake.ts` hold every exchange to the
+#     contract. Relaxing the validator, or letting an unexpected request pass,
+#     turns the tier back into a fake that can only fail when it disagrees with
+#     itself, and every scenario still reads green.
+#   - `fixtures/` is what the fake answers with. A fixture copied from a real
+#     response must not carry the account it came from; review is the check.
+integration/ @sunny-wego @yeouchien-wego @chuyeowego
+
 # Deliberately NOT owned, so the omissions read as decisions rather than gaps:
 #
 #   - `src/commands.ts` and `src/index.ts` wire the login flow together, but they

.github/workflows/ci-cli.yml

@@ -1,6 +1,7 @@
 name: ci-cli
 
-# Lint, typecheck and unit tests. Runs on every pull request and every push to main.
+# Lint, typecheck, unit tests and the integration suite. Runs on every pull
+# request and every push to main.
 #
 # The job name `ci-cli` is the required status check on `main`. Do not rename it
 # without a ruleset change. There is no path filter: a filtered workflow does not
@@ -231,3 +232,20 @@ jobs:
       # so it would audit the same commit concurrently.
       - name: Audit the installed dependency tree (bun.lock)
         run: bun audit --audit-level=high
+
+      # The compiled `wego` binary against a local fake API (`integration/`). The
+      # unit tests above run the source; this runs what `bun build --compile` makes
+      # of it, which is what ships. Hermetic: the fake is a `Bun.serve` on loopback
+      # and every other request is refused, so a red here is this change, never the
+      # network.
+      #
+      # A step of `ci-cli`, not a job of its own, so the one required check on
+      # `main` already blocks on it and no ruleset change is needed. linux-x64 only:
+      # the suite compiles the host binary when `WEGO_INTEGRATION_BINARY` is unset.
+      # The other four targets run in `release-cli.yml`, against the binaries that
+      # release built.
+      - name: Integration tests (compiled binary, fake API)
+        env:
+          # The suite drives a production build. Do not post CI runs as product usage.
+          WEGO_CLI_TELEMETRY: "0"
+        run: bun run test:integration

.github/workflows/promote-cli.yml

@@ -29,6 +29,54 @@ permissions:
   contents: read
 
 jobs:
+  # wego-ai's verdicts for the tag, the smoke (`cli-next-smoke`) and the skill
+  # evals (`cli-next-evals`), one line each with a link, at the top of this run's
+  # summary, so the person promoting sees them beside the gates. The evals can
+  # take hours, so this is often the first place their result is shown.
+  # `release-cli.yml`'s `next-report` shows the smoke's full table.
+  #
+  # READ-ONLY AND GATES NOTHING. No job needs it, it is `continue-on-error`, and
+  # the script exits 0 on every state it knows, so a missing, running or bad
+  # report never holds a promote back and never lets one through. The promoter
+  # decides; the banner only makes sure the verdict was in front of them.
+  #
+  # One look, no polling: "No report yet, started N min ago" is a true answer, and
+  # a promote that waited behind a smoke would be a gate by another name.
+  #
+  # `checks: read` and `contents: read`, nothing else: no environment, no secret,
+  # no `id-token`. It checks out `main` (the dispatch ref), not the tag: a tag cut
+  # before `scripts/next-report.ts` existed has no script to run. The script asks
+  # the API for the tag's commit and reads only a check written by wego-ai's App.
+  next-report:
+    name: next report for ${{ inputs.tag }}
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    continue-on-error: true
+    permissions:
+      contents: read
+      checks: read
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+
+      - uses: ./.github/actions/setup-bun
+
+      - name: Show the next report banner
+        continue-on-error: true
+        env:
+          TAG: ${{ inputs.tag }}
+          GITHUB_TOKEN: ${{ github.token }}
+        run: |
+          set +e
+          bun run scripts/next-report.ts --banner
+          code=$?
+          if [ "$code" -ne 0 ]; then
+            echo "● No report: next-report exited $code" >> "$GITHUB_STEP_SUMMARY"
+            echo "::warning::next-report exited $code before writing a banner."
+          fi
+          exit 0
+
   # The human gate.
   #
   # Not an environment with required reviewers - and no longer because we cannot
@@ -198,31 +246,63 @@ jobs:
           echo "rollback target: $tag via $lane"
 
       # `cli/<tag>/` can exist half-written if the release lane did not finish.
-      # Require one completed, successful `release-cli.yml` run for this tag.
-      # `branch=<tag>` is how the Actions API filters a tag-push run.
-      # `github-script`, not `gh`: one shape across the lanes.
+      # Require one `release-cli.yml` run for this tag in which the publishing job
+      # ran and every job succeeded, apart from the two report-only jobs.
+      #
+      # Job by job, not the run's conclusion: `notify-verify` goes red when the
+      # verification receiver refuses or is down, and `next-report` keeps the run
+      # in progress for up to 45 min. Neither says anything about the bytes, and
+      # gating on the run would let a receiver outage hold back a fix-forward
+      # promote. Any other job, including one added later, is gated by default.
+      # `scripts/workflow-lanes.test.ts` pins both names below to the jobs in
+      # `release-cli.yml`.
+      #
+      # `branch=<tag>` is how the Actions API filters a tag-push run. `filter:
+      # latest` reads a re-run's latest attempt. `github-script`, not `gh`: one
+      # shape across the lanes.
       - name: Require a completed, successful release run for the tag
         uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
         with:
           script: |
             const tag = process.env.TAG;
+            const REPORT_ONLY = new Set(["Request release verification", "next report"]);
+            const PUBLISH = "Publish CLI binaries to cli/next";
             const { data } = await github.rest.actions.listWorkflowRuns({
               owner: context.repo.owner,
               repo: context.repo.repo,
               workflow_id: "release-cli.yml",
               branch: tag,
               per_page: 100,
             });
-            const ok = data.workflow_runs.filter(
-              (r) => r.status === "completed" && r.conclusion === "success",
-            ).length;
+            let ok = 0;
+            for (const run of data.workflow_runs) {
+              const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                run_id: run.id,
+                filter: "latest",
+                per_page: 100,
+              });
+              const gated = jobs.filter((j) => !REPORT_ONLY.has(j.name));
+              const published = gated.some((j) => j.name === PUBLISH);
+              const failing = gated.filter(
+                (j) => j.status !== "completed" || j.conclusion !== "success",
+              );
+              if (published && failing.length === 0) {
+                ok += 1;
+              } else {
+                core.info(
+                  `run ${run.id}: ${published ? "" : `no "${PUBLISH}" job; `}${failing.map((j) => `${j.name}=${j.conclusion ?? j.status}`).join(", ")}`,
+                );
+              }
+            }
             if (ok < 1) {
               core.setFailed(
-                `No completed, successful release-cli.yml run for ${tag}. cli/stable serves every install, so it is only ever advanced onto bytes a release run finished and verified. Re-run or fix the release for ${tag} first.`,
+                `No release-cli.yml run for ${tag} in which every publishing job completed successfully. cli/stable serves every install, so it is only ever advanced onto bytes a release run finished and verified. Re-run or fix the release for ${tag} first.`,
               );
               return;
             }
-            core.info(`ok: release-cli.yml completed successfully for ${tag} (${ok} run(s)).`);
+            core.info(`ok: release-cli.yml published and verified ${tag} (${ok} run(s)). Report-only jobs are not gated.`);
         env:
           TAG: ${{ inputs.tag }}