diff --git a/.github/releases/v1.0.40.md b/.github/releases/v1.0.40.md new file mode 100644 index 0000000000..2df486a0b9 --- /dev/null +++ b/.github/releases/v1.0.40.md @@ -0,0 +1,53 @@ +## opencode {VERSION} + +{Prerelease/Stable} release from `{branch}` branch. Native LLM requests now settle local tools before automatic compaction can close the stream. This release also includes the reviewed event-storage, summary-diff, delivery, and macOS installer fixes already integrated into dev. + +--- + +### 🐛 Bug Fixes + +- **Tools survive automatic compaction, #539**: a high-usage `step-finish` could abort a slow local tool before its result reached the session processor. Native LLM now delivers all local tool results before terminal events. Parallel tools settle completely; explicit user cancellation still interrupts execution. +- **Summary diffs retain later small entries, #526**: skip an oversized diff individually instead of dropping every following entry. Remove the unused legacy `session.summary_diffs` column through a tested database migration. +- **Identical durable events no longer consume storage or sequence numbers, #527**: suppress byte-identical fresh appends within the same aggregate/type while preserving explicit-sequence replay. Batch results retain input alignment; legacy rows require no hash backfill. +- **Config startup preserves npm lock files, #542**: keep an existing lock unchanged when the plugin SDK resolves entirely from local or bundled packages. Mixed registry requests and genuine package changes still regenerate the lock. + +--- + +### 🏗️ Architecture / Refactor + +- **Deleted-session storage reclamation, #537**: remove durable event residue for deleted aggregates, wire cleanup into session deletion, and add tested SQLite reclamation support. This release does not run the deferred #531 maintenance operation on the user's existing database. + +--- + +### ⚙️ CI / Engineering + +- **Delivery tracking, #520 and #532 through #535**: close linked issues after dev merges, preserve repository-specific SpecGit harness files, restore failed bootstrap state, reject unsupported branch types before remote writes, and verify that delivery PRs target dev. +- **macOS installation verification, #536**: verify release archive checksums before extraction and validate the installed binary's signature after quarantine clearing and re-signing. Added a negative checksum control and a real macOS installation acceptance test. +- **Local npm fixture isolation, #540**: keep real package-installation regressions independent of online vulnerability-audit latency while retaining their assertions and deadlines. + +--- + +### 🧪 Test Summary + +``` +Release integration CI (PR #539, 00695dab86): +core: 1225 pass, 6 skip, 0 fail +opencode: 4429 pass, 23 skip, 1 todo, 0 fail +HttpAPI coverage / auth / effect: 230 pass each, no failures or missing routes +Generated client and SDK freshness: passed +Typecheck, DAG core gate, Linux and Windows E2E: passed + +Merged native/session/TUI regressions: 36 pass, 0 fail +Merged npm regressions: 8 pass, 0 fail +Merged opencode package typecheck: passed +``` + +--- + +### 🔍 Verification + +The slow-tool regression was observed failing before the fix and passing afterward through the real session processor and a local HTTP model endpoint. Additional cases cover parallel local tools and explicit cancellation. Independent Standards, Spec and merge reviews found no code blockers. The integration statistics above come from the accepted [PR #539 CI](https://github.com/LeXwDeX/OpenCode-GraphAgent/actions/runs/33894718562), including generated client/SDK freshness and all three HttpAPI modes. [Typecheck, lint and DAG core](https://github.com/LeXwDeX/OpenCode-GraphAgent/actions/runs/33894718548), both E2E platforms and SpecGit 1.10.1 acceptance also passed before merge to dev. The release branch preserves that accepted runtime tree; [release PR #544](https://github.com/LeXwDeX/OpenCode-GraphAgent/pull/544) carries its own binding and main-targeted gates. Reported model usage in the regression is deterministic test input; no live model context limit is inferred from it. + +--- + +**Full changelog:** [`{previous_tag}`...`{current_tag}`](https://github.com/LeXwDeX/OpenCode-GraphAgent/compare/{previous_tag}...{current_tag}) diff --git a/.github/workflows/ci-typecheck.yml b/.github/workflows/ci-typecheck.yml index 863d907511..5ac01932b4 100644 --- a/.github/workflows/ci-typecheck.yml +++ b/.github/workflows/ci-typecheck.yml @@ -54,3 +54,10 @@ jobs: working-directory: packages/opencode run: bun run test:dag-core timeout-minutes: 10 + + # #498 B1: archive integrity boundary of the `oc` installer — SHA256SUMS + # must be verified before extraction and fail closed on mismatch. + # Zero network (stub curl); portable across bash hosts. + - name: Run oc install boundary tests + run: bash script/oc-install-boundary.test.sh + timeout-minutes: 5 diff --git a/.github/workflows/dev-issue-autoclose.yml b/.github/workflows/dev-issue-autoclose.yml new file mode 100644 index 0000000000..330a344891 --- /dev/null +++ b/.github/workflows/dev-issue-autoclose.yml @@ -0,0 +1,98 @@ +# ============================================================================ +# 🧹 Dev · Issue Auto-Close +# ---------------------------------------------------------------------------- +# Purpose: Mirror GitHub's native issue auto-close for PRs merged into `dev`. +# Native auto-close (`Closes #n` in the PR body) only fires when a PR +# merges into the DEFAULT branch (`main`). This repo delivers into +# `dev` first (two-tier Git Workflow), so dev-delivered issues would +# otherwise stay open until manual close (#433/#472/#496/#517 et al.). +# Trigger: `pull_request: types: [closed]`. The job-level `if` gates actual +# work to MERGED PRs whose base is `dev`; merges to `main` keep +# GitHub's native auto-close (no overlap). +# Jobs : autoclose — single Linux runner, pure event payload + `gh` CLI. +# No `actions/checkout`, no third-party actions. The PR body reaches +# the script ONLY via `env:` (script-injection safety); refs are +# matched case-insensitively against the official closing keywords +# followed by bare `#n` (plain-text scan of the whole body, matching +# GitHub's own scanner — refs inside fenced code blocks are included, +# best-effort native parity). Shared issue/PR number space guards: +# numbers resolving to a pull request are skipped, nonexistent +# numbers are skipped, already-CLOSED issues are skipped (no +# duplicate comments). Survivors are closed as `completed` with a +# comment naming the delivery PR. +# Notes : The whole matrix (extraction + guards) is exercised by the dry-run +# harness under /tmp/dev-issue-autoclose/ (see issue #519 evidence). +# Runs on every PR close event; non-dev or unmerged closes exit at +# the job-level `if` without consuming a runner step. +# ============================================================================ + +name: 🧹 Dev · Issue Auto-Close + +on: + pull_request: + types: [closed] + +permissions: + contents: read + issues: write + pull-requests: read + +jobs: + autoclose: + name: Auto-close linked issues + if: github.event.pull_request.merged == true && github.event.pull_request.base.ref == 'dev' + runs-on: ubuntu-latest + steps: + - name: Close linked issues referenced in the PR body + env: + GH_TOKEN: ${{ github.token }} + PR_BODY: ${{ github.event.pull_request.body }} + PR_NUMBER: ${{ github.event.pull_request.number }} + PR_URL: ${{ github.event.pull_request.html_url }} + REPO: ${{ github.repository }} + run: | + set -euo pipefail + + if [ -z "$PR_BODY" ]; then + echo "dev-issue-autoclose: PR #$PR_NUMBER has no body; nothing to do" + exit 0 + fi + + # Official closing keywords + bare #n, case-insensitive, deduped. + # Plain-text scan of the whole body (code fences included) mirrors + # GitHub's own scanner; qualified `owner/repo#n` and URL refs do not + # match (whitespace must sit directly before `#`). + refs=$(printf '%s' "$PR_BODY" \ + | grep -oiE '\b(close|closes|closed|fix|fixes|fixed|resolve|resolves|resolved)[[:space:]]+#[0-9]+\b' \ + | grep -oE '#[0-9]+\b' \ + | tr -d '#' \ + | sort -nu \ + || true) + + if [ -z "$refs" ]; then + echo "dev-issue-autoclose: PR #$PR_NUMBER body has no closing-keyword refs; nothing to do" + exit 0 + fi + + echo "dev-issue-autoclose: PR #$PR_NUMBER -> refs: $(echo "$refs" | tr '\n' ' ')" + + for n in $refs; do + # Shared issue/PR number space: skip numbers that resolve to a PR. + if gh pr view "$n" --repo "$REPO" >/dev/null 2>&1; then + echo "dev-issue-autoclose: #$n is a pull request; skipping" + continue + fi + # Skip numbers that do not exist as issues. + if ! state=$(gh issue view "$n" --repo "$REPO" --json state --jq .state 2>/dev/null); then + echo "dev-issue-autoclose: #$n not found; skipping" + continue + fi + # Skip already-closed issues (no duplicate comments). + if [ "$state" = "CLOSED" ]; then + echo "dev-issue-autoclose: #$n is already CLOSED; skipping (no duplicate comment)" + continue + fi + gh issue close "$n" --repo "$REPO" --reason completed \ + --comment "Auto-closed: delivery PR #$PR_NUMBER ([view]($PR_URL)) merged into \`dev\` with a closing keyword for #$n in its body. GitHub's native auto-close only fires on the default branch (\`main\`); this mirrors it for the dev integration layer ([#519](https://github.com/$REPO/issues/519))." + echo "dev-issue-autoclose: #$n closed (reason: completed) by delivery PR #$PR_NUMBER" + done diff --git a/.github/workflows/release-fork.yml b/.github/workflows/release-fork.yml index 06cf50e097..d65620d6a2 100644 --- a/.github/workflows/release-fork.yml +++ b/.github/workflows/release-fork.yml @@ -241,6 +241,17 @@ jobs: fi done + # #498 B2: macOS release acceptance — after the installer-style xattr + + # ad-hoc re-sign mutation, assert codesign validity and executable smoke. + # The installed binary's hash is intentionally NOT compared to the + # archive payload (ad-hoc re-signing can rewrite bytes, so byte equality + # is not a stable signature-validity boundary and differing hashes are + # legitimate); no post-sign digest. + - name: macOS Install Acceptance + if: matrix.name == 'macos' && (inputs.platforms == '' || contains(inputs.platforms, matrix.name)) + run: bash script/oc-macos-acceptance.test.sh packages/opencode/dist/opencode-darwin-arm64.zip + timeout-minutes: 10 + - name: Upload Artifacts if: inputs.platforms == '' || contains(inputs.platforms, matrix.name) uses: actions/upload-artifact@v4 diff --git a/.specgit.yaml b/.specgit.yaml index 5c6fea502b..f4e219ff24 100644 --- a/.specgit.yaml +++ b/.specgit.yaml @@ -1,11 +1,11 @@ version: 1 -delivery: sync-v1-0-39 +delivery: stable-release context: kind: branch - branch: chore/517-sync-v1-0-39 + branch: chore/543-stable-release issues: - - 517 + - 543 issueKinds: - - issue: 517 + - issue: 543 kind: kind::chore -pr: 518 +pr: 544 diff --git a/AGENTS.md b/AGENTS.md index 3a60205123..34724ac48f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,7 +41,7 @@ feat/**, fix/** ──PR(Typecheck + Unit Tests 门禁)──▶ dev ──push 新功能开发、Debug 等一切交付范畴恒定走此循环；后续所有工作必须遵守该方案，不得另起流程： 1. **确立条目**：明确条目的内容、范围、类型（`feat`/`fix`/…）。一个 issue = 一个可独立验证的 WHY，无法独立验证的先拆分再立项。 -2. **SpecGit 立项**：`specgit issue ` 创建/复用 issues 批次，确立交付分支与草稿 PR 脚手架（`.specgit.yaml` 绑定）；立项前先查重，避免同一 WHY 双开。 +2. **SpecGit 立项**：`script/specgit-bootstrap.sh ` 创建/复用 issues 批次，确立交付分支与草稿 PR 脚手架（`.specgit.yaml` 绑定）；立项前先查重，避免同一 WHY 双开。wrapper 是 canonical 入口（见 "SpecGit harness local specializations"）；直跑裸 `specgit issue` 预期被 harness currency gate 以 `harness_stale` (exit 2) 拒绝。 3. **超流执行**：安排 DAG workflow（超流）承载实现——并行开发 + 多角度 Review + 复合（synthesize），其产出作为交付证据基线。 4. **PR 过门禁**：SpecGit 发起/推进 PR，过 TDD 与 CI 门禁（Typecheck、Unit Tests、DAG gate；`specgit finish` exit 0 是唯一 "done"）。 5. **修复门禁问题**：门禁失败在交付分支修代码/测试，永远不削弱门禁本身。 @@ -273,6 +273,15 @@ Kept OUTSIDE the managed block so `specgit init`/`--force` never rewrites them; - The wait script hand-parses `spec_git/policy.yaml` (minimal line-based parse) instead of importing the `yaml` package: no root-reachable `yaml` exists under workspace catalog isolation, so `import { parse } from 'yaml'` would fail to resolve on the runner. - `spec_git/policy.yaml` `required_checks` uses the template's canonical check IDs (`unit-tests`, `e2e-tests`), not display names. +#### specgit-bootstrap wrapper (canonical `specgit issue` entry, #521) + +`script/specgit-bootstrap.sh ` is THE canonical way to run `specgit issue` in this repository. Bare `specgit issue` is expected to fail with `harness_stale` (exit 2) whenever the pinned CLI's harness template moves — the wrapper satisfies that gate safely: it snapshots the full init write surface to a temp dir outside the repo, runs `specgit init --force --no-protect` (hardcoded, offline), then `specgit issue "$@"` with arguments, exit status, and diagnostics passed through verbatim, and restores the specialized bytes above on success and every failure path (EXIT/INT/TERM/HUP), verifying each file byte-for-byte via `git hash-object`. + +- Never run bare `specgit init --force` here: it overwrites the six specialized bytes; the wrapper exists to make that refresh transient. +- Fail-closed rejections: dirty write-surface paths (tracked/staged/untracked) → exit 2 with the offending paths listed; no SpecGit binding (`.specgit.yaml` or `spec_git/policy.yaml` missing) → exit 3; restore hash mismatch → exit 3 with the snapshot kept for forensics. Rejection paths print plain `specgit-bootstrap:` stderr lines and NEVER produce a `--json` envelope. +- The inner `.specgit.yaml` delivery record is rolled back to its pre-run bytes when the inner `specgit issue` exits nonzero (or a signal/init failure interrupts); a successful call keeps the new binding. Record-restore failure keeps the forensic snapshot and exits 3, overriding the inner exit code. Branches, commits, and remote side effects are never undone (#530). +- Managed-block guidance referencing bare `specgit issue` commands is superseded by this section for this repository. Behavior tests: `bash script/specgit-bootstrap.test.sh` (stubbed CLI, zero network; not CI-wired). + ## SpecGit delivery harness diff --git a/README.md b/README.md index 9fb28054d4..5dba3b7383 100644 --- a/README.md +++ b/README.md @@ -287,6 +287,11 @@ All upstream capabilities (multi-provider, built-in LSP, client/server architect Prebuilt CLI binaries (Linux / macOS / Windows, with SHA256SUMS) are published on the [releases page](https://github.com/LeXwDeX/OpenCode-GraphAgent/releases). Builds from `main` are formal releases; builds from `dev` are prereleases. +Release acceptance enforces two distinct integrity boundaries: + +- **Archive integrity (before extraction)**: the `oc` installer verifies the release `SHA256SUMS` entry before unpacking and refuses to extract on mismatch. If upstream serves no `SHA256SUMS`, it installs with a warning (GitHub HTTPS transport only). +- **Post-install signature validity (macOS)**: the installer clears quarantine attributes (`xattr -cr`) and ad-hoc re-signs (`codesign -fs -`); acceptance then asserts `codesign --verify` passes and the binary runs. The installed binary's hash is intentionally **not** compared to the archive payload — ad-hoc signing can rewrite the binary's bytes, so differing hashes are legitimate. No post-signature digest is published (cross-version reproducibility of codesign output has not been established, and no supported reproducibility matrix exists). + From source (requires [Bun](https://bun.sh) 1.3+): ```bash diff --git a/README.zh.md b/README.zh.md index 02e40fc1e4..5e2e0482f4 100644 --- a/README.zh.md +++ b/README.zh.md @@ -250,6 +250,11 @@ DAG 相关的东西都放在 `.opencode/` 下，在 opencode 配置目录（`OPE 预构建 CLI 二进制（Linux / macOS / Windows，附 SHA256SUMS）发布在 [releases 页面](https://github.com/LeXwDeX/OpenCode-GraphAgent/releases)。从 `main` 构建的是正式版；从 `dev` 构建的是预发布版。 +发布验收维护两条相互独立的完整性边界： + +- **归档完整性（解包前）**：`oc` 安装器在解包前校验 release 的 `SHA256SUMS` 条目，不匹配则拒绝解包。若上游未提供 `SHA256SUMS`，则告警后继续安装（仅依赖 GitHub HTTPS 传输安全）。 +- **安装后签名有效性（macOS）**：安装器清除 quarantine 属性（`xattr -cr`）并做 ad-hoc 重签名（`codesign -fs -`），验收断言 `codesign --verify` 通过且二进制可执行。安装后的二进制 hash **有意**不与归档 payload 对比——ad-hoc 签名可能改写二进制字节，两者 hash 即使不同也属正常。也不发布签名后 digest（codesign 输出的跨版本可复现性尚未确立，亦无受支持的可复现性矩阵）。 + 从源码构建（需要 [Bun](https://bun.sh) 1.3+）： ```bash diff --git a/docs/adr/0001-event-residue-scrub-and-sqlite-auto-vacuum-reclamation.md b/docs/adr/0001-event-residue-scrub-and-sqlite-auto-vacuum-reclamation.md new file mode 100644 index 0000000000..996f8f2b25 --- /dev/null +++ b/docs/adr/0001-event-residue-scrub-and-sqlite-auto-vacuum-reclamation.md @@ -0,0 +1,118 @@ +# ADR 0001: Event residue scrub and SQLite auto-vacuum reclamation + +- **Status:** Accepted +- **Date:** 2026-09-04 +- **Issue:** #524 (delivery: PR #537 → `dev`) +- **Supersedes:** none + +## Context + +The durable event store had no reclamation semantics. `Event.remove(aggregateID)` ran only on +explicit session removal and covered the session aggregate alone, and SQLite ran without +`auto_vacuum`, so deleted rows never returned pages to the filesystem. Two failure shapes +motivated the decision: + +- A crash between the session-row delete and any cleanup stranded durable event aggregates + whose `SessionTable` and `WorkflowTable` read models were both gone. Replaying such an + aggregate is impossible: the `WorkflowCreated` projector INSERT dies on the + `workflow.session_id` foreign key once the session row is gone (pinned by + `packages/opencode/test/dag/dag-replay-idempotency.test.ts`), so residue can neither be + replayed nor re-materialized — it can only be deleted. +- `database.ts` initialized WAL and pragmas after driver open, so an application-level + `PRAGMA auto_vacuum` could not take effect: after WAL initialization the pragma silently + yields NONE even on an empty database. + +The decision checkpoint was approved on 2026-09-04 with the scope locked below. + +## Decisions + +1. **Explicit session + dag scrub.** `Session.remove` captures every related dag aggregate ID + before the `Deleted` publish (the projector's session-row delete FK-cascades the workflow + rows inside the publish transaction, so a post-publish lookup would see nothing) and removes + each dag event aggregate after the session aggregate — terminal workflows included. The + per-dag scrub is soft-degrading (a failure is logged and the aggregate is left for the + startup sweep) but preserves interruption (`Cause.hasInterrupts` re-raise, the + `EventResidueSweep` sibling discipline). +2. **Guarded default-on startup sweep.** `EventResidueSweep` runs one pass per process start, + forked into the layer scope so it can neither block nor fail startup. Eligibility is the + zero-live-read-model rule: an aggregate in `event_sequence` with neither a `session` nor a + `workflow` row. Removal is a single atomic guarded `DELETE` that re-evaluates both + NOT EXISTS guards inside the statement (no select-then-delete TOCTOU window), so an + aggregate recreated concurrently survives. Wired into `AppLayer` and the HttpApiApp node + graph, so every serving process sweeps; the pass is idempotent. +3. **New databases: FULL before WAL.** Both SQLite drivers (`sqlite.bun.ts`, `sqlite.node.ts`) + set `auto_vacuum=FULL` at the driver layer, before `journal_mode=WAL`, and only on a + genuinely empty (0-page) file. An immediate SQLITE_BUSY from a second opener racing the + first is tolerated: the pragma is a persistent header property and runs before any + WAL/migration write, so the first-write winner sets FULL for the database. +4. **Existing databases: explicit conversion only.** Legacy `auto_vacuum=NONE` databases are + never converted at startup — startup is detect-only (a warning pointing at the command). The + only conversion path is `opencode db vacuum --db `: the target must be named + explicitly and must already exist as a regular file (vacuum never creates a database), runs + FULL → VACUUM → `wal_checkpoint(TRUNCATE)` outside any startup path, and fails nonzero + unless the `PRAGMA auto_vacuum` readback is exactly FULL. Exclusive access is a hard + requirement (a concurrent writer fails VACUUM with SQLITE_BUSY). +5. **Archived-session retention: off and deferred.** No retention policy for archived sessions + ships in this decision (Phase 3). +6. **Active truncation: rejected.** Truncating active/retained session event history and event + snapshot folding are rejected; incremental replay (`seq > after`, ascending) and sync + cursors must keep observing unbroken per-aggregate histories. +7. **`incremental_vacuum` is forbidden.** A disposable bun:sqlite prototype reproduced an + exit-139 crash under the incremental mode; no code path may enable it. + +## Consequences and risks + +- Deleting events on legacy NONE databases still does not shrink the file until an operator + runs the explicit conversion; disk usage grows until then. +- `auto_vacuum=FULL` pays its known SQLite overhead (pointer-map pages, per-update mapping) on + every new database in exchange for automatic page reclamation. +- The sweep runs once per process start: residue created and abandoned within a single process + lifetime waits for the next start. This is accepted because the shapes it targets are + crash/in-flight zombies. +- The conversion command requires exclusive access; the error guidance says to close running + opencode processes and retry. +- Replay and sync contracts are preserved by construction: only whole aggregates with no live + read model are ever removed, and such aggregates are unreplayable anyway (FK death), so no + consumer can observe the removal as a gap in a replayable history. + +## Alternatives considered + +- **Rely on replay instead of scrubbing** — rejected: a wiped dag aggregate whose session row + is gone dies on the workflow foreign key during re-materialization, so replay cannot replace + deletion. +- **Silent startup conversion of legacy databases** — rejected: converting requires a blocking + full VACUUM; startup stays non-blocking and detect-only. +- **`PRAGMA incremental_vacuum`** — rejected (decision 7). +- **A recurring background reaper** — rejected in favor of one idempotent guarded pass per + process start; residue is crash-shaped, not steady-state throughput. +- **Truncate or fold active event histories** — rejected (decision 6). + +## Rollout and rollback + +Rollout lands as ordinary PRs through `dev` per the release train; no operator action is +required — new databases get FULL automatically, legacy databases keep working unchanged (with +a detect-only warning), and the sweep is default-on. Rollback is removing the sweep from the +app graphs and reverting the driver pragma: the sweep is additive and idempotent, and legacy +databases were never written by any of this. A database created with FULL keeps its header +mode; reverting one is itself an explicit operator VACUUM and is not automated. + +## Acceptance + +- Active/retained session replay is unchanged; only zero-live-read-model aggregates are + removed (guarded delete re-checked inside the statement). +- Cleanup failures never block the application path; interruption is preserved, not logged as + failure. +- Disposable-file tests demonstrate page reclamation and the new/existing database behavior; + no startup-time full VACUUM exists. +- `bun run test:dag-core`, focused event/session tests, package typecheck, and migration + freshness checks pass in CI. + +## Non-goals + +- **No global bounded-retention claim.** Live and retained sessions keep their full event + history indefinitely; this decision bounds nothing by age, size, or count. +- **No tombstones, unarchive, or sync changes.** Offline deletion tombstones, unarchive + semantics, and sync cursor/protocol changes stay out of scope. +- **No authorization for #531 or live-database work.** This decision does not authorize running + VACUUM or any cleanup against a live local database; the destructive operator procedure + remains the human-only issue #531. diff --git a/oc b/oc index 52da44d157..b935d63caa 100755 --- a/oc +++ b/oc @@ -1270,4 +1270,9 @@ main() { # OC 不提供 CLI 子命令。所有功能通过 TUI 菜单访问。 # 无论传入何种参数，均直接进入 TUI 主界面。 -main +# 仅在直接执行时进入 TUI；被 source（测试/验收 harness）时跳过。 +# 用 if 而非 `cond && main`：sourced 且条件为假时 if 返回 0， +# `cond && main` 会以非 0 结束并在 set -e 下中断 source。 +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main +fi diff --git a/packages/core/schema.json b/packages/core/schema.json index 7df86d7742..96ddf9fd6a 100644 --- a/packages/core/schema.json +++ b/packages/core/schema.json @@ -1,9 +1,9 @@ { "version": "7", "dialect": "sqlite", - "id": "874d8e74-d354-4dcb-b98c-c893660c9371", + "id": "7a2e2a70-584a-4604-bf73-4c6e116c20a3", "prevIds": [ - "4142b961-0712-4834-b475-16ea4a74c43c" + "abadf28b-1770-46c6-bbf6-b7800a9ca874" ], "ddl": [ { @@ -1052,6 +1052,16 @@ "entityType": "columns", "table": "event" }, + { + "type": "text", + "notNull": false, + "autoincrement": false, + "default": null, + "generated": null, + "name": "data_hash", + "entityType": "columns", + "table": "event" + }, { "type": "text", "notNull": false, @@ -1772,16 +1782,6 @@ "entityType": "columns", "table": "session" }, - { - "type": "text", - "notNull": false, - "autoincrement": false, - "default": null, - "generated": null, - "name": "summary_diffs", - "entityType": "columns", - "table": "session" - }, { "type": "text", "notNull": false, diff --git a/packages/core/src/database/database.ts b/packages/core/src/database/database.ts index 6879212c41..a8abe21276 100644 --- a/packages/core/src/database/database.ts +++ b/packages/core/src/database/database.ts @@ -2,7 +2,8 @@ export * as Database from "./database" import { EffectDrizzleSqlite } from "@opencode-ai/effect-drizzle-sqlite" import { layer as sqliteLayer } from "#sqlite" -import { Context, Effect, Layer } from "effect" +import { Cause, Context, Effect, Layer } from "effect" +import { sql } from "drizzle-orm" import { Global } from "../global" import { Flag } from "../flag/flag" import { isAbsolute, join } from "path" @@ -29,6 +30,27 @@ export const layer = Layer.effect( yield* db.run("PRAGMA busy_timeout = 5000") yield* db.run("PRAGMA cache_size = -64000") yield* db.run("PRAGMA foreign_keys = ON") + // #524: genuinely new databases were switched to auto_vacuum=FULL by the + // sqlite driver BEFORE WAL init. A legacy database keeps its NONE mode — + // converting one silently at startup would need a blocking full VACUUM — + // so it is only detected and surfaced softly here; conversion is the + // explicit user-triggered `opencode db vacuum --db ` command. + // Detect-only means detect-only: a failed readback degrades to a warning + // (the layer body is orDie'd, so an unhandled failure would kill startup), + // while an interruption is always re-raised. + const autoVacuum = yield* db.get<{ auto_vacuum: number }>(sql`PRAGMA auto_vacuum`).pipe( + Effect.catchCause((cause) => + Cause.hasInterrupts(cause) + ? Effect.interrupt + : Effect.logWarning("database auto_vacuum readback failed — skipping the detect-only check", { cause }).pipe( + Effect.as(undefined), + ), + ), + ) + if (autoVacuum?.auto_vacuum === 0) + yield* Effect.logWarning( + "database auto_vacuum is NONE — deleted pages stay allocated until converted; run `opencode db vacuum --db ` (prints its path with `opencode db path`)", + ) yield* db.run("PRAGMA wal_checkpoint(PASSIVE)") yield* DatabaseMigration.apply(db) diff --git a/packages/core/src/database/migration.gen.ts b/packages/core/src/database/migration.gen.ts index 3d44bcf97d..de37ec19ab 100644 --- a/packages/core/src/database/migration.gen.ts +++ b/packages/core/src/database/migration.gen.ts @@ -56,5 +56,7 @@ export const migrations = ( import("./migration/20260813040429_workflow_directory"), import("./migration/20260815044858_dag_graph_rev_view"), import("./migration/20260815083000_workflow_directory_convergence"), + import("./migration/20260903044702_drop_session_summary_diffs"), + import("./migration/20260903062324_add_event_data_hash"), ]) ).map((module) => module.default) satisfies DatabaseMigration.Migration[] diff --git a/packages/core/src/database/migration/20260903044702_drop_session_summary_diffs.ts b/packages/core/src/database/migration/20260903044702_drop_session_summary_diffs.ts new file mode 100644 index 0000000000..79c3559aae --- /dev/null +++ b/packages/core/src/database/migration/20260903044702_drop_session_summary_diffs.ts @@ -0,0 +1,11 @@ +import { Effect } from "effect" +import type { DatabaseMigration } from "../migration" + +export default { + id: "20260903044702_drop_session_summary_diffs", + up(tx) { + return Effect.gen(function* () { + yield* tx.run(`ALTER TABLE \`session\` DROP COLUMN \`summary_diffs\`;`) + }) + }, +} satisfies DatabaseMigration.Migration diff --git a/packages/core/src/database/migration/20260903062324_add_event_data_hash.ts b/packages/core/src/database/migration/20260903062324_add_event_data_hash.ts new file mode 100644 index 0000000000..a2ca39d724 --- /dev/null +++ b/packages/core/src/database/migration/20260903062324_add_event_data_hash.ts @@ -0,0 +1,11 @@ +import { Effect } from "effect" +import type { DatabaseMigration } from "../migration" + +export default { + id: "20260903062324_add_event_data_hash", + up(tx) { + return Effect.gen(function* () { + yield* tx.run(`ALTER TABLE \`event\` ADD \`data_hash\` text;`) + }) + }, +} satisfies DatabaseMigration.Migration diff --git a/packages/core/src/database/schema.gen.ts b/packages/core/src/database/schema.gen.ts index 25c9b5657b..293e89b4b0 100644 --- a/packages/core/src/database/schema.gen.ts +++ b/packages/core/src/database/schema.gen.ts @@ -149,6 +149,7 @@ export default { \`seq\` integer NOT NULL, \`type\` text NOT NULL, \`data\` text NOT NULL, + \`data_hash\` text, CONSTRAINT \`fk_event_aggregate_id_event_sequence_aggregate_id_fk\` FOREIGN KEY (\`aggregate_id\`) REFERENCES \`event_sequence\`(\`aggregate_id\`) ON DELETE CASCADE ); `) @@ -274,7 +275,6 @@ export default { \`summary_additions\` integer, \`summary_deletions\` integer, \`summary_files\` integer, - \`summary_diffs\` text, \`metadata\` text, \`cost\` real DEFAULT 0 NOT NULL, \`tokens_input\` integer DEFAULT 0 NOT NULL, diff --git a/packages/core/src/database/sqlite.bun.ts b/packages/core/src/database/sqlite.bun.ts index e15f4c117e..3d35f03193 100644 --- a/packages/core/src/database/sqlite.bun.ts +++ b/packages/core/src/database/sqlite.bun.ts @@ -161,11 +161,40 @@ const nativeLayer = (config: Config) => create: config.create ?? true, }) yield* Effect.addFinalizer(() => Effect.sync(() => native.close())) + // #524: auto_vacuum must be set BEFORE any WAL initialization — after + // WAL init the pragma silently yields NONE even on an empty database. + // Only a genuinely empty (0-page) file is eligible: on any existing + // database the pragma would be a no-op at best, so legacy NONE + // databases are never written here (startup stays detect-only; the + // explicit conversion lives in ./vacuum). + native.run("PRAGMA busy_timeout = 5000;") + if (config.readonly !== true) setAutoVacuumFull(native) if (config.disableWAL !== true) native.run("PRAGMA journal_mode = WAL;") return native }), ) +/** + * Setting auto_vacuum needs the write lock of a read-header-then-write + * upgrade, which SQLite fails with an immediate SQLITE_BUSY the busy handler + * cannot retry — a second opener racing the first one's initialization on the + * same new file hits it. Skipping on BUSY is safe: auto_vacuum is a + * persistent header property and every opener runs this pragma BEFORE any + * WAL/migration write, so whichever connection wins the first-write race sets + * FULL for the database. + */ +function setAutoVacuumFull(native: Database) { + const page = native.query<{ page_count: number }, []>("PRAGMA page_count").get() + if (!page || page.page_count !== 0) return + try { + native.run("PRAGMA auto_vacuum = FULL;") + } catch (cause) { + if (!isSqliteBusy(cause)) throw cause + } +} + +const isSqliteBusy = (cause: unknown) => cause instanceof Error && /SQLITE_BUSY|database is locked/i.test(cause.message) + const sqliteLayer = (config: Config) => Layer.effect(Client.SqlClient, make(config)) const drizzleLayer = Layer.effect( diff --git a/packages/core/src/database/sqlite.node.ts b/packages/core/src/database/sqlite.node.ts index 6eaecbee26..3484be76ac 100644 --- a/packages/core/src/database/sqlite.node.ts +++ b/packages/core/src/database/sqlite.node.ts @@ -156,11 +156,40 @@ const nativeLayer = (config: Config) => open: true, }) yield* Effect.addFinalizer(() => Effect.sync(() => native.close())) + // #524: auto_vacuum must be set BEFORE any WAL initialization — after + // WAL init the pragma silently yields NONE even on an empty database. + // On an existing non-empty database the pragma is a SQLite no-op, so + // legacy NONE databases are never converted here (startup stays + // detect-only; the explicit conversion lives in ./vacuum). + native.exec("PRAGMA busy_timeout = 5000;") + if (config.readonly !== true) setAutoVacuumFull(native) if (config.disableWAL !== true && config.readonly !== true) native.exec("PRAGMA journal_mode = WAL;") return native }), ) +/** + * Setting auto_vacuum needs the write lock of a read-header-then-write + * upgrade, which SQLite fails with an immediate SQLITE_BUSY the busy handler + * cannot retry — a second opener racing the first one's initialization on the + * same new file hits it. Skipping on BUSY is safe: auto_vacuum is a + * persistent header property and every opener runs this pragma BEFORE any + * WAL/migration write, so whichever connection wins the first-write race sets + * FULL for the database. + */ +function setAutoVacuumFull(native: DatabaseSync) { + const page: unknown = native.prepare("PRAGMA page_count").get() + const pageCount = typeof page === "object" && page !== null && "page_count" in page ? page.page_count : undefined + if (typeof pageCount !== "number" || pageCount !== 0) return + try { + native.exec("PRAGMA auto_vacuum = FULL;") + } catch (cause) { + if (!isSqliteBusy(cause)) throw cause + } +} + +const isSqliteBusy = (cause: unknown) => cause instanceof Error && /SQLITE_BUSY|database is locked/i.test(cause.message) + const sqliteLayer = (config: Config) => Layer.effect(Client.SqlClient, make(config)) const drizzleLayer = Layer.effect( diff --git a/packages/core/src/database/vacuum.ts b/packages/core/src/database/vacuum.ts new file mode 100644 index 0000000000..434f789cde --- /dev/null +++ b/packages/core/src/database/vacuum.ts @@ -0,0 +1,97 @@ +// SPDX-FileCopyrightText: 2026 LeXwDeX +// SPDX-License-Identifier: AGPL-3.0-or-later + +export * as Vacuum from "./vacuum" + +import { EffectDrizzleSqlite } from "@opencode-ai/effect-drizzle-sqlite" +import { NodeFileSystem } from "@effect/platform-node" +import { Effect, FileSystem, Schema } from "effect" +import { sql } from "drizzle-orm" +import { layer } from "#sqlite" + +const makeDb = EffectDrizzleSqlite.makeWithDefaults() + +export interface ConvertResult { + /** Readback of `PRAGMA auto_vacuum` after conversion: 1 == FULL. */ + readonly autoVacuum: number +} + +export class Refused extends Schema.TaggedErrorClass()("VacuumRefused", { + filename: Schema.String, + reason: Schema.String, +}) { + override get message() { + return `refusing to vacuum ${this.filename}: ${this.reason}` + } +} + +export class NotFull extends Schema.TaggedErrorClass()("VacuumNotFull", { + filename: Schema.String, + // -1 encodes an unreadable readback (the pragma returned no row). + autoVacuum: Schema.Number, +}) { + override get message() { + return `vacuum did not take full effect for ${this.filename}: PRAGMA auto_vacuum reads back ${this.autoVacuum}, expected 1 (FULL) — close running opencode processes that use this file and retry` + } +} + +/** + * #524 Phase 2 gate and the only success path of `convertToFull`: after the + * FULL -> VACUUM -> wal_checkpoint(TRUNCATE) sequence the readback must be + * exactly FULL (1), otherwise the conversion silently failed (e.g. a writer + * kept the file alive through VACUUM) and must surface as a nonzero failure + * with actionable diagnostics — never as a success result. Exported as the + * deterministic seam that lets tests prove a non-FULL readback cannot report + * success. + */ +export const verifyFull = (filename: string, readback: number | undefined): Effect.Effect => + readback === 1 + ? Effect.succeed({ autoVacuum: readback }) + : Effect.fail(new NotFull({ filename, autoVacuum: readback ?? -1 })) + +// #524: refuse every target that is not an existing regular file BEFORE any +// SQLite open — the driver opens with create enabled, so a typo'd path would +// otherwise silently materialize a fresh empty database. +const validateTarget = Effect.fn("Vacuum.validateTarget")(function* (fs: FileSystem.FileSystem, filename: string) { + if (filename === ":memory:") yield* new Refused({ filename, reason: ":memory: is not a file on disk" }) + const info = yield* fs.stat(filename).pipe(Effect.catch(() => Effect.void)) + if (info === undefined) { + yield* new Refused({ + filename, + reason: "no such file — vacuum never creates a database (print the default path with `opencode db path`)", + }) + return + } + if (info.type !== "File") yield* new Refused({ filename, reason: `not a regular file (${info.type})` }) +}) + +const convert = (filename: string) => + Effect.gen(function* () { + const db = yield* makeDb + yield* db.run(sql`PRAGMA busy_timeout = 5000`) + yield* db.run(sql`PRAGMA auto_vacuum = FULL`) + yield* db.run(sql`VACUUM`) + yield* db.run(sql`PRAGMA wal_checkpoint(TRUNCATE)`) + const mode = yield* db.get<{ auto_vacuum: number }>(sql`PRAGMA auto_vacuum`) + return mode?.auto_vacuum + }).pipe(Effect.provide(layer({ filename }))) + +/** + * #524 Phase 2: explicit, user-triggered conversion of a legacy + * auto_vacuum=NONE database to FULL. Runs OUTSIDE startup and never touches a + * default database path implicitly — the caller names the file (the CLI + * surface requires an explicit `--db`, so tests only ever pass disposable + * temp paths), and the target must already exist as a regular file: vacuum + * never creates a database. The FULL → VACUUM → wal_checkpoint(TRUNCATE) + * sequence rebuilds the database with FULL enabled and truncates the WAL; a + * concurrent writer makes VACUUM fail with SQLITE_BUSY instead of corrupting + * anything, and a non-FULL readback fails via `verifyFull`. Incremental + * auto-vacuum is deliberately never used anywhere. + */ +export const convertToFull = (filename: string) => + Effect.gen(function* () { + const fs = yield* FileSystem.FileSystem + yield* validateTarget(fs, filename) + const readback = yield* convert(filename) + return yield* verifyFull(filename, readback) + }).pipe(Effect.provide(NodeFileSystem.layer)) diff --git a/packages/core/src/event.ts b/packages/core/src/event.ts index ec80977a89..d5ee2e7757 100644 --- a/packages/core/src/event.ts +++ b/packages/core/src/event.ts @@ -3,13 +3,14 @@ export * as EventV2 from "./event" import { Cause, Context, Effect, FiberSet, Layer, Option, PubSub, Schema, Stream } from "effect" import { Event } from "@opencode-ai/schema/event" import type { Data, Definition, Payload } from "@opencode-ai/schema/event" -import { and, asc, eq, gt } from "drizzle-orm" +import { and, asc, desc, eq, gt } from "drizzle-orm" import { Database } from "./database/database" import { EventSequenceTable, EventTable } from "./event/sql" import { Location } from "./location" import { LayerNode } from "./effect/layer-node" import { isDeepStrictEqual } from "node:util" import { Durable } from "@opencode-ai/schema/durable-event-manifest" +import { Hash } from "./util/hash" export const ID = Event.ID export type ID = import("@opencode-ai/schema/event").ID @@ -227,6 +228,30 @@ export const layerWith = (options?: LayerOptions) => if (input && row?.ownerID && row.ownerID !== input.ownerID) { return undefined } + const dataHash = Hash.sha256(JSON.stringify(encoded)) + if (!input) { + // Idempotency gate (#523): a fresh append that byte-for-byte repeats + // this aggregate's latest same-type event carries zero information + // delta. Skip it entirely — no seq consumed, no projectors, no commit + // hook, no durable wake — so the persisted sequence stays dense and + // both the replayAll contiguity check and gt(seq, after) readers are + // unaffected. Replay appends (input) keep their exact-seq contract, + // and legacy rows carry a NULL hash so they never match. + const previous = yield* db + .select({ dataHash: EventTable.data_hash }) + .from(EventTable) + .where( + and( + eq(EventTable.aggregate_id, aggregateID), + eq(EventTable.type, versionedType(definition.type, durable.version)), + ), + ) + .orderBy(desc(EventTable.seq)) + .limit(1) + .get() + .pipe(Effect.orDie) + if (previous && previous.dataHash === dataHash) return undefined + } const seq = input?.seq ?? latest + 1 if (input && seq !== latest + 1) { yield* Effect.die( @@ -278,6 +303,7 @@ export const layerWith = (options?: LayerOptions) => seq, type: versionedType(definition.type, durable.version), data: encoded, + data_hash: dataHash, }, ]) .run() @@ -479,7 +505,9 @@ export const layerWith = (options?: LayerOptions) => .transaction( () => Effect.gen(function* () { - const results = new Array<{ aggregateID: string; seq: number }>() + // Aligned with entries by index: a deduped entry yields + // undefined so the payload pairing below stays positional. + const results = new Array<{ aggregateID: string; seq: number } | undefined>() for (const entry of entries) { // No replay input: seq is allocated contiguously from the latest sequence inside the transaction. const result = yield* commitDurableEventInner( @@ -488,7 +516,7 @@ export const layerWith = (options?: LayerOptions) => undefined, entry.commit, ) - if (result) results.push(result) + results.push(result) } return results }), @@ -499,19 +527,19 @@ export const layerWith = (options?: LayerOptions) => return results }), ) - const payloads = entries.flatMap((entry, index) => { + const payloads = entries.map((entry, index) => { const result = committed[index] - if (!result) return [] - return [ - { - ...entry.event, - durable: { - aggregateID: result.aggregateID, - seq: result.seq, - version: entry.durable.version, - }, - } as Payload, - ] + // A deduped entry is still notified (mirrors the single-publish + // path) but stays unstamped: it occupies no sequence position. + if (!result) return entry.event + return { + ...entry.event, + durable: { + aggregateID: result.aggregateID, + seq: result.seq, + version: entry.durable.version, + }, + } as Payload }) for (const payload of payloads) { yield* notify(payload) diff --git a/packages/core/src/event/residue-sweep.ts b/packages/core/src/event/residue-sweep.ts new file mode 100644 index 0000000000..f16b87e8bc --- /dev/null +++ b/packages/core/src/event/residue-sweep.ts @@ -0,0 +1,129 @@ +// SPDX-FileCopyrightText: 2026 LeXwDeX +// SPDX-License-Identifier: AGPL-3.0-or-later + +export * as EventResidueSweep from "./residue-sweep" + +import { Cause, Context, Effect, Layer, Scope } from "effect" +import { sql } from "drizzle-orm" +import { Database } from "../database/database" +import { LayerNode } from "../effect/layer-node" + +/** + * #524 Phase 1: default-on residue sweep for crash/in-flight zombies. + * + * Session.remove scrubs its session aggregate and every related dag aggregate + * (terminal workflows included), but a crash between the session-row delete + * and the scrub — or a project cascade that removes read-model rows without + * any remove call — leaves durable event aggregates whose SessionTable and + * WorkflowTable read models are BOTH gone. Read models commit atomically with + * an aggregate's first event (the projectors run inside the publish + * transaction), so "events visible, both read models absent" is exactly the + * zombie shape. Live sessions, archived sessions (they keep their row), and + * live workflows never match the predicate. + * + * Removal is a single atomic guarded DELETE (see `removeResidue`): the + * both-read-models-absent guard is re-evaluated inside the delete statement + * itself, so a read model a concurrent replay/publish recreates after + * candidate selection survives — there is no select-then-remove TOCTOU + * window. Soft-degrading: a failed residue read or a failed per-aggregate + * removal is logged and left for a later pass — the sweep never fails the + * application path. The Durable manifest only contains session- and + * dag-family events, so every aggregate id in event_sequence is keyed by one + * of the two checked read models. + */ + +export interface Interface { + /** One sweep pass. Returns the number of residue aggregates removed. */ + readonly sweepOnce: () => Effect.Effect +} + +export class Service extends Context.Service()("@opencode/v2/EventResidueSweep") {} + +/** + * Selection half of the sweep pass. Exported as the seam that lets tests + * deterministically interleave a concurrent read-model recreation between + * candidate selection and deletion (no sleeps). + */ +export const selectResidues = (db: Database.Interface["db"]) => + db.all<{ aggregate_id: string }>(sql` + SELECT aggregate_id FROM event_sequence + WHERE NOT EXISTS (SELECT 1 FROM session WHERE session.id = event_sequence.aggregate_id) + AND NOT EXISTS (SELECT 1 FROM workflow WHERE workflow.id = event_sequence.aggregate_id) + `) + +/** + * Removal half of the sweep pass — the atomic guarded delete. The NOT EXISTS + * guards are re-evaluated inside the DELETE statement itself, so a read model + * recreated between candidate selection and this statement survives; the + * aggregate is only deleted while it is still a zombie. PRAGMA foreign_keys + * is ON on the Database layer's connection, so the delete cascades to the + * aggregate's event rows, and RETURNING makes the removed result reliable + * instead of inferred. Exported for the same test seam as `selectResidues`. + */ +export const removeResidue = (db: Database.Interface["db"], aggregateID: string) => + db + .all<{ aggregate_id: string }>(sql` + DELETE FROM event_sequence + WHERE aggregate_id = ${aggregateID} + AND NOT EXISTS (SELECT 1 FROM session WHERE session.id = event_sequence.aggregate_id) + AND NOT EXISTS (SELECT 1 FROM workflow WHERE workflow.id = event_sequence.aggregate_id) + RETURNING aggregate_id + `) + .pipe(Effect.map((rows) => rows.length > 0)) + +const serviceLayer = Layer.effect( + Service, + Effect.gen(function* () { + const { db } = yield* Database.Service + const scope = yield* Scope.Scope + + const sweepOnce = Effect.fn("EventResidueSweep.sweepOnce")(function* () { + const residues = yield* selectResidues(db).pipe( + Effect.catchCause((cause) => + Cause.hasInterrupts(cause) + ? Effect.interrupt + : Effect.gen(function* () { + yield* Effect.logWarning("EventResidueSweep residue query failed — skipping pass", { cause }) + return [] as Array<{ aggregate_id: string }> + }), + ), + ) + + let removed = 0 + for (const residue of residues) { + const done = yield* removeResidue(db, residue.aggregate_id).pipe( + Effect.catchCause((cause) => + Cause.hasInterrupts(cause) + ? Effect.interrupt + : Effect.gen(function* () { + yield* Effect.logWarning("EventResidueSweep failed to remove a residue aggregate — left for a later pass", { + aggregateID: residue.aggregate_id, + cause, + }) + return false + }), + ), + ) + if (done) removed++ + } + if (removed > 0) yield* Effect.logInfo("EventResidueSweep removed orphaned event aggregates", { removed }) + return removed + }) + + // Default-on: one pass per process start, forked into the layer scope so + // it can neither block nor fail startup (the AGENTS.md background-loop + // convention — no caller has to remember to init it). + yield* sweepOnce().pipe( + Effect.catchCause((cause) => + Cause.hasInterrupts(cause) ? Effect.interrupt : Effect.logWarning("EventResidueSweep startup pass failed", { cause }), + ), + Effect.forkIn(scope), + ) + + return Service.of({ sweepOnce }) + }), +) + +export const defaultLayer = serviceLayer.pipe(Layer.provide(Database.defaultLayer)) + +export const node = LayerNode.make(serviceLayer, [Database.node]) diff --git a/packages/core/src/event/sql.ts b/packages/core/src/event/sql.ts index 38fe34f1e3..74f868e971 100644 --- a/packages/core/src/event/sql.ts +++ b/packages/core/src/event/sql.ts @@ -17,6 +17,11 @@ export const EventTable = sqliteTable( seq: integer().notNull(), type: text().notNull(), data: text({ mode: "json" }).$type>().notNull(), + // sha256 of the serialized payload, written once at append time. The + // idempotency gate compares against the latest same-type row via + // event_aggregate_type_seq_idx instead of re-hashing MiB-scale payloads. + // Nullable: legacy rows predate the column and never match the gate. + data_hash: text(), }, (table) => [ uniqueIndex("event_aggregate_seq_idx").on(table.aggregate_id, table.seq), diff --git a/packages/core/src/npm.ts b/packages/core/src/npm.ts index b5a59f2dcb..f567339f15 100644 --- a/packages/core/src/npm.ts +++ b/packages/core/src/npm.ts @@ -217,7 +217,12 @@ export const layer = Layer.effect( remaining.push(pkg) } - if (remaining.length !== requested.length) { + // Only a mixed batch (part of the deps pinned locally/bundled, part still + // going to the registry) needs the lock dropped so the registry deps are + // re-resolved. An all-local/all-bundled batch must leave a valid lock + // untouched, or every startup deletes and rebuilds it; an all-registry + // batch never deleted it either. + if (remaining.length > 0 && remaining.length < requested.length) { yield* fs.remove(path.join(dir, "package-lock.json")).pipe(Effect.orElseSucceed(() => undefined)) } diff --git a/packages/core/src/session/projector.ts b/packages/core/src/session/projector.ts index 0064fcb1d6..c47cbce2ec 100644 --- a/packages/core/src/session/projector.ts +++ b/packages/core/src/session/projector.ts @@ -60,7 +60,6 @@ function sessionRow(info: SessionV1.SessionInfo): typeof SessionTable.$inferInse summary_additions: info.summary?.additions, summary_deletions: info.summary?.deletions, summary_files: info.summary?.files, - summary_diffs: info.summary?.diffs ? [...info.summary.diffs] : undefined, metadata: info.metadata, cost: info.cost ?? 0, tokens_input: (info.tokens ?? { input: 0 }).input, diff --git a/packages/core/src/session/sql.ts b/packages/core/src/session/sql.ts index a7ce8df496..abdd603601 100644 --- a/packages/core/src/session/sql.ts +++ b/packages/core/src/session/sql.ts @@ -4,7 +4,6 @@ import { ProjectTable } from "../project/sql" import type { SessionMessage } from "./message" import type { Prompt } from "./prompt" import type { SessionInput } from "./input" -import type { Snapshot } from "../snapshot" import { PermissionV1 } from "../v1/permission" import { ProjectV2 } from "../project" import type { SessionSchema } from "./schema" @@ -38,7 +37,6 @@ export const SessionTable = sqliteTable( summary_additions: integer(), summary_deletions: integer(), summary_files: integer(), - summary_diffs: text({ mode: "json" }).$type(), metadata: text({ mode: "json" }).$type>(), cost: real().notNull().default(0), tokens_input: integer().notNull().default(0), diff --git a/packages/core/test/database-migration.test.ts b/packages/core/test/database-migration.test.ts index 339417fbc8..87da1346bc 100644 --- a/packages/core/test/database-migration.test.ts +++ b/packages/core/test/database-migration.test.ts @@ -18,6 +18,7 @@ import simplifySessionInputMigration from "@opencode-ai/core/database/migration/ import capturedOutputMigration from "@opencode-ai/core/database/migration/20260715035022_captured_output" import fearlessCammiMigration from "@opencode-ai/core/database/migration/20260717034735_fearless_cammi" import dagWorkflowNodeIdentityMigration from "@opencode-ai/core/database/migration/20260720013828_dag-workflow-node-identity" +import dropSessionSummaryDiffsMigration from "@opencode-ai/core/database/migration/20260903044702_drop_session_summary_diffs" import { EventV2 } from "@opencode-ai/core/event" import { ProjectV2 } from "@opencode-ai/core/project" import { ProjectTable } from "@opencode-ai/core/project/sql" @@ -100,6 +101,39 @@ describe("DatabaseMigration", () => { ) }) + test("drops the legacy session summary_diffs column", async () => { + await run( + Effect.gen(function* () { + const db = yield* makeDb + // Legacy install: the column still exists with data, and only the drop migration is pending. + yield* db.run(sql`CREATE TABLE session (id text PRIMARY KEY, summary_diffs text)`) + yield* db.run( + sql`INSERT INTO session (id, summary_diffs) VALUES ('ses_legacy', '[{"file":"a.txt","patch":"p","additions":1,"deletions":0,"status":"modified"}]')`, + ) + + yield* DatabaseMigration.applyOnly(db, [dropSessionSummaryDiffsMigration]) + + expect( + yield* db.get(sql`SELECT name FROM pragma_table_info('session') WHERE name = 'summary_diffs'`), + ).toBeUndefined() + expect(yield* db.get(sql`SELECT id FROM session WHERE id = 'ses_legacy'`)).toEqual({ id: "ses_legacy" }) + expect(yield* db.get(sql`SELECT id FROM migration WHERE id = ${dropSessionSummaryDiffsMigration.id}`)).toEqual({ + id: dropSessionSummaryDiffsMigration.id, + }) + }), + ) + + await run( + Effect.gen(function* () { + const db = yield* makeDb + yield* DatabaseMigration.apply(db) + expect( + yield* db.get(sql`SELECT name FROM pragma_table_info('session') WHERE name = 'summary_diffs'`), + ).toBeUndefined() + }), + ) + }) + test("upgrades DAG node storage without duplicate columns or cross-workflow collisions", async () => { await run( Effect.gen(function* () { diff --git a/packages/core/test/database-vacuum.test.ts b/packages/core/test/database-vacuum.test.ts new file mode 100644 index 0000000000..fa6604b24d --- /dev/null +++ b/packages/core/test/database-vacuum.test.ts @@ -0,0 +1,244 @@ +import { describe, expect, test } from "bun:test" +import { Database as BunSqlite } from "bun:sqlite" +import { EffectDrizzleSqlite } from "@opencode-ai/effect-drizzle-sqlite" +import { SqliteClient } from "@effect/sql-sqlite-bun" +import { Cause, Effect, Exit, Layer } from "effect" +import { SqlClient } from "effect/unstable/sql/SqlClient" +import { classifySqliteError, SqlError } from "effect/unstable/sql/SqlError" +import { existsSync } from "fs" +import { sql } from "drizzle-orm" +import path from "path" +import { Database } from "@opencode-ai/core/database/database" +import { DatabaseMigration } from "@opencode-ai/core/database/migration" +import { Vacuum } from "@opencode-ai/core/database/vacuum" +import { ProjectV2 } from "@opencode-ai/core/project" +import { ProjectTable } from "@opencode-ai/core/project/sql" +import { SessionSchema } from "@opencode-ai/core/session/schema" +import { SessionTable } from "@opencode-ai/core/session/sql" +import { AbsolutePath } from "@opencode-ai/core/schema" +import { layer as repoSqliteLayer } from "#sqlite" +import { tmpdir } from "./fixture/tmpdir" + +const makeDb = EffectDrizzleSqlite.makeWithDefaults() + +// Seeds an application-created LEGACY database shape: WAL initialized, +// auto_vacuum left at its NONE default, real migrations applied, user rows +// present. Disposable temp files only — never a real opencode.db path. +const seedLegacyDatabase = (filename: string) => + Effect.runPromise( + Effect.gen(function* () { + const db = yield* makeDb + yield* db.run(sql`PRAGMA journal_mode = WAL`) + yield* DatabaseMigration.apply(db) + yield* db + .insert(ProjectTable) + .values({ id: ProjectV2.ID.make("proj_legacy"), worktree: AbsolutePath.make("/legacy"), sandboxes: [] }) + .run() + .pipe(Effect.orDie) + yield* db + .insert(SessionTable) + .values({ + id: SessionSchema.ID.make("ses_legacy"), + project_id: ProjectV2.ID.make("proj_legacy"), + slug: "legacy", + directory: "/legacy", + title: "legacy", + version: "test", + }) + .run() + .pipe(Effect.orDie) + }).pipe(Effect.provide(SqliteClient.layer({ filename })), Effect.scoped), + ) + +const readFileMode = (filename: string) => { + const native = new BunSqlite(filename, { readonly: true, create: false }) + try { + const autoVacuum = native.query<{ auto_vacuum: number }, []>("PRAGMA auto_vacuum").get() + const freelist = native.query<{ freelist_count: number }, []>("PRAGMA freelist_count").get() + const integrity = native.query<{ integrity_check: string }, []>("PRAGMA integrity_check").get() + const rows = native.query<{ count: number }, []>("SELECT COUNT(*) AS count FROM session").get() + return { + autoVacuum: autoVacuum?.auto_vacuum ?? -1, + freelist: freelist?.freelist_count ?? -1, + integrity: integrity?.integrity_check ?? "unknown", + sessionRows: rows?.count ?? -1, + } + } finally { + native.close() + } +} + +describe("Database auto_vacuum (#524 Phase 2)", () => { + test("initializes genuinely new databases with auto_vacuum=FULL before WAL", async () => { + await using tmp = await tmpdir() + const filename = path.join(tmp.path, "new.sqlite") + await Effect.runPromise( + Effect.gen(function* () { + const { db } = yield* Database.Service + + const mode = yield* db.get<{ auto_vacuum: number }>(sql`PRAGMA auto_vacuum`).pipe(Effect.orDie) + expect(mode?.auto_vacuum).toBe(1) + const journal = yield* db.get<{ journal_mode: string }>(sql`PRAGMA journal_mode`).pipe(Effect.orDie) + expect(String(journal?.journal_mode).toLowerCase()).toBe("wal") + + // The real application layer (migrations included) preserves the mode. + yield* db + .insert(ProjectTable) + .values({ id: ProjectV2.ID.make("proj_new"), worktree: AbsolutePath.make("/new"), sandboxes: [] }) + .run() + .pipe(Effect.orDie) + const after = yield* db.get<{ auto_vacuum: number }>(sql`PRAGMA auto_vacuum`).pipe(Effect.orDie) + expect(after?.auto_vacuum).toBe(1) + }).pipe(Effect.provide(Database.layerFromPath(filename))), + ) + }) + + test("never converts an existing auto_vacuum=NONE database at startup", async () => { + await using tmp = await tmpdir() + const filename = path.join(tmp.path, "legacy.sqlite") + await seedLegacyDatabase(filename) + expect(readFileMode(filename).autoVacuum).toBe(0) + + // The production startup sequence (driver pragmas + migrations) must be a + // silent no-op for the legacy mode — converting without the explicit + // user-triggered command is forbidden. + await Effect.runPromise( + Effect.gen(function* () { + const { db } = yield* Database.Service + const mode = yield* db.get<{ auto_vacuum: number }>(sql`PRAGMA auto_vacuum`).pipe(Effect.orDie) + expect(mode?.auto_vacuum).toBe(0) + expect( + (yield* db.get<{ count: number }>(sql`SELECT COUNT(*) AS count FROM session`).pipe(Effect.orDie))?.count, + ).toBe(1) + }).pipe(Effect.provide(Database.layerFromPath(filename))), + ) + const after = readFileMode(filename) + expect(after.autoVacuum).toBe(0) + expect(after.sessionRows).toBe(1) + expect(after.integrity).toBe("ok") + }) + + test("explicit conversion runs FULL -> VACUUM -> wal TRUNCATE with data intact", async () => { + await using tmp = await tmpdir() + const filename = path.join(tmp.path, "legacy-convert.sqlite") + await seedLegacyDatabase(filename) + expect(readFileMode(filename).autoVacuum).toBe(0) + + const result = await Effect.runPromise(Vacuum.convertToFull(filename)) + expect(result.autoVacuum).toBe(1) + + const after = readFileMode(filename) + expect(after.autoVacuum).toBe(1) + expect(after.freelist).toBe(0) + expect(after.sessionRows).toBe(1) + expect(after.integrity).toBe("ok") + }) + + test("refuses a nonexistent target before opening SQLite and never creates it", async () => { + await using tmp = await tmpdir() + const filename = path.join(tmp.path, "typo.sqlite") + + const exit = await Effect.runPromiseExit(Vacuum.convertToFull(filename)) + expect(Exit.isFailure(exit)).toBe(true) + if (Exit.isFailure(exit)) { + const rendered = Cause.pretty(exit.cause) + expect(rendered).toContain("refusing to vacuum") + expect(rendered).toContain("no such file") + } + // The typo must not have materialized a database (nor WAL/SHM siblings). + expect(existsSync(filename)).toBe(false) + expect(existsSync(`${filename}-wal`)).toBe(false) + expect(existsSync(`${filename}-shm`)).toBe(false) + }) + + test("refuses :memory: and non-file targets", async () => { + const memory = await Effect.runPromiseExit(Vacuum.convertToFull(":memory:")) + expect(Exit.isFailure(memory)).toBe(true) + if (Exit.isFailure(memory)) expect(Cause.pretty(memory.cause)).toContain("not a file on disk") + + await using tmp = await tmpdir() + const directory = await Effect.runPromiseExit(Vacuum.convertToFull(tmp.path)) + expect(Exit.isFailure(directory)).toBe(true) + if (Exit.isFailure(directory)) expect(Cause.pretty(directory.cause)).toContain("not a regular file") + expect(existsSync(tmp.path)).toBe(true) + }) + + // Deterministic proof that a non-FULL readback can never report success: + // `verifyFull` is the only success path of `convertToFull`. + test("a non-FULL readback fails with actionable diagnostics via verifyFull", async () => { + const zero = await Effect.runPromiseExit(Vacuum.verifyFull("stuck.sqlite", 0)) + expect(Exit.isFailure(zero)).toBe(true) + if (Exit.isFailure(zero)) { + const rendered = Cause.pretty(zero.cause) + expect(rendered).toContain("VacuumNotFull") + expect(rendered).toContain("stuck.sqlite") + expect(rendered).toContain("expected 1 (FULL)") + expect(rendered).toContain("retry") + } + + const unreadable = await Effect.runPromiseExit(Vacuum.verifyFull("stuck.sqlite", undefined)) + expect(Exit.isFailure(unreadable)).toBe(true) + + const ok = await Effect.runPromise(Vacuum.verifyFull("converted.sqlite", 1)) + expect(ok.autoVacuum).toBe(1) + }) + + // Layer/failure regression: a failed auto_vacuum readback must soft-degrade + // with a warning — the startup layer must not die (its body is orDie'd), so + // migrations still apply and the service stays usable. + test("a failed auto_vacuum readback soft-degrades instead of killing startup", async () => { + await using tmp = await tmpdir() + const filename = path.join(tmp.path, "readback-failure.sqlite") + + // Real repository sqlite client stack (#sqlite = the production driver), + // except every auto_vacuum statement fails at the Database.layer level. + const failingReadbackLayer = Layer.effect( + SqlClient, + Effect.gen(function* () { + const client = yield* SqlClient + const failure = new SqlError({ + reason: classifySqliteError(new Error("simulated auto_vacuum readback failure"), { + message: "Failed to execute statement", + operation: "execute", + }), + }) + // oxlint-disable-next-line typescript-eslint/no-unsafe-type-assertion -- test decorator over the real client, shape-preserving at runtime + return Object.assign({}, client, { + unsafe: (query: string, params?: ReadonlyArray) => { + const statement = client.unsafe(query, params) + if (!query.toLowerCase().includes("auto_vacuum")) return statement + return Object.assign({}, statement, { + withoutTransform: Effect.fail(failure), + values: Effect.fail(failure), + }) + }, + }) as SqlClient + }), + ).pipe(Layer.provide(repoSqliteLayer({ filename }))) + + await Effect.runPromise( + Effect.gen(function* () { + const { db } = yield* Database.Service + // Startup ran past the failed readback: migrations were applied and + // the service is usable. + const tables = yield* db + .get<{ count: number }>(sql`SELECT COUNT(*) AS count FROM sqlite_master WHERE type = 'table' AND name = 'session'`) + .pipe(Effect.orDie) + expect(tables?.count).toBe(1) + }).pipe(Effect.provide(Database.layer.pipe(Layer.provide(failingReadbackLayer)))), + ) + // The database file itself was created and initialized normally. + expect(readFileMode(filename).autoVacuum).toBe(1) + }) + + test("incremental_vacuum never appears in executable database code", async () => { + const databaseDir = path.join(import.meta.dir, "..", "src", "database") + const glob = new Bun.Glob("**/*.ts") + const offenders: string[] = [] + for await (const file of glob.scan({ cwd: databaseDir })) { + const content = await Bun.file(path.join(databaseDir, file)).text() + if (/incremental_vacuum/i.test(content)) offenders.push(file) + } + expect(offenders).toEqual([]) + }) +}) diff --git a/packages/core/test/event-residue-sweep.test.ts b/packages/core/test/event-residue-sweep.test.ts new file mode 100644 index 0000000000..63e400d4d4 --- /dev/null +++ b/packages/core/test/event-residue-sweep.test.ts @@ -0,0 +1,156 @@ +import { describe, expect, test } from "bun:test" +import { Effect, Layer } from "effect" +import { eq, inArray } from "drizzle-orm" +import { Database } from "@opencode-ai/core/database/database" +import { EventSequenceTable, EventTable } from "@opencode-ai/core/event/sql" +import { EventV2 } from "@opencode-ai/core/event" +import { EventResidueSweep } from "@opencode-ai/core/event/residue-sweep" +import { SessionSchema } from "@opencode-ai/core/session/schema" +import { SessionTable } from "@opencode-ai/core/session/sql" +import { WorkflowTable } from "@opencode-ai/core/dag/sql" +import { ProjectTable } from "@opencode-ai/core/project/sql" +import { ProjectV2 } from "@opencode-ai/core/project" +import { AbsolutePath } from "@opencode-ai/core/schema" + +// #524 Phase 1: crash/in-flight zombie residue — a Session.remove (or a project +// cascade) that crashed between the session-row delete and the event-store +// scrub leaves durable event aggregates whose SessionTable and WorkflowTable +// read models are both gone. The default-on residue sweep removes exactly +// those aggregates and never touches live or archived ones. +const testLayer = Layer.mergeAll(Database.defaultLayer, EventResidueSweep.defaultLayer) + +const seedAggregate = (aggregateID: string) => + Effect.gen(function* () { + const { db } = yield* Database.Service + yield* db.insert(EventSequenceTable).values({ aggregate_id: aggregateID, seq: 1 }).run().pipe(Effect.orDie) + yield* db + .insert(EventTable) + .values({ id: EventV2.ID.make(`evt_${aggregateID}`), aggregate_id: aggregateID, seq: 1, type: "session.updated.1", data: {} }) + .run() + .pipe(Effect.orDie) + }) + +const seedProject = Effect.gen(function* () { + const { db } = yield* Database.Service + yield* db + .insert(ProjectTable) + .values({ id: ProjectV2.ID.make("proj_sweep"), worktree: AbsolutePath.make("/project"), sandboxes: [] }) + .run() + .pipe(Effect.orDie) +}) + +const remainingAggregates = (ids: readonly string[]) => + Effect.gen(function* () { + const { db } = yield* Database.Service + return yield* db + .select({ aggregate: EventSequenceTable.aggregate_id }) + .from(EventSequenceTable) + .where(inArray(EventSequenceTable.aggregate_id, [...ids])) + .all() + .pipe(Effect.orDie) + }) + +describe("EventResidueSweep (#524)", () => { + test("removes only aggregates whose session and workflow read models are both absent", async () => { + await Effect.runPromise( + Effect.gen(function* () { + const { db } = yield* Database.Service + const sweep = yield* EventResidueSweep.Service + + yield* db + .insert(ProjectTable) + .values({ id: ProjectV2.ID.make("proj_sweep"), worktree: AbsolutePath.make("/project"), sandboxes: [] }) + .run() + .pipe(Effect.orDie) + yield* db + .insert(SessionTable) + .values({ id: SessionSchema.ID.make("ses_live"), project_id: ProjectV2.ID.make("proj_sweep"), slug: "live", directory: "/project", title: "live", version: "test" }) + .run() + .pipe(Effect.orDie) + // Archived sessions keep their read-model row — never eligible. + yield* db + .insert(SessionTable) + .values({ id: SessionSchema.ID.make("ses_archived"), project_id: ProjectV2.ID.make("proj_sweep"), slug: "archived", directory: "/project", title: "archived", version: "test", time_archived: 123 }) + .run() + .pipe(Effect.orDie) + yield* db + .insert(WorkflowTable) + .values({ id: "dag_live", project_id: ProjectV2.ID.make("proj_sweep"), session_id: "ses_live", title: "live", status: "running", config: "{}", seq: 0 }) + .run() + .pipe(Effect.orDie) + + yield* seedAggregate("ses_live") + yield* seedAggregate("ses_archived") + yield* seedAggregate("dag_live") + yield* seedAggregate("ses_zombie") + yield* seedAggregate("dag_zombie") + + const removed = yield* sweep.sweepOnce() + expect(removed).toBe(2) + + const survivors = yield* remainingAggregates(["ses_live", "ses_archived", "dag_live", "ses_zombie", "dag_zombie"]) + expect(survivors.map((row) => row.aggregate).sort()).toEqual(["dag_live", "ses_archived", "ses_live"]) + // Read models of live/archived aggregates are untouched. + expect((yield* db.select().from(SessionTable).where(eq(SessionTable.id, SessionSchema.ID.make("ses_live"))).all().pipe(Effect.orDie)).length).toBe(1) + expect((yield* db.select().from(SessionTable).where(eq(SessionTable.id, SessionSchema.ID.make("ses_archived"))).all().pipe(Effect.orDie)).length).toBe(1) + expect((yield* db.select().from(WorkflowTable).where(eq(WorkflowTable.id, "dag_live")).all().pipe(Effect.orDie)).length).toBe(1) + }).pipe(Effect.provide(testLayer)), + ) + }) + + test("a repeated pass finds nothing to remove", async () => { + await Effect.runPromise( + Effect.gen(function* () { + const sweep = yield* EventResidueSweep.Service + expect(yield* sweep.sweepOnce()).toBe(0) + }).pipe(Effect.provide(testLayer)), + ) + }) + + // Deterministic TOCTOU regression: a read model recreated between candidate + // selection and deletion (the concurrent replay/publish race) survives the + // guarded delete. Uses the sweep's own select/remove seam instead of sleeps. + test("a read model recreated after candidate selection survives the guarded delete", async () => { + await Effect.runPromise( + Effect.gen(function* () { + yield* seedProject + const { db } = yield* Database.Service + + yield* seedAggregate("ses_zombie") + yield* seedAggregate("dag_zombie") + + const candidates = yield* EventResidueSweep.selectResidues(db).pipe(Effect.orDie) + expect(candidates.map((row) => row.aggregate_id).sort()).toEqual(["dag_zombie", "ses_zombie"]) + + // Concurrent replay/publish lands here: the session read model is + // re-materialized after selection, before deletion. + yield* db + .insert(SessionTable) + .values({ + id: SessionSchema.ID.make("ses_zombie"), + project_id: ProjectV2.ID.make("proj_sweep"), + slug: "reanimated", + directory: "/project", + title: "reanimated", + version: "test", + }) + .run() + .pipe(Effect.orDie) + + expect(yield* EventResidueSweep.removeResidue(db, "ses_zombie").pipe(Effect.orDie)).toBe(false) + // The still-zombie aggregate is removed, its event rows cascading with it. + expect(yield* EventResidueSweep.removeResidue(db, "dag_zombie").pipe(Effect.orDie)).toBe(true) + + const survivors = yield* remainingAggregates(["ses_zombie", "dag_zombie"]) + expect(survivors.map((row) => row.aggregate)).toEqual(["ses_zombie"]) + expect( + (yield* db.select({ id: EventTable.id }).from(EventTable).where(eq(EventTable.aggregate_id, "ses_zombie")).all().pipe(Effect.orDie)) + .length, + ).toBe(1) + expect( + yield* db.select({ id: EventTable.id }).from(EventTable).where(eq(EventTable.aggregate_id, "dag_zombie")).all().pipe(Effect.orDie), + ).toEqual([]) + }).pipe(Effect.provide(testLayer)), + ) + }) +}) diff --git a/packages/core/test/event.test.ts b/packages/core/test/event.test.ts index 7034c1cccc..c6722ce2f6 100644 --- a/packages/core/test/event.test.ts +++ b/packages/core/test/event.test.ts @@ -10,7 +10,7 @@ import { EventSequenceTable, EventTable } from "@opencode-ai/core/event/sql" import { Location } from "@opencode-ai/core/location" import { AbsolutePath } from "@opencode-ai/core/schema" import { WorkspaceV2 } from "@opencode-ai/core/workspace" -import { eq } from "drizzle-orm" +import { asc, eq } from "drizzle-orm" import { location } from "./fixture/location" import { testEffect } from "./lib/effect" @@ -382,6 +382,238 @@ describe("EventV2", () => { }), ) + it.effect("skips a byte-identical duplicate of the latest same-type durable event", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const { db } = yield* Database.Service + const aggregateID = EventV2.ID.create() + + const first = yield* events.publish(SyncMessage, { id: aggregateID, text: "hello" }) + const duplicate = yield* events.publish(SyncMessage, { id: aggregateID, text: "hello" }) + const rows = yield* db + .select({ seq: EventTable.seq, dataHash: EventTable.data_hash }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, aggregateID)) + .all() + .pipe(Effect.orDie) + + expect(first.durable?.seq).toBe(0) + expect(duplicate.durable).toBeUndefined() + expect(rows).toHaveLength(1) + expect(rows[0]?.dataHash).toHaveLength(64) + }), + ) + + it.effect("keeps the persisted sequence dense when a duplicate is skipped", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const { db } = yield* Database.Service + const aggregateID = EventV2.ID.create() + + yield* events.publish(SyncMessage, { id: aggregateID, text: "hello" }) + yield* events.publish(SyncMessage, { id: aggregateID, text: "hello" }) + const third = yield* events.publish(SyncMessage, { id: aggregateID, text: "world" }) + const rows = yield* db + .select({ seq: EventTable.seq, data: EventTable.data }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, aggregateID)) + .orderBy(asc(EventTable.seq)) + .all() + .pipe(Effect.orDie) + + expect(third.durable?.seq).toBe(1) + expect(rows.map((row) => [row.seq, row.data["text"]])).toEqual([ + [0, "hello"], + [1, "world"], + ]) + }), + ) + + it.effect("appends when the payload differs from the latest same-type event", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const { db } = yield* Database.Service + const aggregateID = EventV2.ID.create() + + yield* events.publish(SyncMessage, { id: aggregateID, text: "a" }) + yield* events.publish(SyncMessage, { id: aggregateID, text: "b" }) + yield* events.publish(SyncMessage, { id: aggregateID, text: "a" }) + const rows = yield* db + .select({ seq: EventTable.seq }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, aggregateID)) + .all() + .pipe(Effect.orDie) + + expect(rows).toHaveLength(3) + }), + ) + + it.effect("dedupes only against the same aggregate and event type", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const { db } = yield* Database.Service + const aggregateID = EventV2.ID.create() + const otherAggregateID = EventV2.ID.create() + + yield* events.publish(SyncMessage, { id: aggregateID, text: "same" }) + yield* events.publish(SyncSent, { messageID: aggregateID, text: "same" }) + yield* events.publish(SyncMessage, { id: otherAggregateID, text: "same" }) + const own = yield* db + .select({ type: EventTable.type }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, aggregateID)) + .all() + .pipe(Effect.orDie) + const other = yield* db + .select({ seq: EventTable.seq }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, otherAggregateID)) + .all() + .pipe(Effect.orDie) + + expect(new Set(own.map((row) => row.type))).toHaveLength(2) + expect(other).toHaveLength(1) + }), + ) + + it.effect("skips duplicates inside a publishMany batch and keeps payloads aligned", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const { db } = yield* Database.Service + const aggregateID = EventV2.ID.create() + + const payloads = yield* events.publishMany([ + { definition: SyncMessage, data: { id: aggregateID, text: "a" } }, + { definition: SyncMessage, data: { id: aggregateID, text: "a" } }, + { definition: SyncMessage, data: { id: aggregateID, text: "b" } }, + ]) + const rows = yield* db + .select({ seq: EventTable.seq }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, aggregateID)) + .all() + .pipe(Effect.orDie) + + expect(rows).toHaveLength(2) + expect(payloads.map((event) => event.data)).toEqual([ + { id: aggregateID, text: "a" }, + { id: aggregateID, text: "a" }, + { id: aggregateID, text: "b" }, + ]) + expect(payloads.map((event) => event.durable?.seq)).toEqual([0, undefined, 1]) + }), + ) + + it.effect("runs projectors and commit hooks only for persisted appends", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const projected = new Array() + yield* events.project(SyncMessage, (event) => + Effect.sync(() => { + projected.push(event) + }), + ) + const commits = new Array() + const aggregateID = EventV2.ID.create() + const publishWithCommit = () => + events.publish( + SyncMessage, + { id: aggregateID, text: "hello" }, + { commit: (seq) => Effect.sync(() => commits.push(seq)) }, + ) + + yield* publishWithCommit() + yield* publishWithCommit() + + expect(projected.map((event) => event.durable?.seq)).toEqual([0]) + expect(commits).toEqual([0]) + }), + ) + + it.effect("never dedupes against legacy rows with a NULL hash", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const { db } = yield* Database.Service + const aggregateID = EventV2.ID.create() + + yield* db + .insert(EventSequenceTable) + .values([{ aggregate_id: aggregateID, seq: 0 }]) + .run() + .pipe(Effect.orDie) + yield* db + .insert(EventTable) + .values([ + { + id: EventV2.ID.create(), + aggregate_id: aggregateID, + seq: 0, + type: EventV2.versionedType(SyncMessage.type, 1), + data: { id: aggregateID, text: "legacy" }, + }, + ]) + .run() + .pipe(Effect.orDie) + + const published = yield* events.publish(SyncMessage, { id: aggregateID, text: "legacy" }) + const rows = yield* db + .select({ seq: EventTable.seq }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, aggregateID)) + .all() + .pipe(Effect.orDie) + + expect(published.durable?.seq).toBe(1) + expect(rows).toHaveLength(2) + }), + ) + + it.effect("replay with an explicit seq is never deduped", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const { db } = yield* Database.Service + const aggregateID = Session.ID.create() + + yield* events.publish(DurableMessage, durableData(aggregateID, "same")) + yield* events.replay({ + id: EventV2.ID.create(), + type: EventV2.versionedType(DurableMessage.type, 1), + seq: 1, + aggregateID, + data: durableData(aggregateID, "same"), + }) + const rows = yield* db + .select({ seq: EventTable.seq }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, aggregateID)) + .all() + .pipe(Effect.orDie) + + expect(rows).toHaveLength(2) + }), + ) + + it.effect("durable readers observe only persisted events after a skipped duplicate", () => + Effect.gen(function* () { + const events = yield* EventV2.Service + const aggregateID = Session.ID.create() + yield* events.publish(DurableMessage, durableData(aggregateID, "zero")) + const fiber = yield* events + .durable({ aggregateID }) + .pipe(Stream.take(2), Stream.runCollect, Effect.forkScoped) + yield* Effect.yieldNow + + yield* events.publish(DurableMessage, durableData(aggregateID, "zero")) + yield* events.publish(DurableMessage, durableData(aggregateID, "one")) + + expect(Array.from(yield* Fiber.join(fiber)).map((event) => [event.durable?.seq, event.data])).toEqual([ + [0, durableData(aggregateID, "zero")], + [1, durableData(aggregateID, "one")], + ]) + }), + ) + it.effect("replays durable aggregate events after a sequence and tails new events", () => Effect.gen(function* () { const events = yield* EventV2.Service diff --git a/packages/core/test/npm.test.ts b/packages/core/test/npm.test.ts index f66734962e..96e75e6b56 100644 --- a/packages/core/test/npm.test.ts +++ b/packages/core/test/npm.test.ts @@ -10,6 +10,12 @@ import { PluginSdk } from "@opencode-ai/core/plugin-sdk" import { EffectFlock } from "@opencode-ai/core/util/effect-flock" import { tmpdir } from "./fixture/tmpdir" +// CI runners blackhole the registry audit POST that arborist.reify issues, +// hanging these fixtures past Bun's default 5000ms test timeout. Audit is +// incidental to what these tests assert; NpmConfig.load spreads process.env +// into Arborist, so disabling it here keeps reify hermetic. +process.env.npm_config_audit = "false" + const win = process.platform === "win32" const writePackage = (dir: string, pkg: Record) => @@ -65,6 +71,48 @@ describe("Npm.add", () => { }) }) +interface AddSpec { + name: string + version?: string +} + +const lockSnapshot = async (lockPath: string) => { + const [bytes, stat] = await Promise.all([fs.readFile(lockPath), fs.stat(lockPath)]) + return { bytes: bytes.toString(), ino: stat.ino, mtimeMs: stat.mtimeMs } +} + +const install = (dir: string, cache: string, add: AddSpec[] = []) => + Effect.gen(function* () { + const npm = yield* Npm.Service + yield* npm.install(dir, add.length ? { add } : undefined) + }).pipe(Effect.scoped, Effect.provide(npmLayer(cache)), Effect.runPromise) + +// Seeds a project whose plugin dependency resolves from an existing local copy +// (declared via a file: spec so the initial reify stays offline), then builds a +// real package-lock.json through the genuine forcing path: one install with a +// missing file: dependency reaches reify and writes the lock arborist owns. +const seedPluginProject = async (dir: string) => { + const localPlugin = path.join(dir, "local-plugin") + await fs.mkdir(localPlugin, { recursive: true }) + await writePackage(localPlugin, { name: PluginSdk.packageName, main: "index.js" }) + await Bun.write(path.join(localPlugin, "index.js"), "export const plugin = true\n") + + const helper = path.join(dir, "helper-dep") + await fs.mkdir(helper, { recursive: true }) + await writePackage(helper, { name: "fixture-helper-dep", main: "index.js" }) + await Bun.write(path.join(helper, "index.js"), "export const helper = true\n") + + await writePackage(dir, { + name: "fixture", + dependencies: { + [PluginSdk.packageName]: "file:./local-plugin", + "fixture-helper-dep": "file:./helper-dep", + }, + }) + + await install(dir, path.join(dir, "cache"), [{ name: "fixture-helper-dep", version: "file:./helper-dep" }]) +} + describe("Npm.install", () => { test("respects omit from project .npmrc", async () => { await using tmp = await tmpdir() @@ -90,38 +138,103 @@ describe("Npm.install", () => { await expect(fs.stat(path.join(tmp.path, "node_modules", "dev-pkg"))).rejects.toThrow() }) - test("skips registry when plugin dependency already exists locally", async () => { + test("preserves package-lock across consecutive installs when plugin dependency already exists locally", async () => { await using tmp = await tmpdir() - await fs.mkdir(path.join(tmp.path, "node_modules", "@opencode-ai", "plugin"), { recursive: true }) - await writePackage(path.join(tmp.path, "node_modules", "@opencode-ai", "plugin"), { name: "@opencode-ai/plugin" }) + await seedPluginProject(tmp.path) - await Effect.gen(function* () { - const npm = yield* Npm.Service - yield* npm.install(tmp.path, { add: [{ name: "@opencode-ai/plugin", version: "1.17.11-main.3" }] }) - }).pipe(Effect.scoped, Effect.provide(npmLayer(path.join(tmp.path, "cache"))), Effect.runPromise) + const lockPath = path.join(tmp.path, "package-lock.json") + const bootstrapped = await lockSnapshot(lockPath) + + await install(tmp.path, path.join(tmp.path, "cache"), [ + { name: PluginSdk.packageName, version: "1.17.11-main.3" }, + ]) + const afterFirst = await lockSnapshot(lockPath) + + await install(tmp.path, path.join(tmp.path, "cache"), [ + { name: PluginSdk.packageName, version: "1.17.11-main.3" }, + ]) + const afterSecond = await lockSnapshot(lockPath) - await expect(fs.stat(path.join(tmp.path, "package-lock.json"))).rejects.toThrow() + expect(afterFirst).toEqual(bootstrapped) + expect(afterSecond).toEqual(afterFirst) }) - test("copies bundled plugin dependency before registry fallback", async () => { + test("copies bundled plugin dependency before registry fallback and preserves package-lock on re-install", async () => { await using tmp = await tmpdir() + await seedPluginProject(tmp.path) const bundled = path.join(tmp.path, "bundled-plugin-sdk") + const previous = process.env.OPENCODE_PLUGIN_SDK_PATH process.env.OPENCODE_PLUGIN_SDK_PATH = bundled await fs.mkdir(path.join(bundled, "src"), { recursive: true }) - await writePackage(bundled, { name: "@opencode-ai/plugin", exports: { ".": "./src/index.ts", "./tui": "./src/tui.ts" } }) + await writePackage(bundled, { name: PluginSdk.packageName, exports: { ".": "./src/index.ts", "./tui": "./src/tui.ts" } }) await Bun.write(path.join(bundled, "src", "index.ts"), "export const plugin = true\n") await Bun.write(path.join(bundled, "src", "tui.ts"), "export const tui = true\n") try { - await Effect.gen(function* () { - const npm = yield* Npm.Service - yield* npm.install(tmp.path, { add: [{ name: "@opencode-ai/plugin" }] }) - }).pipe(Effect.scoped, Effect.provide(npmLayer(path.join(tmp.path, "cache"))), Effect.runPromise) + await fs.rm(path.join(tmp.path, "node_modules"), { recursive: true, force: true }) + const lockPath = path.join(tmp.path, "package-lock.json") + const bootstrapped = await lockSnapshot(lockPath) - await expect(fs.stat(path.join(tmp.path, "node_modules", "@opencode-ai", "plugin", "src", "tui.ts"))).resolves.toBeDefined() - await expect(fs.stat(path.join(tmp.path, "package-lock.json"))).rejects.toThrow() + await install(tmp.path, path.join(tmp.path, "cache"), [{ name: PluginSdk.packageName }]) + await expect( + fs.stat(path.join(tmp.path, "node_modules", PluginSdk.packageName, "src", "tui.ts")), + ).resolves.toBeDefined() + const afterFirst = await lockSnapshot(lockPath) + + await install(tmp.path, path.join(tmp.path, "cache"), [{ name: PluginSdk.packageName }]) + const afterSecond = await lockSnapshot(lockPath) + + expect(afterFirst).toEqual(bootstrapped) + expect(afterSecond).toEqual(afterFirst) } finally { - delete process.env.OPENCODE_PLUGIN_SDK_PATH + if (previous === undefined) delete process.env.OPENCODE_PLUGIN_SDK_PATH + else process.env.OPENCODE_PLUGIN_SDK_PATH = previous } }) + + test("still reifies package-lock when a local plugin install is mixed with a missing dependency", async () => { + await using tmp = await tmpdir() + await seedPluginProject(tmp.path) + + const extra = path.join(tmp.path, "extra-dep") + await fs.mkdir(extra, { recursive: true }) + await writePackage(extra, { name: "fixture-extra-dep", main: "index.js" }) + await Bun.write(path.join(extra, "index.js"), "export const extra = true\n") + + await install(tmp.path, path.join(tmp.path, "cache"), [ + { name: PluginSdk.packageName, version: "1.17.11-main.3" }, + { name: "fixture-extra-dep", version: "file:./extra-dep" }, + ]) + + const lockPath = path.join(tmp.path, "package-lock.json") + const lock = JSON.parse(await fs.readFile(lockPath, "utf8")) + expect(lock.lockfileVersion).toBe(3) + expect(lock.packages[""].dependencies["fixture-extra-dep"]).toMatch(/^file:/) + expect(lock.packages["node_modules/fixture-extra-dep"]).toBeDefined() + await expect(fs.stat(path.join(tmp.path, "node_modules", "fixture-extra-dep"))).resolves.toBeDefined() + }) + + test("reifies package-lock when package.json drifts from the lock", async () => { + await using tmp = await tmpdir() + await seedPluginProject(tmp.path) + + const drift = path.join(tmp.path, "drift-dep") + await fs.mkdir(drift, { recursive: true }) + await writePackage(drift, { name: "fixture-drift-dep", main: "index.js" }) + await Bun.write(path.join(drift, "index.js"), "export const drift = true\n") + + const pkgPath = path.join(tmp.path, "package.json") + const pkg = JSON.parse(await fs.readFile(pkgPath, "utf8")) + pkg.dependencies["fixture-drift-dep"] = "file:./drift-dep" + await Bun.write(pkgPath, JSON.stringify(pkg, null, 2)) + + await install(tmp.path, path.join(tmp.path, "cache")) + + const lockPath = path.join(tmp.path, "package-lock.json") + const lock = JSON.parse(await fs.readFile(lockPath, "utf8")) + expect(lock.lockfileVersion).toBe(3) + expect(lock.packages[""].dependencies).toMatchObject({ "fixture-drift-dep": "file:./drift-dep" }) + expect(lock.packages["node_modules/fixture-drift-dep"]).toBeDefined() + await expect(fs.stat(path.join(tmp.path, "node_modules", "fixture-drift-dep"))).resolves.toBeDefined() + }) }) diff --git a/packages/opencode/src/cli/cmd/db.ts b/packages/opencode/src/cli/cmd/db.ts index 9e7e37e18e..5a3c980169 100644 --- a/packages/opencode/src/cli/cmd/db.ts +++ b/packages/opencode/src/cli/cmd/db.ts @@ -1,9 +1,10 @@ import type { Argv } from "yargs" import { spawn } from "child_process" import { Database } from "@opencode-ai/core/database/database" +import { Vacuum } from "@opencode-ai/core/database/vacuum" import { Effect } from "effect" import { sql } from "drizzle-orm" -import { effectCmd } from "../effect-cmd" +import { effectCmd, fail } from "../effect-cmd" const QueryCommand = effectCmd({ command: "$0 [query]", @@ -51,12 +52,43 @@ const PathCommand = effectCmd({ }), }) +// #524: the ONLY conversion path for legacy auto_vacuum=NONE databases. +// Deliberate invocation by design — the target file must be named explicitly +// with --db, never a default path; pair it with `opencode db path`. Refuses +// anything that is not an existing regular file (a typo must not create a +// database). Converts FULL -> VACUUM -> wal_checkpoint(TRUNCATE) outside any +// startup path and fails nonzero unless the readback is FULL. +const VacuumCommand = effectCmd({ + command: "vacuum", + describe: "convert a database file to full auto_vacuum (FULL -> VACUUM -> truncate WAL)", + instance: false, + builder: (yargs: Argv) => { + return yargs.option("db", { + type: "string", + demandOption: true, + describe: "path to the SQLite database file (print the default with `opencode db path`)", + }) + }, + handler: Effect.fn("Cli.db.vacuum")(function* (args: { db: string }) { + const result = yield* Vacuum.convertToFull(args.db).pipe( + Effect.catch((cause) => + cause._tag === "VacuumRefused" + ? fail(cause.message) + : fail( + `vacuum failed for ${args.db} — close running opencode processes that use this file and retry (${cause.message})`, + ), + ), + ) + console.log(`auto_vacuum=${result.autoVacuum}`) + }), +}) + export const DbCommand = effectCmd({ command: "db", describe: "database tools", instance: false, builder: (yargs: Argv) => { - return yargs.command(QueryCommand).command(PathCommand).demandCommand() + return yargs.command(QueryCommand).command(PathCommand).command(VacuumCommand).demandCommand() }, handler: Effect.fn("Cli.db")(function* () {}), }) diff --git a/packages/opencode/src/effect/app-runtime.ts b/packages/opencode/src/effect/app-runtime.ts index 435b173794..24a186854c 100644 --- a/packages/opencode/src/effect/app-runtime.ts +++ b/packages/opencode/src/effect/app-runtime.ts @@ -61,6 +61,7 @@ import { DagStore } from "@opencode-ai/core/dag/store" import { DagLoop } from "@/dag/runtime/loop" import { DagSummaryPublisher } from "@/dag/runtime/summary-publisher" import { DagSupervisionSweep } from "@/dag/runtime/supervision-sweep" +import { EventResidueSweep } from "@opencode-ai/core/event/residue-sweep" import { Memory } from "@/memory/memory" export const AppLayer = Layer.mergeAll( @@ -138,6 +139,11 @@ export const AppLayer = Layer.mergeAll( // DagLoop it must NOT die with a per-directory instance teardown, or a // `running` node with dead supervision would rot forever. Layer.provideMerge(DagSupervisionSweep.defaultLayer), + // #524: default-on startup residue sweep for crash/in-flight zombie event + // aggregates (both read models absent). Host-level like the supervision + // sweep: one pass per process start, forked into the layer scope, + // soft-degrading — never blocks or fails startup. + Layer.provideMerge(EventResidueSweep.defaultLayer), Layer.provideMerge(SettingsHook.defaultLayer), ) diff --git a/packages/opencode/src/server/routes/instance/httpapi/handlers/sync.ts b/packages/opencode/src/server/routes/instance/httpapi/handlers/sync.ts index 28fd245a63..4bae9e965f 100644 --- a/packages/opencode/src/server/routes/instance/httpapi/handlers/sync.ts +++ b/packages/opencode/src/server/routes/instance/httpapi/handlers/sync.ts @@ -72,7 +72,13 @@ export const syncHandlers = HttpApiBuilder.group(InstanceHttpApi, "sync", (handl const history = Effect.fn("SyncHttpApi.history")(function* (ctx: { payload: typeof HistoryPayload.Type }) { const exclude = Object.entries(ctx.payload) return yield* db - .select() + .select({ + id: EventTable.id, + aggregate_id: EventTable.aggregate_id, + seq: EventTable.seq, + type: EventTable.type, + data: EventTable.data, + }) .from(EventTable) .where( exclude.length > 0 diff --git a/packages/opencode/src/server/routes/instance/httpapi/server.ts b/packages/opencode/src/server/routes/instance/httpapi/server.ts index 0e7f278937..5bae37ef5c 100644 --- a/packages/opencode/src/server/routes/instance/httpapi/server.ts +++ b/packages/opencode/src/server/routes/instance/httpapi/server.ts @@ -50,6 +50,7 @@ import { Storage } from "@/storage/storage" import { Goal } from "@/goal/goal" import { GoalLoop } from "@/goal/loop" import { DagSupervisionSweep } from "@/dag/runtime/supervision-sweep" +import { EventResidueSweep } from "@opencode-ai/core/event/residue-sweep" import { SettingsHook } from "@/hook/settings" import { HookRewakeLive } from "@/hook/rewake-live" import { SessionHooks } from "@/hook/session-hooks" @@ -318,6 +319,12 @@ export const app = LayerNode.group([ // conditional projector UPDATE), so the duplicate is safe — see the sweep // header's multi-host convergence notes. DagSupervisionSweep.node, + // EventResidueSweep (#524): default-on startup residue sweep for crash/ + // in-flight zombie event aggregates. Same app-graph-level placement + // rationale as DagSupervisionSweep above — the desktop sidecar and headless + // serving processes build this node graph without AppLayer, and the sweep + // is idempotent (a second pass in AppLayer processes removes nothing). + EventResidueSweep.node, ]) export function createRoutes( diff --git a/packages/opencode/src/session/llm/native-runtime.ts b/packages/opencode/src/session/llm/native-runtime.ts index 02a3c1902d..cfce400d45 100644 --- a/packages/opencode/src/session/llm/native-runtime.ts +++ b/packages/opencode/src/session/llm/native-runtime.ts @@ -105,6 +105,7 @@ export function stream(input: StreamInput): StreamResult { Effect.gen(function* () { const settlements = yield* FiberSet.make() const results = yield* Queue.unbounded() + const completion: LLMEvent[] = [] const provider = input.llmClient .stream( LLMRequest.update(request, { @@ -112,8 +113,14 @@ export function stream(input: StreamInput): StreamResult { }), ) .pipe( - Stream.flatMap((event) => - event.type !== "tool-call" || event.providerExecuted + Stream.flatMap((event) => { + // The processor may close the stream for compaction at step-finish. + // Deliver every local settlement before exposing that boundary. + if (event.type === "step-finish" || event.type === "finish") { + completion.push(event) + return Stream.empty + } + return event.type !== "tool-call" || event.providerExecuted ? Stream.make(event) : Stream.make(event).pipe( Stream.concat( @@ -126,15 +133,18 @@ export function stream(input: StreamInput): StreamResult { ), ), ), - ), - ), + ) + }), Stream.concat( Stream.fromEffectDrain( FiberSet.awaitEmpty(settlements).pipe(Effect.andThen(Queue.end(results)), Effect.asVoid), ), ), ) - return provider.pipe(Stream.concat(Stream.fromQueue(results))) + return provider.pipe( + Stream.concat(Stream.fromQueue(results)), + Stream.concat(Stream.suspend(() => Stream.fromIterable(completion))), + ) }), ), ) diff --git a/packages/opencode/src/session/session.ts b/packages/opencode/src/session/session.ts index 36a6440a08..ddfe22e664 100644 --- a/packages/opencode/src/session/session.ts +++ b/packages/opencode/src/session/session.ts @@ -38,7 +38,7 @@ import { SessionID, MessageID, PartID } from "./schema" import type { Provider } from "@/provider/provider" import { Global } from "@opencode-ai/core/global" -import { Effect, Layer, Option, Context, Schema, Types } from "effect" +import { Cause, Effect, Layer, Option, Context, Schema, Types } from "effect" import { NonNegativeInt, optionalOmitUndefined } from "@opencode-ai/core/schema" import { RuntimeFlags } from "@/effect/runtime-flags" import { ProviderV2 } from "@opencode-ai/core/provider" @@ -68,25 +68,21 @@ export const MAX_SUMMARY_DIFF_BYTES = 256 * 1024 // Byte accounting mirrors the JSON serialization: 2 bytes for the "[]" wrapper, // +1 per comma separator, so kept output never exceeds MAX_SUMMARY_DIFF_BYTES. +// Entries that do not fit are skipped individually so later, smaller entries +// are still kept. export function truncateSummaryDiffs(diffs: Snapshot.FileDiff[] | undefined) { if (!diffs) return undefined let total = 2 const kept: Snapshot.FileDiff[] = [] for (const item of diffs) { const size = Buffer.byteLength(JSON.stringify(item)) + (kept.length > 0 ? 1 : 0) - if (total + size > MAX_SUMMARY_DIFF_BYTES) break + if (total + size > MAX_SUMMARY_DIFF_BYTES) continue total += size kept.push(item) } return kept } -function stripOversizedDiffs(diffs: T[] | null | undefined) { - if (!diffs) return undefined - if (Buffer.byteLength(JSON.stringify(diffs)) > MAX_SUMMARY_DIFF_BYTES) return undefined - return diffs -} - export function fromRow(row: SessionRow): Info { const summary = row.summary_additions !== null || row.summary_deletions !== null || row.summary_files !== null @@ -94,7 +90,6 @@ export function fromRow(row: SessionRow): Info { additions: row.summary_additions ?? 0, deletions: row.summary_deletions ?? 0, files: row.summary_files ?? 0, - diffs: stripOversizedDiffs(row.summary_diffs), } : undefined const share = row.share_url ? { url: row.share_url } : undefined @@ -165,7 +160,6 @@ export function toRow(info: Info) { summary_additions: info.summary?.additions, summary_deletions: info.summary?.deletions, summary_files: info.summary?.files, - summary_diffs: truncateSummaryDiffs(info.summary?.diffs), metadata: info.metadata, cost: info.cost ?? 0, tokens_input: (info.tokens ?? EmptyTokens).input, @@ -715,6 +709,11 @@ export const layer: Layer.Layer< // the startup orphan-pending sweep (cancel is not a valid transition // from pending). const workflows = yield* dag.store.listBySession(sessionID).pipe(Effect.orDie) + // #524: capture EVERY related dag aggregate before the Deleted publish — + // the projector's session-row delete FK-cascades the workflow rows away + // inside the publish transaction, so listBySession after it returns [] + // and terminal aggregates would be stranded as event-store residue. + const dagIDs = workflows.map((workflow) => workflow.id) for (const workflow of workflows) { // oxlint-disable-next-line typescript-eslint/no-unsafe-type-assertion -- WorkflowRow.status is a plain string column whose values are the WorkflowStatus literals (only the projector writes it, via validated transitions). if (isWorkflowTerminalStatus(workflow.status as never)) continue @@ -740,6 +739,24 @@ export const layer: Layer.Layer< // comes LAST, after every cleanup step above. yield* events.publish(SessionV1.Event.Deleted, { sessionID, info: session }) yield* events.remove(sessionID) + // #524: scrub the related dag event aggregates after the session + // aggregate — terminal workflows included (the cancel loop above skips + // them). Soft-degrading like the EventResidueSweep sibling: a failed + // scrub is logged and leaves the aggregate for the startup residue + // sweep, never fails the removal. Interruption is preserved, not + // degraded into a warning (same hasInterrupts re-raise discipline). + yield* Effect.forEach( + dagIDs, + (dagID) => + events.remove(dagID).pipe( + Effect.catchCause((cause) => + Cause.hasInterrupts(cause) + ? Effect.interrupt + : Effect.logWarning("dag aggregate scrub failed during session remove", { sessionID, dagID, cause }), + ), + ), + { discard: true }, + ) } catch (error) { yield* Effect.logError("failed to remove session", { sessionID, error }) } diff --git a/packages/opencode/test/cli/help/__snapshots__/help-snapshots.test.ts.snap b/packages/opencode/test/cli/help/__snapshots__/help-snapshots.test.ts.snap index 25a4c38f90..dcc795f2bf 100644 --- a/packages/opencode/test/cli/help/__snapshots__/help-snapshots.test.ts.snap +++ b/packages/opencode/test/cli/help/__snapshots__/help-snapshots.test.ts.snap @@ -394,6 +394,8 @@ database tools Commands: opencode db [query] open an interactive sqlite3 shell or run a query [default] opencode db path print the database path + opencode db vacuum convert a database file to full auto_vacuum (FULL -> VACUUM -> truncate + WAL) Positionals: query SQL query to execute [string] @@ -625,3 +627,18 @@ Options: --log-level log level [string] [choices: "DEBUG", "INFO", "WARN", "ERROR"] --pure run without external plugins [boolean]" `; + +exports[`opencode CLI help-text snapshots every documented command emits stable help text: opencode db vacuum --help 1`] = ` +"opencode db vacuum + +convert a database file to full auto_vacuum (FULL -> VACUUM -> truncate WAL) + +Options: + -h, --help show help [boolean] + -v, --version show version number [boolean] + --print-logs print logs to stderr [boolean] + --log-level log level [string] [choices: "DEBUG", "INFO", "WARN", "ERROR"] + --pure run without external plugins [boolean] + --db path to the SQLite database file (print the default with \`opencode db path\`) + [string] [required]" +`; diff --git a/packages/opencode/test/cli/help/help-snapshots.test.ts b/packages/opencode/test/cli/help/help-snapshots.test.ts index 3a14d0d7ec..bdd2af786d 100644 --- a/packages/opencode/test/cli/help/help-snapshots.test.ts +++ b/packages/opencode/test/cli/help/help-snapshots.test.ts @@ -83,6 +83,7 @@ const SUBCOMMANDS = [ ["github", "install"], ["github", "run"], ["db", "path"], + ["db", "vacuum"], ] as const // Fixed wrap width so a developer's terminal doesn't affect snapshots. diff --git a/packages/opencode/test/config/tui-plugin-lock.test.ts b/packages/opencode/test/config/tui-plugin-lock.test.ts new file mode 100644 index 0000000000..284b8f5620 --- /dev/null +++ b/packages/opencode/test/config/tui-plugin-lock.test.ts @@ -0,0 +1,97 @@ +import { expect } from "bun:test" +import fs from "fs/promises" +import path from "path" +import { Effect, Layer } from "effect" +import { FSUtil } from "@opencode-ai/core/fs-util" +import { PluginSdk } from "@opencode-ai/core/plugin-sdk" +import { CurrentWorkingDirectory } from "@/config/tui-cwd" +import { TuiConfig } from "../../src/config/tui" +import { TestInstance } from "../fixture/fixture" +import { testEffect } from "../lib/effect" + +const it = testEffect(Layer.mergeAll(FSUtil.defaultLayer)) + +const withEnv = (name: string, value: string | undefined, self: Effect.Effect) => + Effect.acquireUseRelease( + Effect.sync(() => { + const previous = process.env[name] + if (value === undefined) delete process.env[name] + else process.env[name] = value + return previous + }), + () => self, + (previous) => + Effect.sync(() => { + if (previous === undefined) delete process.env[name] + else process.env[name] = previous + }), + ) + +// Seeds the on-disk steady state of a config directory after a first successful +// startup: package.json declaring the plugin sdk, node_modules populated from a +// local copy, and a tui.json whose path plugin keeps dependency installs armed. +const seedConfigDir = async (dir: string) => { + const localPlugin = path.join(dir, "local-plugin") + await fs.mkdir(localPlugin, { recursive: true }) + await Bun.write( + path.join(localPlugin, "package.json"), + JSON.stringify({ name: PluginSdk.packageName, version: "1.0.0", main: "index.js" }), + ) + await Bun.write(path.join(localPlugin, "index.js"), "export const plugin = true\n") + + await Bun.write( + path.join(dir, "package.json"), + JSON.stringify({ + name: "tui-deps-fixture", + version: "1.0.0", + dependencies: { [PluginSdk.packageName]: "file:./local-plugin" }, + }), + ) + await fs.cp(localPlugin, path.join(dir, "node_modules", ...PluginSdk.packageName.split("/")), { recursive: true }) + + await Bun.write(path.join(dir, "test-plugin.ts"), "export const fixture_plugin = true\n") + await Bun.write(path.join(dir, "tui.json"), JSON.stringify({ plugin: ["./test-plugin.ts"] })) +} + +const lockSnapshot = async (lockPath: string) => { + const [bytes, stat] = await Promise.all([fs.readFile(lockPath), fs.stat(lockPath)]) + return { bytes: bytes.toString(), ino: stat.ino, mtimeMs: stat.mtimeMs } +} + +// One full TUI/config dependency initialization: a fresh TuiConfig layer build +// (the startup path that forks npm.install for the config dir) followed by +// waiting for the forked installs to settle. +const startup = (directory: string, configDir: string) => + withEnv( + "OPENCODE_CONFIG_DIR", + configDir, + TuiConfig.Service.use((svc) => svc.waitForDependencies()).pipe( + Effect.provide(TuiConfig.defaultLayer.pipe(Layer.provide(Layer.succeed(CurrentWorkingDirectory, directory)))), + ), + ) + +it.instance("keeps the config package-lock stable across two dependency initializations", () => + withEnv( + "npm_config_audit", + "false", + Effect.gen(function* () { + const test = yield* TestInstance + const configDir = path.join(test.directory, "deps-config") + yield* Effect.promise(() => fs.mkdir(configDir, { recursive: true })) + yield* Effect.promise(() => seedConfigDir(configDir)) + + const lockPath = path.join(configDir, "package-lock.json") + + yield* startup(test.directory, configDir) + const afterFirst = yield* Effect.promise(() => lockSnapshot(lockPath)) + const lock = JSON.parse(yield* Effect.promise(() => fs.readFile(lockPath, "utf8"))) + expect(lock.lockfileVersion).toBe(3) + expect(lock.packages[""].dependencies[PluginSdk.packageName]).toMatch(/^file:/) + + yield* startup(test.directory, configDir) + const afterSecond = yield* Effect.promise(() => lockSnapshot(lockPath)) + + expect(afterSecond).toEqual(afterFirst) + }), + ), +) diff --git a/packages/opencode/test/dag/dag-replay-idempotency.test.ts b/packages/opencode/test/dag/dag-replay-idempotency.test.ts index cf9afe40ac..8b985ffb7f 100644 --- a/packages/opencode/test/dag/dag-replay-idempotency.test.ts +++ b/packages/opencode/test/dag/dag-replay-idempotency.test.ts @@ -1,12 +1,14 @@ import { describe, expect, it } from "bun:test" -import { DateTime, Effect, Layer } from "effect" -import { sql } from "drizzle-orm" +import { Cause, DateTime, Effect, Exit, Layer } from "effect" +import { eq, sql } from "drizzle-orm" import { Database } from "@opencode-ai/core/database/database" import { EventV2 } from "@opencode-ai/core/event" import { EventTable, EventSequenceTable } from "@opencode-ai/core/event/sql" import { DagProjector } from "@opencode-ai/core/dag/projector" import { DagStore } from "@opencode-ai/core/dag/store" -import { DagEvent } from "@opencode-ai/schema/dag-event" +import { DagEvent, DagID } from "@opencode-ai/schema/dag-event" +import { ProjectID } from "@opencode-ai/schema/project-id" +import { SessionID } from "@opencode-ai/schema/session-id" import { Project } from "@opencode-ai/core/project" import { ProjectTable } from "@opencode-ai/core/project/sql" import { SessionTable } from "@opencode-ai/core/session/sql" @@ -181,4 +183,45 @@ describe("DagProjector: replay idempotency", () => { }).pipe(Effect.provide(projectorLayer)) as Effect.Effect, ) }) + + it("re-materializing a dag aggregate without its session row dies on the workflow FK (#524 zombie shape)", async () => { + await Effect.runPromise( + Effect.gen(function* () { + yield* setupFKs() + const { db } = yield* Database.Service + const events = yield* EventV2.Service + const sessionID = SessionID.make("ses_doomed") + + yield* db + .insert(SessionTable) + .values({ id: sessionID, project_id: Project.ID.global, slug: "doomed", directory: "/project", title: "doomed", version: "test" }) + .run() + .pipe(Effect.orDie) + const dagID = DagID.make("dag_replay_zombie") + yield* events.publish(DagEvent.WorkflowCreated, { dagID, projectID: ProjectID.global, sessionID, title: "zombie-test", config: "{}", status: "pending", timestamp: ts(0) }) + yield* events.publish(DagEvent.WorkflowCompleted, { dagID, durationMs: 0, timestamp: ts(1) }) + + const serialized = yield* serializeAndWipe(dagID) + yield* db.delete(SessionTable).where(eq(SessionTable.id, sessionID)).run().pipe(Effect.orDie) + + // Crash/in-flight zombie shape (#524): the aggregate's session row is + // gone, so the WorkflowCreated read-model INSERT dies on the + // workflow.session_id FK — a wiped dag aggregate whose session was + // removed can never be re-materialized. This is why Session.remove + // scrubs the dag event aggregates instead of relying on replay. + const exit = yield* events.replayAll(serialized).pipe(Effect.exit) + expect(Exit.isFailure(exit)).toBe(true) + if (Exit.isFailure(exit)) expect(Cause.pretty(exit.cause)).toContain("FOREIGN KEY constraint failed") + + // The FK death aborts the replay transaction — no partial-commit garbage. + const residue = yield* db + .select({ id: EventTable.id }) + .from(EventTable) + .where(eq(EventTable.aggregate_id, dagID)) + .all() + .pipe(Effect.orDie) + expect(residue).toEqual([]) + }).pipe(Effect.provide(projectorLayer)), + ) + }) }) diff --git a/packages/opencode/test/mcp/fixtures/process-tree-probe.ts b/packages/opencode/test/mcp/fixtures/process-tree-probe.ts index 9806e2a5f7..1e91aaebd4 100644 --- a/packages/opencode/test/mcp/fixtures/process-tree-probe.ts +++ b/packages/opencode/test/mcp/fixtures/process-tree-probe.ts @@ -80,5 +80,5 @@ const result = await Effect.runPromise( ).pipe(Effect.scoped, Effect.provide(MCP.defaultLayer)), ) -console.log(JSON.stringify(result)) -if (!result.ok || !result.rootDead || !result.childDead) process.exit(1) +await Bun.write(Bun.stdout, `${JSON.stringify(result)}\n`) +process.exit(result.ok && result.rootDead && result.childDead ? 0 : 1) diff --git a/packages/opencode/test/server/httpapi-residue-sweep-wiring.test.ts b/packages/opencode/test/server/httpapi-residue-sweep-wiring.test.ts new file mode 100644 index 0000000000..68a5caee1c --- /dev/null +++ b/packages/opencode/test/server/httpapi-residue-sweep-wiring.test.ts @@ -0,0 +1,26 @@ +import { describe, expect } from "bun:test" +import { CrossSpawnSpawner } from "@opencode-ai/core/cross-spawn-spawner" +import { EventResidueSweep } from "@opencode-ai/core/event/residue-sweep" +import { LayerNode } from "@opencode-ai/core/effect/layer-node" +import { Effect, Layer, Option } from "effect" +import { HttpApiApp } from "@/server/routes/instance/httpapi/server" +import { testEffect } from "../lib/effect" + +// #524 wiring regression: the default-on event residue sweep must reach every +// serving process. Mirrors httpapi-sweep-wiring.test.ts — the desktop sidecar +// and headless serve build this app node graph without AppLayer, so listing +// EventResidueSweep.node here (not just app-runtime.ts) is what makes the +// startup pass run on those paths. + +const appIt = testEffect( + Layer.mergeAll(LayerNode.buildLayer(HttpApiApp.app), CrossSpawnSpawner.defaultLayer), +) + +describe("server app graph event residue sweep wiring", () => { + appIt.instance("exposes EventResidueSweep.Service in the serving context", () => + Effect.gen(function* () { + const sweep = yield* Effect.serviceOption(EventResidueSweep.Service) + expect(Option.isSome(sweep)).toBe(true) + }), + ) +}) diff --git a/packages/opencode/test/server/httpapi-session.test.ts b/packages/opencode/test/server/httpapi-session.test.ts index ae72bc4d0a..75257a8e12 100644 --- a/packages/opencode/test/server/httpapi-session.test.ts +++ b/packages/opencode/test/server/httpapi-session.test.ts @@ -182,22 +182,6 @@ const insertCorruptV2Message = (sessionID: SessionIDType, time = 1) => .pipe(Effect.orDie) }) -const setLegacySummaryDiff = (sessionID: SessionIDType) => - Effect.gen(function* () { - const { db } = yield* Database.Service - yield* db - .update(SessionTable) - .set({ - summary_additions: 1, - summary_deletions: 0, - summary_files: 1, - summary_diffs: [{ additions: 1, deletions: 0 }], - }) - .where(eq(SessionTable.id, sessionID)) - .run() - .pipe(Effect.orDie) - }) - const getWorkspaceID = (sessionID: SessionIDType) => Effect.gen(function* () { const { db } = yield* Database.Service @@ -690,24 +674,6 @@ describe("session HttpApi", () => { { git: true, config: { formatter: false, lsp: false } }, ) - it.instance( - "serves sessions with migrated summary diffs missing file details", - () => - Effect.gen(function* () { - const test = yield* TestInstance - const session = yield* createSession({ title: "legacy diff" }) - yield* setLegacySummaryDiff(session.id) - - const response = yield* request(pathFor(SessionPaths.get, { sessionID: session.id }), { - headers: { "x-opencode-directory": test.directory }, - }) - - expect(response.status).toBe(200) - expect((yield* json(response)).summary?.diffs).toEqual([{ additions: 1, deletions: 0 }]) - }), - { git: true, config: { formatter: false, lsp: false } }, - ) - it.instance( "serves lifecycle mutation routes", () => diff --git a/packages/opencode/test/session/llm-native.test.ts b/packages/opencode/test/session/llm-native.test.ts index 3be43cf929..aa7219a09b 100644 --- a/packages/opencode/test/session/llm-native.test.ts +++ b/packages/opencode/test/session/llm-native.test.ts @@ -557,7 +557,7 @@ describe("session.llm-native.request", () => { }), ) - it.effect("emits native tool calls before overlapping local settlements complete", () => + it.effect("settles parallel native tools before completing the provider step", () => Effect.gen(function* () { const observed: string[] = [] const started: string[] = [] @@ -585,6 +585,7 @@ describe("session.llm-native.request", () => { Stream.fromIterable([ LLMEvent.toolCall({ id: "call-1", name: "lookup", input: {} }), LLMEvent.toolCall({ id: "call-2", name: "lookup", input: {} }), + LLMEvent.stepFinish({ index: 0, reason: "tool-calls", usage: { inputTokens: 30_000, outputTokens: 1 } }), LLMEvent.finish({ reason: "tool-calls" }), ]), generate: () => Effect.die("unused"), @@ -609,11 +610,11 @@ describe("session.llm-native.request", () => { yield* Effect.promise(() => bothStarted) expect(started).toEqual(["call-1", "call-2"]) - expect(observed).toEqual(["tool-call", "tool-call", "finish"]) + expect(observed).toEqual(["tool-call", "tool-call"]) release?.() yield* Fiber.join(fiber) - expect(observed).toEqual(["tool-call", "tool-call", "finish", "tool-result", "tool-result"]) + expect(observed).toEqual(["tool-call", "tool-call", "tool-result", "tool-result", "step-finish", "finish"]) }), ) diff --git a/packages/opencode/test/session/processor-effect.test.ts b/packages/opencode/test/session/processor-effect.test.ts index 106067c6b2..a8c777675b 100644 --- a/packages/opencode/test/session/processor-effect.test.ts +++ b/packages/opencode/test/session/processor-effect.test.ts @@ -19,7 +19,7 @@ import { SessionStatus } from "../../src/session/status" import { SessionSummary } from "../../src/session/summary" import { CrossSpawnSpawner } from "@opencode-ai/core/cross-spawn-spawner" import { provideTmpdirInstance, provideTmpdirServer } from "../fixture/fixture" -import { testEffect } from "../lib/effect" +import { awaitWithTimeout, testEffect } from "../lib/effect" import { raw, reply, TestLLMServer } from "../lib/llm-server" import { RuntimeFlags } from "@/effect/runtime-flags" import { ProviderV2 } from "@opencode-ai/core/provider" @@ -184,6 +184,18 @@ const env = LayerNode.buildLayer(LayerNode.group([root, LayerNode.make(TestLLMSe const it = testEffect(env) +const native = testEffect( + LayerNode.buildLayer(LayerNode.group([root, LayerNode.make(TestLLMServer.layer, [])]), { + replacements: [ + LayerNode.replace(SessionSummary.node, summary), + LayerNode.replace( + RuntimeFlags.node, + RuntimeFlags.layer({ experimentalEventSystem: true, experimentalNativeLlm: true }), + ), + ], + }), +) + const providerErrorLLM = Layer.succeed( LLM.Service, LLM.Service.of({ @@ -235,6 +247,17 @@ const boot = Effect.fn("test.boot")(function* () { return { processors, session, provider } }) +const nativeCompactionProcessor = Effect.fn("test.nativeCompactionProcessor")(function* (msg: SessionV1.Assistant) { + const processors = yield* SessionProcessor.Service + const provider = yield* Provider.Service + const model = { + ...(yield* provider.getModel(ref.providerID, ref.modelID)), + limit: { context: 32_000, output: 4_000 }, + } + const handle = yield* processors.create({ assistantMessage: msg, sessionID: msg.sessionID, model }) + return { model, handle } +}) + // --------------------------------------------------------------------------- // Tests // --------------------------------------------------------------------------- @@ -824,6 +847,190 @@ it.live("session.processor effect tests complete AI SDK tool calls when native f ), ) +native.live("native tools settle before high usage requests compaction", () => + provideTmpdirServer( + ({ dir, llm }) => + Effect.gen(function* () { + const { session } = yield* boot() + yield* llm.push(reply().tool("lookup", { query: "weather" }).usage({ input: 30_000, output: 1 })) + const chat = yield* session.create({}) + const parent = yield* user(chat.id, "finish the slow lookup before compacting") + const msg = yield* assistant(chat.id, parent.id, path.resolve(dir)) + const { model, handle } = yield* nativeCompactionProcessor(msg) + const value = yield* handle.process({ + user: parent, + sessionID: chat.id, + model, + agent: agent(), + system: [], + messages: [{ role: "user", content: "finish the slow lookup before compacting" }], + tools: { + lookup: tool({ + description: "Delayed lookup", + inputSchema: z.object({ query: z.string() }), + execute: async (input, options) => { + await new Promise((resolve, reject) => { + const timer = setTimeout(resolve, 500) + options.abortSignal?.addEventListener( + "abort", + () => { + clearTimeout(timer) + reject(new Error("lookup interrupted")) + }, + { once: true }, + ) + }) + return { title: "Lookup", output: `result:${input.query}`, metadata: {} } + }, + }), + }, + }) + const parts = yield* MessageV2.parts(msg.id) + const call = parts.find((part): part is SessionV1.ToolPart => part.type === "tool") + expect(value).toBe("compact") + expect(call?.state.status).toBe("completed") + if (call?.state.status !== "completed") return + expect(call.state.output).toBe("result:weather") + expect(call.state.input).toEqual({ query: "weather" }) + expect(handle.message.tokens.input).toBe(30_000) + }), + { config: (url) => providerCfg(url) }, + ), +) + +native.live("native parallel tools all deliver results before compaction", () => + provideTmpdirServer( + ({ dir, llm }) => + Effect.gen(function* () { + const { session } = yield* boot() + yield* llm.push( + raw({ + chunks: [ + { + id: "chatcmpl-parallel", + object: "chat.completion.chunk", + choices: [ + { + index: 0, + delta: { + tool_calls: ["first", "second"].map((query, index) => ({ + index, + id: `call_${query}`, + type: "function", + function: { name: "lookup", arguments: JSON.stringify({ query }) }, + })), + }, + finish_reason: "tool_calls", + }, + ], + usage: { prompt_tokens: 30_000, completion_tokens: 1, total_tokens: 30_001 }, + }, + ], + }), + ) + const chat = yield* session.create({}) + const parent = yield* user(chat.id, "complete both lookups") + const msg = yield* assistant(chat.id, parent.id, path.resolve(dir)) + const { model, handle } = yield* nativeCompactionProcessor(msg) + const started: string[] = [] + const bothStarted = defer() + const value = yield* handle + .process({ + user: parent, + sessionID: chat.id, + model, + agent: agent(), + system: [], + messages: [{ role: "user", content: "complete both lookups" }], + tools: { + lookup: tool({ + description: "Parallel lookup", + inputSchema: z.object({ query: z.string() }), + execute: async (input) => { + started.push(input.query) + if (started.length === 2) bothStarted.resolve() + await bothStarted.promise + return { title: "Lookup", output: `result:${input.query}`, metadata: {} } + }, + }), + }, + }) + .pipe((effect) => awaitWithTimeout(effect, "parallel native tools did not complete", "5 seconds")) + expect(value).toBe("compact") + const calls = (yield* MessageV2.parts(msg.id)).filter((part) => part.type === "tool") + expect( + calls.map((part) => ({ + id: part.callID, + state: part.state.status, + output: part.state.status === "completed" ? part.state.output : undefined, + })), + ).toEqual([ + { id: "call_first", state: "completed", output: "result:first" }, + { id: "call_second", state: "completed", output: "result:second" }, + ]) + }), + { config: (url) => providerCfg(url) }, + ), +) + +native.live("user interruption still aborts a native tool waiting to settle", () => + provideTmpdirServer( + ({ dir, llm }) => + Effect.gen(function* () { + const { session } = yield* boot() + yield* llm.push(reply().tool("lookup", { query: "weather" }).usage({ input: 30_000, output: 1 })) + const chat = yield* session.create({}) + const parent = yield* user(chat.id, "cancel the lookup") + const msg = yield* assistant(chat.id, parent.id, path.resolve(dir)) + const { model, handle } = yield* nativeCompactionProcessor(msg) + const started = defer() + let aborted = false + const run = yield* handle + .process({ + user: parent, + sessionID: chat.id, + model, + agent: agent(), + system: [], + messages: [{ role: "user", content: "cancel the lookup" }], + tools: { + lookup: tool({ + description: "Pending lookup", + inputSchema: z.object({ query: z.string() }), + execute: async (_input, options) => { + await new Promise((_resolve, reject) => { + options.abortSignal?.addEventListener( + "abort", + () => { + aborted = true + reject(new Error("lookup interrupted")) + }, + { once: true }, + ) + started.resolve() + }) + return { title: "Lookup", output: "unexpected completion", metadata: {} } + }, + }), + }, + }) + .pipe(Effect.forkChild) + yield* awaitWithTimeout( + Effect.promise(() => started.promise), + "native tool did not start", + ) + yield* awaitWithTimeout(Fiber.interrupt(run), "native tool ignored user interruption") + expect(aborted).toBe(true) + const exit = yield* Fiber.await(run) + expect(Exit.isFailure(exit) && Cause.hasInterruptsOnly(exit.cause)).toBe(true) + const call = (yield* MessageV2.parts(msg.id)).find((part) => part.type === "tool") + expect(call?.state.status).toBe("error") + if (call?.state.status === "error") expect(call.state.metadata?.interrupted).toBe(true) + }), + { config: (url) => providerCfg(url) }, + ), +) + it.live("session.processor effect tests mark pending tools as aborted on cleanup", () => provideTmpdirServer( ({ dir, llm }) => diff --git a/packages/opencode/test/session/session-remove-cleanup.test.ts b/packages/opencode/test/session/session-remove-cleanup.test.ts index 41e7488475..0212446324 100644 --- a/packages/opencode/test/session/session-remove-cleanup.test.ts +++ b/packages/opencode/test/session/session-remove-cleanup.test.ts @@ -1,12 +1,14 @@ import { describe, expect } from "bun:test" -import { Effect, Layer, Option } from "effect" -import { and, eq } from "drizzle-orm" +import { Cause, Context, Effect, Exit, Layer, Option } from "effect" +import { eq, inArray } from "drizzle-orm" import { Database } from "@opencode-ai/core/database/database" -import { EventTable } from "@opencode-ai/core/event/sql" import { EventV2 } from "@opencode-ai/core/event" -import { DagEvent } from "@opencode-ai/schema/dag-event" +import { LayerNode } from "@opencode-ai/core/effect/layer-node" +import { SessionProjector } from "@opencode-ai/core/session/projector" +import { EventSequenceTable, EventTable } from "@opencode-ai/core/event/sql" import { GoalOutcomeTable, GoalStateTable } from "@opencode-ai/core/goal/sql" import { CrossSpawnSpawner } from "@opencode-ai/core/cross-spawn-spawner" +import { EventV2Bridge } from "@/event-v2-bridge" import { Session as SessionNs } from "@/session/session" import { SessionAutomationLease } from "@/session/automation-lease" import { Goal } from "@/goal/goal" @@ -127,28 +129,29 @@ describe("Session.remove dag lease cleanup (GOAL-FP-01-06)", () => { }) expect((yield* dag.store.getWorkflow(dagID).pipe(Effect.orDie))?.status).toBe("running") + // #524 supersession: this pin used to observe the cancel through the + // durable dag.workflow.cancelled event row. Since Session.remove now + // scrubs the whole dag event aggregate AFTER the cancel transition (the + // transition itself is pinned by the dag lifecycle tests), the boundary + // observable is the absence of residue: the cancelled workflow leaves no + // read-model row and no event-store rows behind. yield* session.remove(sessionID) - // The workflow READ row is FK-cascaded away with the session row, so - // the cancellation contract observable here is the durable - // dag.workflow.cancelled event — the terminalization that stops the - // running DagLoop runtime (aborting child sessions and releasing the - // dag lease) and keeps the workflow out of the restart recovery scan. - const cancelledEvent = yield* db - .select() + const sequences = yield* db + .select({ aggregate: EventSequenceTable.aggregate_id }) + .from(EventSequenceTable) + .where(eq(EventSequenceTable.aggregate_id, dagID)) + .all() + .pipe(Effect.orDie) + expect(sequences).toEqual([]) + + const events = yield* db + .select({ aggregate: EventTable.aggregate_id }) .from(EventTable) - .where( - and( - eq(EventTable.aggregate_id, dagID), - eq(EventTable.type, EventV2.versionedType(DagEvent.WorkflowCancelled.type, 1)), - ), - ) - .get() + .where(eq(EventTable.aggregate_id, dagID)) + .all() .pipe(Effect.orDie) - // P2-B: toBeNull() was vacuous — drizzle .get() returns undefined for a - // missing row and `expect(undefined).not.toBeNull()` always passes. - // toBeDefined() actually pins the durable dag.workflow.cancelled event. - expect(cancelledEvent).toBeDefined() + expect(events).toEqual([]) // Recovery scan contract (dag/runtime/loop.ts adopts only // running/paused/stepping rows): the workflow must not be re-adoptable. @@ -157,3 +160,132 @@ describe("Session.remove dag lease cleanup (GOAL-FP-01-06)", () => { }), ) }) + +const workflowConfig = (name: string) => ({ + name, + nodes: [ + { + id: "n1", + name: "n1", + worker_type: "build", + depends_on: [], + required: true, + prompt_template: { inline: "do work" }, + }, + ], +}) + +describe("Session.remove dag aggregate scrub (#524)", () => { + it.instance("removes every related dag event aggregate including terminal workflows", () => + Effect.gen(function* () { + const session = yield* SessionNs.Service + const dag = yield* Dag.Service + const { db } = yield* Database.Service + + const info = yield* session.create({}) + const sessionID = info.id + const terminalDag = yield* dag.create({ + projectID: info.projectID, + sessionID, + title: "scrub-terminal", + config: workflowConfig("scrub-terminal"), + }) + // Terminal BEFORE remove: the pre-publish capture must include it even + // though the cancel loop skips terminal rows as already inert. + yield* dag.cancel(terminalDag) + const liveDag = yield* dag.create({ + projectID: info.projectID, + sessionID, + title: "scrub-live", + config: workflowConfig("scrub-live"), + }) + + const aggregateIDs = [terminalDag, liveDag, sessionID] + const pre = yield* db + .select({ aggregate: EventSequenceTable.aggregate_id }) + .from(EventSequenceTable) + .where(inArray(EventSequenceTable.aggregate_id, aggregateIDs)) + .all() + .pipe(Effect.orDie) + expect(new Set(pre.map((row) => row.aggregate)).size).toBe(3) + + yield* session.remove(sessionID) + + const sequences = yield* db + .select({ aggregate: EventSequenceTable.aggregate_id }) + .from(EventSequenceTable) + .where(inArray(EventSequenceTable.aggregate_id, aggregateIDs)) + .all() + .pipe(Effect.orDie) + expect(sequences).toEqual([]) + const events = yield* db + .select({ aggregate: EventTable.aggregate_id }) + .from(EventTable) + .where(inArray(EventTable.aggregate_id, aggregateIDs)) + .all() + .pipe(Effect.orDie) + expect(events).toEqual([]) + }), + ) +}) + +// #524 interrupt-contract regression: the per-dag scrub catchCause must +// preserve interruption (the EventResidueSweep sibling discipline) instead of +// degrading it into a logWarning. The stub bridge fails events.remove with a +// self-thrown interrupt cause — the only cause shape catchCause can +// intercept; external interrupts bypass it — for every aggregate EXCEPT the +// session's own, so any interrupt surfacing from session.remove can only +// originate from the dag scrub step. +function interruptingScrubBridgeNode(gate: { sessionID?: string }) { + return LayerNode.make( + Layer.effect( + EventV2Bridge.Service, + Effect.gen(function* () { + const bridge = Context.get(yield* Layer.build(EventV2Bridge.layer), EventV2Bridge.Service) + return EventV2Bridge.Service.of({ + ...bridge, + remove: (aggregateID) => + Effect.suspend(() => + gate.sessionID !== undefined && aggregateID !== gate.sessionID + ? Effect.interrupt + : bridge.remove(aggregateID), + ), + }) + }), + ), + [EventV2.node], + ) +} + +const scrubGate: { sessionID?: string } = {} +const scrubInterruptIt = testEffect( + Layer.mergeAll( + LayerNode.buildLayer(LayerNode.group([SessionNs.node, SessionProjector.node, Dag.node]), { + replacements: [LayerNode.replaceWithNode(EventV2Bridge.node, interruptingScrubBridgeNode(scrubGate))], + }), + CrossSpawnSpawner.defaultLayer, + ), +) + +describe("Session.remove dag aggregate scrub interrupt contract (#524)", () => { + scrubInterruptIt.instance("scrub interruption propagates out of remove instead of degrading to a warning", () => + Effect.gen(function* () { + const session = yield* SessionNs.Service + const dag = yield* Dag.Service + const info = yield* session.create({}) + const sessionID = info.id + yield* dag.create({ + projectID: info.projectID, + sessionID, + title: "scrub-interrupt", + config: workflowConfig("scrub-interrupt"), + }) + + scrubGate.sessionID = sessionID + const exit = yield* session.remove(sessionID).pipe(Effect.exit) + scrubGate.sessionID = undefined + + expect(Exit.isFailure(exit) && Cause.hasInterrupts(exit.cause)).toBe(true) + }), + ) +}) diff --git a/packages/opencode/test/session/summary-diff-guard.test.ts b/packages/opencode/test/session/summary-diff-guard.test.ts index 7d109b40c6..ae8c2f0383 100644 --- a/packages/opencode/test/session/summary-diff-guard.test.ts +++ b/packages/opencode/test/session/summary-diff-guard.test.ts @@ -3,12 +3,10 @@ import { Database } from "@opencode-ai/core/database/database" import { LayerNode } from "@opencode-ai/core/effect/layer-node" import { CrossSpawnSpawner } from "@opencode-ai/core/cross-spawn-spawner" import { SessionProjector } from "@opencode-ai/core/session/projector" -import { SessionTable } from "@opencode-ai/core/session/sql" import { SessionV1 } from "@opencode-ai/core/v1/session" import { ModelV2 } from "@opencode-ai/core/model" import { ProviderV2 } from "@opencode-ai/core/provider" import { Effect, Layer } from "effect" -import { eq } from "drizzle-orm" import { Snapshot } from "@/snapshot" import { Session as SessionNs, truncateSummaryDiffs, MAX_SUMMARY_DIFF_BYTES } from "@/session/session" import { SessionSummary } from "@/session/summary" @@ -48,22 +46,6 @@ const giantDiffs = (count: number) => status: "modified" as const, })) -const setSummaryRow = (sessionID: SessionID, summary: { additions: number; deletions: number; files: number; diffs: Snapshot.FileDiff[] }) => - Effect.gen(function* () { - const database = yield* Database.Service - yield* database.db - .update(SessionTable) - .set({ - summary_additions: summary.additions, - summary_deletions: summary.deletions, - summary_files: summary.files, - summary_diffs: summary.diffs, - }) - .where(eq(SessionTable.id, sessionID)) - .run() - .pipe(Effect.orDie) - }) - const seedUserTurn = Effect.fnUntraced(function* (sessionID: SessionID) { const sessions = yield* SessionNs.Service const userMessageID = MessageID.ascending() @@ -140,34 +122,6 @@ describe("summary.diffs source truncation", () => { ) }) -describe("summary_diffs read guard", () => { - it.instance("strips oversized legacy summary_diffs on read and keeps stats", () => - Effect.gen(function* () { - const sessions = yield* SessionNs.Service - const database = yield* Database.Service - const session = yield* sessions.create({ title: "legacy-giant-diffs" }) - - yield* database.db - .update(SessionTable) - .set({ - summary_additions: 12, - summary_deletions: 34, - summary_files: 56, - summary_diffs: giantDiffs(300), - }) - .where(eq(SessionTable.id, session.id)) - .run() - .pipe(Effect.orDie) - - const info = yield* sessions.get(session.id) - expect(info.summary?.additions).toBe(12) - expect(info.summary?.deletions).toBe(34) - expect(info.summary?.files).toBe(56) - expect(info.summary?.diffs).toBeUndefined() - }), - ) -}) - describe("truncateSummaryDiffs boundaries", () => { const item = { file: "a.txt", @@ -194,45 +148,31 @@ describe("truncateSummaryDiffs boundaries", () => { expect(kept?.length).toBe(count) expect(Buffer.byteLength(JSON.stringify(kept))).toBeLessThanOrEqual(MAX_SUMMARY_DIFF_BYTES) }) -}) -describe("summary diffs budget boundary", () => { - it.instance("keeps diffs at just under the budget on write and read", () => - Effect.gen(function* () { - const sessions = yield* SessionNs.Service - const session = yield* sessions.create({ title: "under-budget" }) - const info = yield* sessions.get(session.id) - - const diffs = giantDiffs(100) - expect(Buffer.byteLength(JSON.stringify(diffs))).toBeLessThan(SessionNs.MAX_SUMMARY_DIFF_BYTES) - const row = SessionNs.toRow({ ...info, summary: { additions: 5, deletions: 6, files: 100, diffs } }) - expect(row.summary_diffs).toEqual(diffs) - - yield* setSummaryRow(session.id, { additions: 5, deletions: 6, files: 100, diffs }) - const back = yield* sessions.get(session.id) - expect(back.summary?.diffs).toEqual(diffs) - expect(back.summary?.additions).toBe(5) - expect(back.summary?.deletions).toBe(6) - expect(back.summary?.files).toBe(100) - }), - ) + test("skips an oversized entry and keeps later entries that still fit", () => { + const huge = { ...item, file: "huge.txt", patch: "x".repeat(MAX_SUMMARY_DIFF_BYTES) } + const smallA = { ...item, file: "small-a.txt", patch: "y".repeat(64) } + const smallB = { ...item, file: "small-b.txt", patch: "z".repeat(64) } + const kept = truncateSummaryDiffs([huge, smallA, smallB]) + expect(kept).toEqual([smallA, smallB]) + expect(Buffer.byteLength(JSON.stringify(kept))).toBeLessThanOrEqual(MAX_SUMMARY_DIFF_BYTES) + }) - it.instance("truncates oversized diffs on write within the budget", () => - Effect.gen(function* () { - const sessions = yield* SessionNs.Service - const session = yield* sessions.create({ title: "over-budget-write" }) - const info = yield* sessions.get(session.id) - - const row = SessionNs.toRow({ - ...info, - summary: { additions: 5, deletions: 6, files: 300, diffs: giantDiffs(300) }, - }) - const kept = row.summary_diffs - expect(kept?.length).toBeGreaterThan(0) - expect(kept?.length).toBeLessThan(300) - expect(Buffer.byteLength(JSON.stringify(kept))).toBeLessThanOrEqual(SessionNs.MAX_SUMMARY_DIFF_BYTES) - expect(kept?.[0]?.file).toBe("f000.txt") - expect(kept?.at(-1)?.file).toBe(`f${String((kept?.length ?? 1) - 1).padStart(3, "0")}.txt`) - }), - ) + test("keeps serialized output within the budget for mixed oversized inputs", () => { + const huge = { ...item, patch: "x".repeat(MAX_SUMMARY_DIFF_BYTES) } + const nearBudget = { ...item, patch: "x".repeat(MAX_SUMMARY_DIFF_BYTES - 120) } + const shapes = [ + [huge, item, item], + [item, huge, item], + [item, item, huge], + [nearBudget, item, nearBudget, item], + [huge, nearBudget, huge, item], + [nearBudget, nearBudget], + [item, nearBudget, item, huge, item], + ] + shapes.forEach((shape) => { + const kept = truncateSummaryDiffs(shape) + expect(Buffer.byteLength(JSON.stringify(kept))).toBeLessThanOrEqual(MAX_SUMMARY_DIFF_BYTES) + }) + }) }) diff --git a/script/oc-install-boundary.test.sh b/script/oc-install-boundary.test.sh new file mode 100755 index 0000000000..5c0b4267d0 --- /dev/null +++ b/script/oc-install-boundary.test.sh @@ -0,0 +1,222 @@ +#!/usr/bin/env bash +# shellcheck disable=SC2015,SC2317 +# ok/bad always return 0, so `cond && ok .. || bad ..` cannot mis-fire (SC2015); +# cleanup() runs via the EXIT trap, which shellcheck does not count (SC2317). +# +# B1 (archive integrity, #498): the installer must verify SHA256SUMS BEFORE +# extraction and fail closed on mismatch. These tests pin that boundary by +# driving the real `do_upgrade` path A (remote hash → download → verify → +# extract) from `./oc` sourced in a sandboxed subshell. +# +# tamper — SUMS declares hash A, archive is hash B: do_upgrade must exit +# non-zero AND the install target must not exist (extraction never +# happened). If anyone moves verify after extract, this goes red. +# match — consistent SUMS: end-to-end install succeeds and the installed +# binary smokes (`--version`). +# missing — upstream serves no SUMS: current behavior is warn-and-continue +# (HTTPS transport only). Pinned explicitly so a silent change to +# that policy is a visible test change, not a drift. +# +# Zero network: `curl` is a stub placed first on PATH mapping release URLs to +# local fixtures (same pattern as script/specgit-bootstrap.test.sh); `fzf` is +# stubbed so hosts without it can still source ./oc. HOME, OC_INSTALL_DIR and +# OC_LOCAL_DIR point into a throwaway sandbox; the repo and host are never +# touched. All do_upgrade invocations run in subshells because `die` calls +# exit, which under `source` would kill the harness. +# +# Wired into ci-typecheck.yml (#498); also runnable manually on any bash host: +# bash script/oc-install-boundary.test.sh +set -u + +ROOT=$(cd "$(dirname "$0")/.." && pwd) +# OC_TEST_OC_PATH: target script override (falsifier drills can point at a +# mutated copy, e.g. verify/extract order swapped, to show the suite goes red). +OC="${OC_TEST_OC_PATH:-$ROOT/oc}" +WORK=$(mktemp -d "${TMPDIR:-/tmp}/oc-install-boundary-test.XXXXXX") +PASS=0 +FAIL=0 +TAG="v9.9.8-oc498" +FAKE_VERSION="9.9.8-oc498" + +cleanup() { rm -rf "$WORK"; } +trap cleanup EXIT + +ok() { PASS=$((PASS + 1)); printf 'ok - %s\n' "$1"; } +bad() { FAIL=$((FAIL + 1)); printf 'FAIL - %s\n' "$1"; } + +report() { + printf '\n%d passed, %d failed\n' "$PASS" "$FAIL" + [ "$FAIL" -eq 0 ] +} + +digest() { + if command -v sha256sum >/dev/null 2>&1; then + sha256sum "$1" | awk '{print $1}' + else + shasum -a 256 "$1" | awk '{print $1}' + fi +} + +# Mirror oc's detect_asset for hosts this suite supports; anything else skips. +asset_for_host() { + local os arch + os=$(uname -s) + arch=$(uname -m) + case "$os" in + Linux) + case "$arch" in + x86_64|amd64) echo "opencode-linux-x64.tar.gz" ;; + *) echo "" ;; + esac + ;; + Darwin) + case "$arch" in + arm64|aarch64) echo "opencode-darwin-arm64.zip" ;; + *) echo "" ;; + esac + ;; + *) echo "" ;; + esac +} + +ASSET=$(asset_for_host) +if [ -z "$ASSET" ]; then + echo "skip: unsupported host platform ($(uname -s)/$(uname -m)); suite covers linux-x64 and darwin-arm64" + exit 0 +fi + +# ---- fixtures --------------------------------------------------------------- + +# A minimal "binary" whose --version smoke output the cases assert on. +FIXTURE_BIN_DIR="$WORK/fixture/root" +mkdir -p "$FIXTURE_BIN_DIR" +cat > "$FIXTURE_BIN_DIR/opencode" < "$SUMS_DIR/match" +printf '%s %s\n' "0000000000000000000000000000000000000000000000000000000000000000" "$ASSET" > "$SUMS_DIR/tamper" +# missing: discovery (stdout) still sees valid SUMS; only try_verify's fetch +# fails, which is how oc's warn-and-continue verify branch is reachable. +cp "$SUMS_DIR/match" "$SUMS_DIR/missing" + +# ---- stub tools ------------------------------------------------------------- + +make_stub_dir() { # + local dir="$WORK/stubs/$1" + mkdir -p "$dir" + cat > "$dir/curl" <<'EOF' +#!/usr/bin/env bash +# stub curl: maps release URLs to local fixtures. -o writes to a file, +# otherwise SUMS content goes to stdout (fetch_remote_hash's contract). +# Fully literal: all knobs (OC_TEST_*, SUMS_DIR) arrive via the environment. +set -u +url="" +outfile="" +args=("$@") +for ((i = 0; i < ${#args[@]}; i++)); do + case "${args[$i]}" in + -o) i=$((i + 1)); outfile="${args[$i]}" ;; + -*) ;; + *) url="${args[$i]}" ;; + esac +done +mode="${OC_TEST_SUMS:-match}" +case "$url" in + */SHA256SUMS) + # "missing" fails only the -o fetch (try_verify_sha256); the stdout fetch + # (fetch_remote_hash) still succeeds, so path A runs and oc's + # warn-and-continue branch at the verify step is what gets exercised. + [ "$mode" = "missing" ] && [ -n "$outfile" ] && exit 22 + if [ -n "$outfile" ]; then + cat "$SUMS_DIR/$mode" > "$outfile" + else + cat "$SUMS_DIR/$mode" + fi + exit 0 + ;; + */$OC_TEST_ASSET) + [ -n "$outfile" ] || exit 22 + cat "$OC_TEST_ARCHIVE" > "$outfile" + exit 0 + ;; +esac +exit 22 +EOF + # oc `need fzf` before anything runs; the TUI never starts under the guard. + printf '#!/bin/sh\nexit 0\n' > "$dir/fzf" + chmod +x "$dir/curl" "$dir/fzf" +} + +# run_case — env knobs (OC_TEST_SUMS) must already be exported. +run_case() { # + local case="$1" + local sbx="$WORK/$case" + mkdir -p "$sbx/home" "$sbx/bin" "$sbx/local-missing" + make_stub_dir "$case" + ( + cd "$sbx" + export PATH="$WORK/stubs/$case:$PATH" + export HOME="$sbx/home" + export OC_INSTALL_DIR="$sbx/bin" + export OC_OPENCODE_NAME="opencode" + # No candidate dir has VERSION + asset (sandbox dirs are empty), so + # find_local_dir fails and do_upgrade takes path A (online download). + export OC_LOCAL_DIR="$sbx/local-missing" + export OC_TEST_ARCHIVE="$ARCHIVE" + export OC_TEST_ASSET="$ASSET" + export OC_TEST_TAG="$TAG" + export OC_TEST_OC="$OC" + export SUMS_DIR + bash -c ' + set -u + source "$OC_TEST_OC" + set +euo pipefail + rc=0 + do_upgrade "$OC_TEST_TAG" || rc=$? + exit "$rc" + ' + ) > "$WORK/$case.out" 2> "$WORK/$case.err" +} + +target_of() { # + printf '%s' "$WORK/$1/bin/opencode" +} + +# ---- case tamper: SUMS mismatch must fail closed BEFORE extraction ---------- +OC_TEST_SUMS=tamper run_case tamper +rc=$? +assert_rc() { [ "$1" = "$2" ]; } +assert_rc 1 "$rc" && ok "tamper: do_upgrade exits non-zero on hash mismatch" || bad "tamper: exit $rc, want non-zero (1)" +grep -q "SHA256 不匹配" "$WORK/tamper.err" && ok "tamper: die reports SHA256 mismatch" || bad "tamper: no mismatch diagnostic: $(tr '\n' '|' < "$WORK/tamper.err")" +[ ! -e "$(target_of tamper)" ] && ok "tamper: install target absent — extraction never ran after verify" || bad "tamper: target exists; verification did not gate extraction" + +# ---- case match: consistent SUMS installs end-to-end ------------------------ +run_case match +rc=$? +assert_rc 0 "$rc" && ok "match: do_upgrade exits 0" || bad "match: exit $rc, want 0: $(tr '\n' '|' < "$WORK/match.err")" +target=$(target_of match) +[ -x "$target" ] && ok "match: target installed and executable" || bad "match: target missing or not executable: $target" +[ "$("$target" --version 2>/dev/null)" = "$FAKE_VERSION" ] && ok "match: installed binary smoke --version correct" || bad "match: smoke output '$("$target" --version 2>/dev/null)', want $FAKE_VERSION" + +# ---- case missing: warn-and-continue policy pinned as-is (#498 non-goal) ---- +OC_TEST_SUMS=missing run_case missing +rc=$? +assert_rc 0 "$rc" && ok "missing: warn-skip policy continues to install (current behavior)" || bad "missing: exit $rc, want 0: $(tr '\n' '|' < "$WORK/missing.err")" +grep -q "未提供 SHA256SUMS" "$WORK/missing.err" && ok "missing: warns about absent SHA256SUMS" || bad "missing: no warn diagnostic: $(tr '\n' '|' < "$WORK/missing.err")" +[ -x "$(target_of missing)" ] && ok "missing: install completed under warn-skip" || bad "missing: target not installed" + +report +exit $? diff --git a/script/oc-macos-acceptance.test.sh b/script/oc-macos-acceptance.test.sh new file mode 100755 index 0000000000..77e6470a1a --- /dev/null +++ b/script/oc-macos-acceptance.test.sh @@ -0,0 +1,227 @@ +#!/usr/bin/env bash +# shellcheck disable=SC2015,SC2317 +# ok/bad always return 0, so `cond && ok .. || bad ..` cannot mis-fire (SC2015); +# cleanup() runs via the EXIT trap, which shellcheck does not count (SC2317). +# +# B2 (macOS post-install acceptance, #498): after the installer-style mutation +# (quarantine clearing via `xattr -cr` + ad-hoc re-sign via `codesign -fs -`, +# the real code path in oc's extract_and_install), acceptance must assert: +# +# 1. code-signature VALIDITY: `codesign --verify --strict` passes +# 2. executable smoke: the installed binary runs and answers --version +# 3. structural: the installer never compares the installed binary's hash to +# the archive payload — ad-hoc re-signing can rewrite bytes, so byte +# equality is not a stable signature-validity boundary, and #498 forbids +# publishing a post-sign digest without a supported codesign +# reproducibility matrix +# +# The verify assertion is self-checked: a negative control tampers a copy of +# the installed binary and must FAIL codesign --verify, proving assertion 1 is +# not vacuous. +# +# Modes: +# bash script/oc-macos-acceptance.test.sh +# self-sufficient: compiles an unsigned C stub and exercises the full +# installer flow through the same stub-curl harness as +# script/oc-install-boundary.test.sh (zero network). +# bash script/oc-macos-acceptance.test.sh +# CI release mode: runs the flow against a real packaged artifact, +# e.g. packages/opencode/dist/opencode-darwin-arm64.zip. +# +# Darwin-only; any other host exits 0. Wired into release-fork.yml (macOS job). +set -u + +ROOT=$(cd "$(dirname "$0")/.." && pwd) +OC="$ROOT/oc" +WORK=$(mktemp -d "${TMPDIR:-/tmp}/oc-macos-acceptance.XXXXXX") +PASS=0 +FAIL=0 +TAG="v9.9.8-oc498" +STUB_VERSION="9.9.8-oc498" +ASSET="opencode-darwin-arm64.zip" + +cleanup() { rm -rf "$WORK"; } +trap cleanup EXIT + +ok() { PASS=$((PASS + 1)); printf 'ok - %s\n' "$1"; } +bad() { FAIL=$((FAIL + 1)); printf 'FAIL - %s\n' "$1"; } + +report() { + printf '\n%d passed, %d failed\n' "$PASS" "$FAIL" + [ "$FAIL" -eq 0 ] +} + +if [ "$(uname -s)" != "Darwin" ]; then + echo "skip: macOS acceptance boundary runs on Darwin only ($(uname -s))" + exit 0 +fi + +for tool in cc codesign xattr zip unzip; do + if ! command -v "$tool" >/dev/null 2>&1; then + bad "required tool present: $tool" + report + exit 1 + fi +done + +digest() { shasum -a 256 "$1" | awk '{print $1}'; } + +# ---- fixture archive --------------------------------------------------------- + +ARTIFACT="${1:-}" +if [ -n "$ARTIFACT" ]; then + if [ ! -s "$ARTIFACT" ]; then + bad "artifact zip exists: $ARTIFACT" + report + exit 1 + fi + # Use the real artifact verbatim: extract_and_install finds the binary via + # `find -name "opencode*"`, which matches the packaged layout. + cp "$ARTIFACT" "$WORK/$ASSET" +else + cat > "$WORK/stub.c" <<'EOF' +#include +int main(void) { printf("STUB_VERSION_PLACEHOLDER\n"); return 0; } +EOF + sed -i '' "s/STUB_VERSION_PLACEHOLDER/$STUB_VERSION/" "$WORK/stub.c" + mkdir -p "$WORK/root" + cc -o "$WORK/root/opencode" "$WORK/stub.c" || { + bad "stub binary compiles" + report + exit 1 + } + # Simulate an untrusted payload: strip the linker's ad-hoc signature so the + # installer's re-sign step is what makes the binary valid. + codesign --remove-signature "$WORK/root/opencode" 2>/dev/null || true + (cd "$WORK/root" && zip -q "$WORK/$ASSET" opencode) +fi +ARCHIVE="$WORK/$ASSET" +[ -s "$ARCHIVE" ] || { bad "fixture archive built"; report; exit 1; } + +REAL_HASH=$(digest "$ARCHIVE") +SUMS_DIR="$WORK/sums" +mkdir -p "$SUMS_DIR" +printf '%s %s\n' "$REAL_HASH" "$ASSET" > "$SUMS_DIR/match" + +# ---- stub tools (same contract as oc-install-boundary.test.sh) --------------- + +STUBDIR="$WORK/stubs" +mkdir -p "$STUBDIR" +cat > "$STUBDIR/curl" <<'EOF' +#!/usr/bin/env bash +set -u +url="" +outfile="" +args=("$@") +for ((i = 0; i < ${#args[@]}; i++)); do + case "${args[$i]}" in + -o) i=$((i + 1)); outfile="${args[$i]}" ;; + -*) ;; + *) url="${args[$i]}" ;; + esac +done +case "$url" in + */SHA256SUMS) + if [ -n "$outfile" ]; then + cat "$SUMS_DIR/match" > "$outfile" + else + cat "$SUMS_DIR/match" + fi + exit 0 + ;; + */$OC_TEST_ASSET) + [ -n "$outfile" ] || exit 22 + cat "$OC_TEST_ARCHIVE" > "$outfile" + exit 0 + ;; +esac +exit 22 +EOF +printf '#!/bin/sh\nexit 0\n' > "$STUBDIR/fzf" +chmod +x "$STUBDIR/curl" "$STUBDIR/fzf" + +# ---- run the installer flow against the real Darwin mutation path ---------- + +SBX="$WORK/sbx" +mkdir -p "$SBX/home" "$SBX/bin" "$SBX/local-missing" +( + cd "$SBX" + export PATH="$STUBDIR:$PATH" + export HOME="$SBX/home" + export OC_INSTALL_DIR="$SBX/bin" + export OC_OPENCODE_NAME="opencode" + export OC_LOCAL_DIR="$SBX/local-missing" + export SUMS_DIR + export OC_TEST_ARCHIVE="$ARCHIVE" + export OC_TEST_ASSET="$ASSET" + export OC_TEST_OC="$OC" + export OC_TEST_TAG="$TAG" + bash -c ' + set -u + source "$OC_TEST_OC" + set +euo pipefail + rc=0 + do_upgrade "$OC_TEST_TAG" || rc=$? + exit "$rc" + ' +) > "$WORK/run.out" 2> "$WORK/run.err" + +rc=$? +[ "$rc" -eq 0 ] || bad "do_upgrade exits 0 (installer flow): $(tr '\n' '|' < "$WORK/run.err")" + +TARGET="$SBX/bin/opencode" +[ -x "$TARGET" ] && ok "installer flow completed: target installed and executable" \ + || { bad "target missing or not executable: $TARGET"; report; exit 1; } + +# ---- assertion 1: signature validity after installer-style re-sign ---------- + +if codesign --verify --strict "$TARGET" 2>"$WORK/verify.err"; then + ok "codesign --verify --strict passes on installed binary (B2 validity)" +else + bad "codesign --verify failed on installed binary: $(tr '\n' '|' < "$WORK/verify.err")" +fi + +# Negative control: the validity assertion must be able to fail. Tampering a +# copy breaks the sealed resources, so verify must reject it. +cp "$TARGET" "$WORK/tampered" +printf 'x' >> "$WORK/tampered" +if codesign --verify --strict "$WORK/tampered" 2>/dev/null; then + bad "negative control: tampered copy passed codesign --verify (assertion is vacuous)" +else + ok "negative control: tampered copy rejected by codesign --verify (assertion non-vacuous)" +fi + +# ---- assertion 2: executable smoke ------------------------------------------- + +smoke=$("$TARGET" --version 2>/dev/null) +if [ -n "$ARTIFACT" ]; then + [ "$("$TARGET" --version >/dev/null 2>&1; echo $?)" = "0" ] && [ -n "$smoke" ] \ + && ok "artifact smoke: --version exits 0 with output" \ + || bad "artifact smoke failed: output '$smoke'" +else + [ "$smoke" = "$STUB_VERSION" ] \ + && ok "smoke: installed binary --version correct ($STUB_VERSION)" \ + || bad "smoke output '$smoke', want $STUB_VERSION" +fi + +# ---- assertion 3: structural — installer never hash-compares the target ----- + +# B2 draws the boundary at signature validity + smoke, NOT at byte identity: +# ad-hoc re-signing can rewrite bytes, so byte equality is not a stable +# signature-validity boundary and differing hashes are legitimate. Pin the +# absence of such a comparison. +if grep -nE 'file_sha256.*\$\{?target' "$OC" > "$WORK/struct-grep.txt"; then + bad "structural: installer compares installed binary hash to payload: $(tr '\n' '|' < "$WORK/struct-grep.txt")" +else + ok "structural: no installed-binary hash comparison in oc (B2 is not a digest boundary)" +fi +# The only hash machinery in the install path must live on the archive side +# (try_verify_sha256), before extraction. +if grep -nE 'try_verify_sha256' "$OC" | grep -q 'extract_and_install' ; then + bad "structural: verify and extract lines fused unexpectedly" +else + ok "structural: SHA256SUMS machinery stays on the pre-extract side" +fi + +report +exit $? diff --git a/script/specgit-bootstrap.sh b/script/specgit-bootstrap.sh new file mode 100755 index 0000000000..1037d60d62 --- /dev/null +++ b/script/specgit-bootstrap.sh @@ -0,0 +1,368 @@ +#!/bin/sh +# specgit-bootstrap — repository-local fail-safe wrapper around `specgit issue` (#521). +# +# Why: `specgit issue` (1.10.1) runs an unconditional harness-currency gate that +# exits 2 (`harness_stale`) unless the managed harness was refreshed by +# `specgit init --force`. But `init --force` overwrites this repository's six +# hand-applied specializations (see AGENTS.md, "SpecGit harness local +# specializations"). This wrapper makes the refresh safe: +# +# 1. refuses to run when any init write-surface path has uncommitted changes +# (tracked, staged, or untracked), when the repo has no SpecGit binding, +# or when any requested issue title carries a type outside the allowed +# branch-type vocabulary (#529; AGENTS.md, "Branch Names"); +# 2. snapshots every existing write-surface path to a temp directory OUTSIDE +# the repository, recording each file's `git hash-object` content hash; +# 3. runs `specgit init --force --no-protect` (hardcoded, offline; init's +# stdout prose is routed to stderr so a wrapped `--json` call's stdout +# stays exactly one JSON document) then `specgit issue "$@"` with all +# arguments preserved verbatim and stdin/stdout/stderr inherited; +# 4. restores the snapshots on every exit path (EXIT/INT/TERM/HUP) and +# verifies each restored file byte-for-byte against the recorded hash; +# any mismatch is reported loudly and exits 3. +# 5. on inner success only, verifies the delivered PR targets the dev +# integration base (#528; AGENTS.md, "Git Workflow"): the PR number is +# discovered from the `.specgit.yaml` `pr:` entry on disk, a wrong base +# is corrected with `gh pr edit --base dev`, and `baseRefName` is +# re-verified; discovery/edit/verification failures exit 3 with +# `specgit-bootstrap:` diagnostics while keeping the successful binding. +# Fail-closed: a successful delivery without a unique, readable `pr:` +# record cannot have its PR base verified and exits 3 - a missing, +# unreadable, ambiguous, or malformed entry is never tolerated. +# +# The `.specgit.yaml` delivery record gets conditional rollback (#530): a +# failed inner `specgit issue` (nonzero exit, signal, or init failure) has its +# pre-run bytes restored byte-for-byte; a successful inner call keeps the new +# binding. Record-restore failure keeps the snapshot for forensics and exits +# 3, overriding the inner exit code. Branches, commits, and remote side +# effects are never undone. Wrapper rejections print plain stderr lines +# prefixed `specgit-bootstrap:` — never a `--json` envelope; only the inner +# CLI receives the wrapped arguments. +# +# Usage: script/specgit-bootstrap.sh +# +# Write surface below mirrors specgit 1.10.1 harness-placement; it is +# version-coupled to the pinned CLI in .github/workflows/specgit-accept.yml. + +set -u + +SURFACE=' +.github/workflows/specgit-accept.yml +AGENTS.md +CLAUDE.md +.opencode/hooks.json +.opencode/hooks/specgit-merge-guard.sh +.git/hooks/pre-push +.husky/_/pre-push +' + +say() { + printf 'specgit-bootstrap: %s\n' "$1" >&2 +} + +REPO=$(git rev-parse --show-toplevel 2>/dev/null) || { + say "not inside a git repository" + exit 3 +} +cd "$REPO" || exit 3 + +# Fail-closed: serve only bound delivery repositories; a fresh repo has no +# specializations to protect, so bare `specgit issue` is fine there. +if [ ! -f .specgit.yaml ] || [ ! -f spec_git/policy.yaml ]; then + say "no SpecGit binding (.specgit.yaml / spec_git/policy.yaml missing) - run bare 'specgit issue' instead" + exit 3 +fi + +# Fail-closed: ambiguous pre-existing changes on the write surface could be +# clobbered by init and could not be told apart from init's own writes. +# shellcheck disable=SC2086 +dirty=$(git status --porcelain -- $SURFACE) +if [ -n "$dirty" ]; then + say "refusing to run - init write-surface paths have uncommitted changes (inner CLI NOT executed):" + printf '%s\n' "$dirty" | sed 's/^/ /' >&2 + say "commit or stash those changes first, then retry" + exit 2 +fi + +# ---- #529 preflight: reject unsupported issue title types BEFORE any side +# effect (no snapshot, no init, no inner CLI). The allowed vocabulary is owned +# by AGENTS.md ("Branch Names"); this constant mirrors it verbatim so drift +# surfaces in review. Types are only accepted or rejected - never mapped. +# Rejections are fail-closed: exit 2, `specgit-bootstrap:` diagnostics on +# stderr, no `--json` envelope on stdout. + +PF_ALLOWED_TYPES='chore docs feat fix hotfix refactor release test' + +pf_trim() { # POSIX-whitespace trim; result in $pf_t + pf_t=$1 + pf_t=${pf_t#"${pf_t%%[![:space:]]*}"} + pf_t=${pf_t%"${pf_t##*[![:space:]]}"} +} + +pf_hint() { + say "allowed types: $PF_ALLOWED_TYPES (AGENTS.md \"Branch Names\")" + say "edit the title type to an allowed one, then re-run (no type mapping is performed)" +} + +# Scans the wrapped argv left to right, mirroring the specgit 1.10.1 option +# grammar: --delivery/--tags consume one value each, --name= carries it +# inline, -- ends options, and -h/--help is answered by the CLI without +# operand validation. Positional operands are issue numbers (pure digits - +# the reuse path, "007" -> 7) or titles that must start with +# ': '. The first violation exits 2. +pf_check_issue_titles() { + pf_seen_dd=0 + pf_swallow=0 + for pf_arg in "$@"; do + if [ "$pf_swallow" -eq 1 ]; then + pf_swallow=0 + continue + fi + if [ "$pf_seen_dd" -eq 0 ]; then + case $pf_arg in + --) + pf_seen_dd=1 + continue + ;; + --delivery|--tags) + pf_swallow=1 + continue + ;; + --delivery=*|--tags=*) + continue + ;; + -h|--help) + return 0 + ;; + -*) + continue + ;; + esac + fi + pf_trim "$pf_arg" + if [ -z "$pf_t" ]; then + say "empty issue title argument: '$pf_arg'" + pf_hint + exit 2 + fi + case $pf_t in + *[!0123456789]*) ;; + *) continue ;; # pure digits: issue-number reuse + esac + pf_type=${pf_t%%:*} + pf_rest=${pf_t#*:} + if [ "$pf_rest" = "$pf_t" ]; then + say "unsupported issue title type '$pf_t' in: $pf_arg" + pf_hint + exit 2 + fi + # Conventional titles need whitespace after the colon ("type: desc"); + # trimming changes pf_rest iff it has leading whitespace. + pf_trim "$pf_rest" + if [ "$pf_t" = "$pf_rest" ]; then + say "unsupported issue title syntax in: $pf_arg - need 'type: description' (whitespace after the colon)" + pf_hint + exit 2 + fi + case " $PF_ALLOWED_TYPES " in + *" $pf_type "*) + ;; + *) + say "unsupported issue title type '$pf_type' in: $pf_arg" + pf_hint + exit 2 + ;; + esac + done +} + +pf_check_issue_titles "$@" + +SNAP=$(mktemp -d "${TMPDIR:-/tmp}/specgit-bootstrap.XXXXXX") || { + say "cannot create snapshot directory under \${TMPDIR:-/tmp}" + exit 3 +} +mkdir "$SNAP/tree" "$SNAP/hashes" || { + rm -rf "$SNAP" + say "cannot prepare snapshot directory layout" + exit 3 +} + +# shellcheck disable=SC2086 +for rel in $SURFACE; do + [ -f "$rel" ] || continue + mkdir -p "$SNAP/tree/$(dirname "$rel")" "$SNAP/hashes/$(dirname "$rel")" || { + rm -rf "$SNAP" + say "cannot stage snapshot for $rel" + exit 3 + } + cp "$rel" "$SNAP/tree/$rel" || { + rm -rf "$SNAP" + say "snapshot copy failed for $rel" + exit 3 + } + git hash-object -- "$rel" > "$SNAP/hashes/$rel" || { + rm -rf "$SNAP" + say "content hash failed for $rel" + exit 3 + } +done + +# Conditional record rollback (#530): snapshot the pre-run `.specgit.yaml` +# bytes so a failed inner call can restore them; snapshot failure aborts +# before any side effect. +RECORD_SNAPPED=0 +cp .specgit.yaml "$SNAP/tree/.specgit.yaml" || { + rm -rf "$SNAP" + say "snapshot copy failed for .specgit.yaml" + exit 3 +} +git hash-object -- .specgit.yaml > "$SNAP/hashes/.specgit.yaml" || { + rm -rf "$SNAP" + say "content hash failed for .specgit.yaml" + exit 3 +} +RECORD_SNAPPED=1 + +RESTORED=0 +# Default "failed until the inner call proves success": signal and init +# failure paths hit restore_all before `issue_status` is ever assigned. +issue_status=1 + +# Idempotent restore + byte verification. On mismatch the snapshot directory +# is KEPT for forensics and the wrapper exits 3 (fail-closed, aligning with +# the CLI's exit contract for "cannot proceed"). +restore_all() { + [ "$RESTORED" -eq 1 ] && return 0 + RESTORED=1 + mismatched=0 + # shellcheck disable=SC2086 + for rel in $SURFACE; do + [ -f "$SNAP/tree/$rel" ] || continue + cp "$SNAP/tree/$rel" "$rel" + now=$(git hash-object -- "$rel" 2>/dev/null) + want=$(cat "$SNAP/hashes/$rel" 2>/dev/null) + if [ "$now" != "$want" ]; then + printf 'specgit-bootstrap: RESTORE MISMATCH for %s (got %s, expected %s)\n' \ + "$rel" "${now:-}" "${want:-}" >&2 + mismatched=1 + fi + done + # Roll back the delivery record only when the inner bootstrap failed; + # success keeps the new binding verbatim (#530). + if [ "$RECORD_SNAPPED" -eq 1 ] && [ "$issue_status" -ne 0 ]; then + if [ -f "$SNAP/tree/.specgit.yaml" ]; then + cp "$SNAP/tree/.specgit.yaml" .specgit.yaml + now=$(git hash-object -- .specgit.yaml 2>/dev/null) + want=$(cat "$SNAP/hashes/.specgit.yaml" 2>/dev/null) + if [ "$now" != "$want" ]; then + printf 'specgit-bootstrap: RESTORE MISMATCH for %s (got %s, expected %s)\n' \ + ".specgit.yaml" "${now:-}" "${want:-}" >&2 + mismatched=1 + fi + else + printf 'specgit-bootstrap: RESTORE MISMATCH for %s (snapshot missing)\n' ".specgit.yaml" >&2 + mismatched=1 + fi + fi + if [ "$mismatched" -eq 1 ]; then + say "restored bytes differ from pre-run snapshots - specialized harness bytes may be corrupted." + say "snapshot kept for forensics at $SNAP; inspect 'git diff' before continuing." + trap - EXIT + exit 3 + fi + rm -rf "$SNAP" +} + +trap 'restore_all' EXIT +trap 'restore_all; exit 129' HUP +trap 'restore_all; exit 130' INT +trap 'restore_all; exit 143' TERM + +# ---- #528: PR base verification --------------------------------------------- +# AGENTS.md ("Git Workflow") routes every {type}/** pull request to the dev +# integration branch. A successful inner `specgit issue` therefore must not +# report success while its PR targets another base. The constant mirrors that +# guidance verbatim so drift surfaces in review (same pattern as +# PF_ALLOWED_TYPES above). +PR_TARGET_BASE='dev' + +# Runs ONLY after a successful inner call (issue_status 0). Discovers the PR +# number from the delivery record on disk (never from the inner CLI's stdout, +# which stays byte-exact), corrects a wrong base via `gh pr edit`, and +# re-verifies. Every discovery/edit/verification failure exits 3 with a +# `specgit-bootstrap:` diagnostic; because issue_status is 0, restore_all +# keeps the successful binding (#530) — the PR is real on the remote, so a +# retry can resume from it. Fail-closed (#528): the PR base cannot be +# verified without a unique `pr:` record, so a missing entry exits 3 (no +# success tolerance), and an unreadable/failed scan exits 3 with its own +# diagnosis instead of degrading to a zero-match count. Ambiguous or +# malformed entries also fail closed. +verify_pr_base() { + if [ ! -f .specgit.yaml ]; then + say "PR base verification failed: .specgit.yaml missing after successful bootstrap" + exit 3 + fi + pr_matches=$(sed -n '/^pr:/p' .specgit.yaml) || { + say "PR base verification failed: .specgit.yaml could not be scanned for the pr: entry" + exit 3 + } + if [ -z "$pr_matches" ]; then + say "PR base verification failed: .specgit.yaml has no pr: entry - the PR base cannot be verified" + exit 3 + fi + pr_count=$(printf '%s\n' "$pr_matches" | wc -l | tr -d ' ') + if [ "$pr_count" -gt 1 ]; then + say "PR base verification failed: .specgit.yaml has $pr_count pr: entries (ambiguous)" + exit 3 + fi + command -v gh >/dev/null 2>&1 || { + say "PR base verification failed: 'gh' not found on PATH - cannot verify the PR base" + exit 3 + } + pr_number=$(sed -n 's/^pr:[[:space:]]*\([0-9][0-9]*\)[[:space:]]*$/\1/p' .specgit.yaml) + if [ -z "$pr_number" ]; then + say "PR base verification failed: .specgit.yaml pr: entry is not a plain number" + exit 3 + fi + pr_base=$(gh pr view "$pr_number" --json baseRefName --jq .baseRefName) || { + say "PR base verification failed: gh pr view $pr_number could not read the PR base" + exit 3 + } + [ "$pr_base" = "$PR_TARGET_BASE" ] && return 0 + say "PR #$pr_number targets base '$pr_base' - correcting to '$PR_TARGET_BASE' (AGENTS.md \"Git Workflow\")" + # gh edit prose goes to stderr so a wrapped --json stdout stays byte-exact. + gh pr edit "$pr_number" --base "$PR_TARGET_BASE" >&2 || { + say "PR base verification failed: gh pr edit $pr_number --base $PR_TARGET_BASE did not succeed" + exit 3 + } + pr_base=$(gh pr view "$pr_number" --json baseRefName --jq .baseRefName) || { + say "PR base verification failed: gh pr view $pr_number (re-check) could not read the PR base" + exit 3 + } + if [ "$pr_base" != "$PR_TARGET_BASE" ]; then + say "PR base verification failed: PR #$pr_number still targets '$pr_base' after correction" + exit 3 + fi +} + +# init's stdout prose must not pollute the wrapped --json parse surface. +specgit init --force --no-protect >&2 +init_status=$? +if [ "$init_status" -ne 0 ]; then + say "specgit init --force --no-protect failed (exit $init_status); restoring harness bytes" + restore_all + exit "$init_status" +fi + +# All arguments pass through verbatim; exit status and diagnostics inherit. +specgit issue "$@" +issue_status=$? +if [ "$issue_status" -eq 0 ]; then + # #528: verify (and correct) the PR base before reporting success. On a + # verification failure verify_pr_base exits 3 while issue_status stays 0, + # so restore_all keeps the successful binding for retries. + verify_pr_base +fi +restore_all +exit "$issue_status" diff --git a/script/specgit-bootstrap.test.sh b/script/specgit-bootstrap.test.sh new file mode 100755 index 0000000000..51820179a2 --- /dev/null +++ b/script/specgit-bootstrap.test.sh @@ -0,0 +1,903 @@ +#!/usr/bin/env bash +# shellcheck disable=SC2015,SC2329 +# ok/bad always return 0, so `cond && ok .. || bad ..` cannot mis-fire (SC2015); +# cleanup() runs via the EXIT trap, which shellcheck does not count (SC2329). +# Behavior tests for script/specgit-bootstrap.sh (#521, #530 record rollback, +# #529 issue-title type preflight, #528 PR base verification). +# +# Zero network, zero forge: `specgit` is a stub placed first on PATH; every +# fixture is a throwaway git repo under $TMPDIR. The real repository is never +# touched: wrapper invocations run with cwd set to the fixture, and all stub +# artifacts (log, captured output) live outside the fixture worktree. +# +# Not wired into CI (#521 scope): run manually from anywhere via +# bash script/specgit-bootstrap.test.sh +set -u + +ROOT=$(cd "$(dirname "$0")/.." && pwd) +WRAPPER="$ROOT/script/specgit-bootstrap.sh" +WORK=$(mktemp -d "${TMPDIR:-/tmp}/specgit-bootstrap-test.XXXXXX") +PASS=0 +FAIL=0 + +cleanup() { rm -rf "$WORK"; } +trap cleanup EXIT + +if command -v sha256sum >/dev/null 2>&1; then + digest() { sha256sum "$1" | cut -d' ' -f1; } +else + digest() { shasum -a 256 "$1" | cut -d' ' -f1; } +fi + +SURFACE_FILES=( + .github/workflows/specgit-accept.yml + AGENTS.md + .opencode/hooks.json + .opencode/hooks/specgit-merge-guard.sh + .git/hooks/pre-push +) + +ok() { PASS=$((PASS + 1)); printf 'ok - %s\n' "$1"; } +bad() { FAIL=$((FAIL + 1)); printf 'FAIL - %s\n' "$1"; } + +surface_digests() { # + local fx="$1" f + for f in "${SURFACE_FILES[@]}"; do + printf '%s %s\n' "$(digest "$fx/$f")" "$f" + done +} + +assert_surface_restored() { # + diff <(surface_digests "$1") "$2" >/dev/null +} + +assert_clean() { # + [ -z "$(git -C "$1" status --porcelain)" ] +} + +assert_rc() { #