Repository navigation
Conversation
There was a problem hiding this comment.
I recommend merging this after two changes. I reproduced the core problem (a slow but live service gets killed during recovery) with the PR's own tests rather than in a real disk-stall. On the base, both new unresponsive: "wait" reconnect tests fail for the Promise and Effect clients, because the slow owner is killed or replaced. With this change they pass. I couldn't reproduce a real heavy-disk or swap stall here.
Checks: the client, cli, server and desktop tests closest to the change all pass. Type checks are clean for client, core, server and cli; the desktop type check ran out of memory in my environment, so it's unchecked.
Public API: @opencode/client adds an optional EnsureOptions.unresponsive?: "evict" | "wait" (default "evict") and exports the UnresponsivePolicy type. The internal EnsureTiming gains unresponsiveAttempts. ensure can now fail with a new message, Timed out waiting for the background service to respond. Default eviction now takes 6 × 5s instead of 3 × 2s. All of this is backwards compatible.
Concerns:
- The state-D guard in
terminate()also applies to explicitstop/restart(see the inline comment). - The fake-timer backoff test is fragile and should be cut down, and the constants-only timing test can be dropped (inline comments).
There was a problem hiding this comment.
The new commits fix all three earlier points. terminate() keeps SIGKILL for explicit stop/restart again, and the state-D check now only skips automatic eviction. The backoff test is now small and deterministic, and the constants-only test is gone. I re-ran the client tests (the backoff test three times, all passing) and the client type-check is clean. The public API is the same as before: an optional EnsureOptions.unresponsive setting, which is backwards compatible. I have no new concerns.
There was a problem hiding this comment.
My earlier points were already addressed, with nothing left open, and these commits don't bring any of them back. I have no new concerns. The new time-based wait looks correct in both clients. While the wait time is running, the startup deadline holds off; once a service is replaced, the deadline restarts. A service in disk sleep (state D) is only postponed, not reset. Neither client starts extra processes against a service that still holds the port. I re-ran the client tests (268 pass), the cli and desktop service tests, and the client and cli type checks, all clean.
Public API (@opencode/client, backwards compatible): adds an optional EnsureOptions.unresponsiveTimeout (default 30s, where eviction used to come after about 6s), exports reconnectUnresponsiveTimeout (3 minutes) and adds the error message "Timed out waiting for the background service to respond". The earlier unresponsive: "evict" | "wait" option and UnresponsivePolicy type are gone, but they never shipped.
There was a problem hiding this comment.
The new commit's change looks correct. The growing retry delay now only applies when a reconnect function is configured, the one that can start a server. A plain event-stream retry keeps its 1s delay, and the backoff test now covers both cases. I re-ran the client tests (269 pass), ran solid-connection.test.ts five times with no failures, and the client type check is clean. The public API is the same as in my last review. One minor point about the new test helper is in an inline comment.
There was a problem hiding this comment.
The new commit looks correct and I have no new concerns. Every timed-out probe now checks whether the service is in disk sleep, in both clients, and that time doesn't count toward its wait limit. Real elapsed time still ends the startup-deadline hold, so a service stuck in disk sleep makes ensure fail with the "did not respond" error instead of waiting forever. The two new tests drive the timer directly and are small and deterministic. I re-ran the client tests (271 pass) and the client type check is clean. The public API is the same as in my last review. My open point on the solid-connection.test.ts wait loop is not addressed yet.
There was a problem hiding this comment.
The new commit fixes my open point on the solid-connection.test.ts wait loop. It now gives up after a bounded number of spins and fails with a clear error, so it can no longer hang the test run. I ran that test file three times with no failures, and the client type check is clean. The public API is the same as in my last review. I have no new concerns.
…ponsive service Every client shares one background service, so a per-caller budget let the least patient caller (a fresh launch, any CLI command) kill a service a reconnecting client was still waiting for. The budget is now internal and the same for every ensure() call, which also removes EnsureOptions.unresponsiveTimeout and reconnectUnresponsiveTimeout from the public API.
A contender registers as soon as it listens and then opens the database on its event loop. On a slow disk that boot can outlast the unresponsive budget, and the same ensure() call killed it and started the same boot again, without end now that eviction holds and resets the startup deadline. A service this call started is bounded by the startup deadline instead, and times out with the 'to start' message rather than 'to respond'.
Service.ensure now waits minutes for a registered service that stopped answering before it replaces it, and the CLI printed nothing meanwhile. After 5 seconds without an endpoint, and before any server was started, print a notice on stderr.
Removing it is unrelated to client recovery and moves to its own pull request.
There was a problem hiding this comment.
I have no new concerns about these commits. Every caller now waits the same 3 minutes before replacing a service that stops answering. A server that ensure started itself is never killed and is limited only by the startup deadline; matching its process ID against the registration works for the commands the CLI and desktop start. The CLI's 5-second "waiting" notice is cancelled as soon as resolving finishes or fails, and is skipped once a server has been started. The WAL checkpoint going back into its own PR is fine. I re-ran the client tests (273 pass, and the service test files three times with no failures), the cli and desktop service tests, and the client and cli type checks, all clean.
Public API: unchanged compared with the base. The unresponsiveTimeout option and reconnectUnresponsiveTimeout from earlier revisions are gone. The only visible addition is that ensure can fail with the new message "Timed out waiting for the background service to respond".
Issue for this PR
N/A
Type of change
What does this PR do?
When the background service stalls under heavy disk or swap pressure (synchronous
bun:sqliteI/O on a largeopencode.dbwhile many subagents saturate the disk), client recovery turned a temporary stall into a kill-and-restart loop.Before. The reconnecting client killed a live server after about 6 s, and every replacement stalled on the same disk and was killed the same way. Any client that started during the stall (
opencode,opencode stats, the desktop app) did the same.sequenceDiagram participant T as TUI participant E as Service.ensure participant S as Background service participant R as Replacement Note over S: SQLite I/O on a saturated disk blocks its event loop T->>T: 45 s without SSE bytes T->>E: reconnect loop 3 probes, about 6 s E-xS: GET /api/info times out after 2 s end E->>S: clear persistent terminals, SIGTERM then SIGKILL Note over S: every session and terminal is gone E->>R: start a replacement Note over R: same database on the same disk, so it stalls too T->>E: reconnect again after 1 s, no backoff E->>R: 3 timed-out probes, killed the same wayAfter. Every caller gives a server that stopped answering the same 3 minutes, so a stall recovers on the same server. A server that stays hung is still replaced.
sequenceDiagram participant T as TUI participant E as Service.ensure participant S as Background service participant R as Replacement Note over S: SQLite I/O on a saturated disk blocks its event loop T->>T: 45 s without SSE bytes T->>E: reconnect, retries back off from 1 s to 15 s loop until it answers or 3 min of unanswered time pass E-xS: GET /api/info times out after 2 s Note over E: no contenders meanwhile, disk sleep does not count (Linux) end alt it answers in time S-->>E: answers once the disk catches up E-->>T: same server, sessions and terminals intact else still unanswered after 3 min E->>S: clear persistent terminals, SIGTERM then SIGKILL E->>R: start a replacement Note over R: never killed by this call for booting slowly endChanges:
/api/infofor 3 minutes before it is stopped and replaced, whoever callsService.ensure: TUI/CLI startup and reconnect, every other CLI command, desktop launch and reconnect. Every client shares one service, so a per-caller budget let the least patient caller kill it while a reconnecting client was still waiting (the session loss in [service]: managed service stalls, watchdog recovery loses sessions (404 "session not found"), restart stampede races the service slot (2.0.25 vs 2.0.26) #54181 came from a CLI starting up). A stall shorter than that recovers on the same server with sessions and terminals intact; a server that stays hung is still replaced. The probe timeout stays at 2s, sodiscover,incumbentand terminal handoff are unchanged. While an unresponsive owner is within its budget the startup deadline waits for it, so a budget longer thanpromiseTimeoutis not cut short.ensurestarted is never evicted. A contender registers as soon as it listens and then opens the database on its event loop. If that boot outlasted the budget (Managed background service startup can take minutes and saturates the disk; the CLI's short health check turns it into a spawn/retry storm #54810), the same call killed it and started the same boot again, indefinitely now that eviction holds and resets the startup deadline. The startup deadline bounds its own contenders instead, and they time out with "Timed out waiting for the background service to start", which desktop still retries./proc/<pid>/stat; time the process spends inDstate does not count toward the budget. Checking only at the end of the budget would evict the server the moment the disk caught up (verified, see below). The budget's wall time still ends the deadline hold, so a process stuck inDmakesensurefail with "Timed out waiting for the background service to respond" instead of hanging it. One sample per probe, so the excluded time is an estimate. ExplicitService.stop/restart keep their SIGKILL escalation.reconnectfunction the delay grows1s → 2s → 4s → 8s → 15sand resets after a stable connection; plain stream retries (explicit--server, remote and plugin servers) keep the 1s delay.default.tsandmini.tsrunservice.reconnect/restartwith the full Effect context so recovery warnings reachopencode.log.Removing the startup
PRAGMA wal_checkpoint(PASSIVE)moved to its own PR, #64532.Public API: unchanged. The budget is internal (
defaultEnsureTiming), and tests shorten it throughwithEnsureTiming.ensurecan fail with a new message, "Timed out waiting for the background service to respond", when a service it did not start is still unresponsive at the deadline. Default eviction moves from about 6s to 3 minutes.Not addressed: SQLite still runs on the server's event loop, so a disk stall still freezes every client until it ends; a hung server shows "reconnecting" for up to 3 minutes with no restart prompt, and a fresh launch waits as long before replacing it (
opencode service restartreplaces it at once) (follow-up).No feature flag: bug fix.
How did you verify your code works?
packages/clientbun run testand typechecks on macOS, Windows and Linux, including: a slow owner within its budget is kept with no contenders; a hung owner is evicted once a budget longer than the startup deadline runs out; disk sleep pauses the budget (pure tracker test, fails with check-at-threshold); backoff applies only with a reconnect functionpackages/desktopservice tests,packages/cliserver-connection/service,packages/serverprocess.test.ts,packages/coremigration tests on macOS, Windows and Linuxserve --serviceprocesses on macOS (SIGSTOP), Windows (NtSuspendProcess) and Linux (SIGSTOP): a stall within the reconnect budget recovers on the same PID; a server hung past the budget is replacedio.maxsat inDfor 15s. With this PR it was never signalled andensurereturned the same server at 15.1s; with the disk-sleep check disabled it was killed after about 8spackages/clientbun run test,bun run typecheckfor every package,packages/cliserver-connectiontest, andServerConnection.resolveagainst a registered server that never answers (the notice prints after 5s; nothing is started or killed). New tests: a serviceensurestarted that stops answering is kept until the deadline and fails with "…to start"; a registered contender that stops answering after its sibling failed is kept and returned once it answers. Both fail with the contender check removedbun run checkfrom the repository root. On Windowsscript/check-greys.tsfails on files this PR does not touch, because its exception paths use forward slashesScreenshots / recordings
N/A
Checklist
Added by the opencode bot (AI-generated).
Related: #58111, #63357, #54181, #54810
Public API:
Service.ensurecan fail with a new 'Timed out waiting for the background service to respond' message; no other public API change (backwards compatible)