Repository navigation
Description
Description
When the background service is busy with many parallel agent sessions, GET /api/info often takes more than 2 s. It sometimes takes 10 to 28 s, while the service keeps working and then answers again by itself. Any client that calls Service.ensure() during one of these stalls (opencode api …, the TUI, opencode run) counts three 2 s probe timeouts. It then logs Background service is unresponsive; recovery cannot preserve persistent terminals, sends SIGTERM and then SIGKILL to the service, and starts a new one. Every in-flight turn is interrupted, and the new service resumes each session with "The server restarted while you were working…".
The old process was not hung. In each reproduction:
- An independent
/api/infoprobe got an answer from the old PID within a second of the client deciding to kill it (79 ms in the run with millisecond timestamps). - The old service logged
shutdown started signal=SIGTERM0.2 to 2.4 s after the warning, then answered/api/infowith 503 while it shut down. - Right before the stall, it was making 20 to 30 model requests per second to the provider.
With the same load and no client, the service stayed up for the whole run. Its slowest /api/info answer took 9.7 s.
Expected: a client does not kill a live service that is doing work only because /api/info was slow for 6 s. Alternatively, /api/info keeps answering under load.
Actual: the service is replaced after about 6 s (3 × 2 s), and every running session and subagent is interrupted.
In #51216, the bot wrote: "Anything else that blocks the server for more than about 6s could still cause this restart." This report is that case on 2.3.0, reproducible on Linux, with no snapshot-heavy tree and no watcher churn. Related: #58111 (same kill path, triggered by host sleep) and #58130 (30 s window plus liveness check). The longest stall I measured lasted about 28 s (14 consecutive 2 s timeouts, after which the same PID answered), which is close to that window.
I first saw this on Windows 11 with a shared service running many parallel sessions and subagents (about 100 % CPU, 1.5 to 2 GB RSS). Opening the TUI logged the warning about 6 s after start. Later, a one-shot opencode api get /api/session/<id> did the same 6 s later. All in-flight runs were lost both times.
Measured on Linux (8 vCPU VM), with a fake provider where every turn is a tool call (read, grep, shell, write):
| Load | Main thread | /api/info (fresh connection) |
Replaced by a client |
|---|---|---|---|
| 200 sessions streaming text only | not saturated | ≤ 0.1 s | no |
| 150 sessions in tool loops | ~100 % | ≤ 0.8 s | no |
| 300–500 sessions in tool loops | 100 % | up to 2.7 s; 16 % of probes > 2 s | no |
| 900–1000 sessions in tool loops | 100 % | stalls of 6 to 28 s; 49 runs of ≥ 3 consecutive 2 s timeouts in 19 min | yes, 4 times out of 4 |
| same, no client (control, 60 s probe timeout) | 100 % | max 9.7 s; 31 answers > 2 s | service stays up |
Client logic in v2.3.0: packages/client/src/effect/service.ts → ensure(). After timeouts.count >= 3, it calls PtyHandoff.clear and terminate() (SIGTERM, 5 s, SIGKILL). requestTimeout: 2_000 is set in service-timing.ts.
Plugins
None
OpenCode version
2.3.0
Steps to reproduce
- On Linux x86_64 or arm64, with
bash,curl,git,node18 or later andnpm, save the script below asrepro.sh. - Run
bash repro.sh. It installs@opencode/cli-linux-<arch>@2.3.0into a temporary sandbox with its ownHOMEandXDG_*directories, so it never touches an existing service. It also starts a local fake OpenAI-compatible provider (no API key), creates a small git repo, starts the sandbox service and launches 1000 sessions in waves of 100. It then runsopencode api get /api/sessionrepeatedly until the service PID changes. - Optional:
CONTROL=1 DURATION=240 bash repro.shruns the same load with no client and measures how long/api/inforeally takes.
Output from one run, trimmed (other runs reproduced after 45 s and after 19 min):
[14:37:38] service pid=2534678 url=http://127.0.0.1:49374
[14:40:27] client call #140 took 5828 ms; provider: provider_requests=8612 open_streams=4; 16 probe timeout runs so far
[14:40:58] REPRODUCED: client call #145 (12072 ms) replaced the service: pid 2534678 -> 2599794
--- client/service log ---
14:40:46.916Z INFO cli starting args=["api","get","/api/session"]
14:40:52.975Z WARN Background service is unresponsive; recovery cannot preserve persistent terminals
14:40:53.388Z INFO shutdown started signal=SIGTERM <- old service, still running its event loop
14:40:58.452Z INFO cli starting args=["serve","--service"]
--- independent /api/info probe (2 s, back to back) ---
14:40:43 /api/info timed out (2s) 3 time(s) in a row, then answered in 1.384547s
14:40:53 /api/info timed out (2s) 3 time(s) in a row, then answered in 0.416798s <- old PID answers again
--- provider traffic (model calls made by the service) ---
14:40:45Z provider_requests=9433
14:40:50Z provider_requests=9582
14:40:55Z provider_requests=9693
--- sessions told 'The server restarted while you were working' (sample of 10) ---
10/10 sampled sessions carry the restart notice (of 900 sessions running)
repro.sh
#!/usr/bin/env bash
# Reproduce: under heavy but healthy load, a one-shot `opencode api` client replaces the live
# background service (three 2 s /api/info probe timeouts), which interrupts every running session.
#
# Fully sandboxed: private HOME and XDG_* dirs under $ROOT, a local fake OpenAI-compatible provider
# (no API key, no network calls to model vendors), its own service on its own port. It never talks
# to a background service you may already run.
#
# Needs: bash, curl, git, node >= 18, npm. ~300 MB of disk for the binary, ~200 MB for data.
# Tested on Ubuntu 24.04 x86_64, 8 vCPU VM: replaced after 45 s, 3.3 min and 19 min in three runs.
# CONTROL=1 runs the same load with no client and a 60 s probe timeout: the service stays up.
# Keep ROOT outside your home: project config is also looked up in parent directories.
#
# Usage: bash repro.sh # defaults below
# SESSIONS=1500 DURATION=2400 bash repro.sh
set -u
VERSION=${VERSION:-2.3.0}
SESSIONS=${SESSIONS:-1000} # parallel agent sessions
WAVE=${WAVE:-100} # sessions started per wave, one wave every WAVE_GAP seconds
WAVE_GAP=${WAVE_GAP:-20}
STEPS=${STEPS:-150} # tool-call rounds per session before its final answer
DURATION=${DURATION:-2400} # give up after this many seconds
PORT=${PORT:-4010} # fake provider port
ROOT=${ROOT:-$(mktemp -d /tmp/oc-busy-service.XXXXXX)}
mkdir -p "$ROOT"; cd "$ROOT" || exit 1
log() { echo "[$(date -u +%H:%M:%S)] $*"; }
# --- 1. OpenCode, isolated -----------------------------------------------------------------------
case "$(uname -m)" in x86_64) ARCH=x64 ;; aarch64|arm64) ARCH=arm64 ;; *) echo "unsupported arch"; exit 1 ;; esac
if [ ! -x "$ROOT/npm/node_modules/@opencode/cli-linux-$ARCH/bin/opencode" ]; then
log "installing @opencode/cli-linux-$ARCH@$VERSION into $ROOT/npm"
npm install --silent --no-audit --no-fund --prefix "$ROOT/npm" "@opencode/cli-linux-$ARCH@$VERSION" || exit 1
fi
OC="$ROOT/npm/node_modules/@opencode/cli-linux-$ARCH/bin/opencode"
export HOME="$ROOT/home" XDG_CONFIG_HOME="$ROOT/home/.config" XDG_DATA_HOME="$ROOT/home/.local/share" \
XDG_STATE_HOME="$ROOT/home/.local/state" XDG_CACHE_HOME="$ROOT/home/.cache" XDG_RUNTIME_DIR="$ROOT/run"
mkdir -p "$XDG_CONFIG_HOME/opencode" "$XDG_RUNTIME_DIR" && chmod 700 "$XDG_RUNTIME_DIR"
REG="$XDG_STATE_HOME/opencode/service.json"
SLOG="$XDG_DATA_HOME/opencode/log/opencode.log"
log "opencode $("$OC" --version) sandbox=$ROOT"
cat > "$XDG_CONFIG_HOME/opencode/opencode.json" <<EOF
{
"\$schema": "https://opencode.ai/config.json",
"autoupdate": false,
"share": "disabled",
"model": "fake/slow",
"small_model": "fake/slow",
"provider": {
"fake": {
"npm": "@ai-sdk/openai-compatible",
"name": "Fake provider",
"options": { "baseURL": "http://127.0.0.1:$PORT/v1", "apiKey": "not-a-key" },
"models": { "slow": { "name": "slow", "limit": { "context": 1000000, "output": 100000 } } }
}
}
}
EOF
# --- 2. Fake OpenAI-compatible provider: each turn is a tool call (read/grep/shell/write) ---------
cat > "$ROOT/fake-provider.mjs" <<'EOF'
import http from "node:http";
const [port, steps] = [Number(process.argv[2]), Number(process.argv[3])];
let served = 0, active = 0;
const sse = (res, o) => res.write(`data: ${JSON.stringify(o)}\n\n`);
const tool = (round, tag) => [
["read", { path: "big.txt", offset: round * 10 + 1, limit: 40 }],
["grep", { pattern: "line 1[0-9]*7 ", include: "*.txt", limit: 20 }],
["shell", { command: "seq 1 200" }],
["write", { path: `out/${tag}-${round}.txt`, content: `round ${round}\n`.repeat(50) }],
][round % 4];
http.createServer((req, res) => {
let body = "";
req.on("data", (d) => (body += d));
req.on("end", () => {
if (!req.url.endsWith("/chat/completions")) return res.writeHead(404).end();
let p = {}; try { p = JSON.parse(body); } catch {}
const msgs = p.messages ?? [], rounds = msgs.filter((m) => m.role === "tool").length;
const first = msgs.find((m) => m.role === "user");
const tag = String(typeof first?.content === "string" ? first.content : "s").replace(/[^a-z0-9]+/gi, "-").slice(0, 24);
const base = { id: `c${++served}`, object: "chat.completion.chunk", created: 1, model: "slow" };
active++; let done = false; const fin = () => { if (!done) { done = true; active--; } };
res.on("close", fin);
res.writeHead(200, { "content-type": "text/event-stream" });
sse(res, { ...base, choices: [{ index: 0, delta: { role: "assistant", content: "" }, finish_reason: null }] });
const end = (reason) => { sse(res, { ...base, choices: [{ index: 0, delta: {}, finish_reason: reason }], usage: { prompt_tokens: 10, completion_tokens: 10, total_tokens: 20 } }); res.end("data: [DONE]\n\n"); fin(); };
if (p.tools?.length && rounds < steps) {
const [name, args] = tool(rounds, tag);
return setTimeout(() => { sse(res, { ...base, choices: [{ index: 0, delta: { tool_calls: [{ index: 0, id: `call_${served}`, type: "function", function: { name, arguments: JSON.stringify(args) } }] }, finish_reason: null }] }); end("tool_calls"); }, 50);
}
setTimeout(() => { sse(res, { ...base, choices: [{ index: 0, delta: { content: "done" }, finish_reason: null }] }); end("stop"); }, 50);
});
}).listen(port, "127.0.0.1");
setInterval(() => console.log(`${new Date().toISOString()} provider_requests=${served} open_streams=${active}`), 5000);
EOF
node "$ROOT/fake-provider.mjs" "$PORT" "$STEPS" > "$ROOT/provider.log" 2>&1 &
PROVIDER=$!
# --- 3. A small git repository for the agents to work in ------------------------------------------
WORK="$ROOT/work"
if [ ! -d "$WORK/.git" ]; then
mkdir -p "$WORK/out" && cd "$WORK" && seq 1 60000 | sed 's/^/line /' > big.txt
for d in $(seq 1 50); do mkdir -p "src/m$d"; for f in $(seq 1 60); do seq "$f" 40 | sed "s/^/line m$d f$f /" > "src/m$d/f$f.txt"; done; done
git init -q && git add -A && git -c user.name=r -c user.email=r@example.invalid commit -qm init; cd "$ROOT"
fi
# --- 4. Start the sandbox background service -------------------------------------------------------
"$OC" service start >/dev/null || exit 1
reg() { node -e 'try{const r=require(process.argv[1]);console.log(r[process.argv[2]])}catch{console.log("")}' "$REG" "$1"; }
URL=$(reg url); PW=$(reg password); PID0=$(reg pid)
api() { # api PATH [JSON body => POST] [max seconds]
local method=GET; [ -n "${2:-}" ] && method=POST
curl -s -m "${3:-30}" -u "opencode:$PW" -H 'content-type: application/json' -X "$method" ${2:+-d "$2"} "$URL$1"
}
log "service pid=$PID0 url=$URL"
# --- 5. Load: SESSIONS agent sessions, started in waves -------------------------------------------
start_session() {
local id
id=$(api /api/session "{\"location\":{\"directory\":\"$WORK\"},\"model\":{\"providerID\":\"fake\",\"id\":\"slow\"},\"permissions\":[{\"action\":\"*\",\"resource\":\"*\",\"effect\":\"allow\"}]}" | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{console.log(JSON.parse(s).data.id)}catch{}})')
[ -n "$id" ] && api "/api/session/$id/prompt" "{\"text\":\"task $1\"}" >/dev/null && echo "$id" >> "$ROOT/sessions.txt"
}
( for ((i = 0; i < SESSIONS; i += WAVE)); do
for ((j = i; j < i + WAVE && j < SESSIONS; j++)); do start_session "$j" & done; wait
sleep "$WAVE_GAP"
done ) > /dev/null 2>&1 &
LOADER=$!
# --- 6. Watch /api/info back to back. Default: 2 s timeout, like the client's probe.
# CONTROL=1: no client at all and a 60 s timeout, to measure how long the busy service
# really takes to answer when nobody kills it.
CONTROL=${CONTROL:-0}
[ "$CONTROL" = 1 ] && PROBE_TIMEOUT=${PROBE_TIMEOUT:-60} || PROBE_TIMEOUT=${PROBE_TIMEOUT:-2}
( streak=0; down=0
while kill -0 "$PROVIDER" 2>/dev/null; do
r=$(curl -s -o /dev/null -m "$PROBE_TIMEOUT" -u "opencode:$PW" -w '%{http_code} %{time_total}' "$URL/api/info"); rc=$?
t=${r#* }
if [ "$rc" = 28 ]; then streak=$((streak + 1))
elif [ "$rc" = 0 ]; then
[ "$streak" -gt 0 ] && echo "$(date -u +%T.%3N) /api/info timed out (${PROBE_TIMEOUT}s) $streak time(s) in a row, then answered in ${t}s"
awk -v t="$t" 'BEGIN{exit !(t > 2)}' && echo "$(date -u +%T.%3N) /api/info answered after ${t}s"
streak=0; down=0
else
[ "$down" = 0 ] && echo "$(date -u +%T.%3N) /api/info connection refused (curl exit $rc): service process gone"
streak=0; down=1
fi
sleep 0.025
done ) > "$ROOT/probe.log" 2>&1 &
PROBER=$!
if [ "$CONTROL" = 1 ]; then
log "control run: no client for ${DURATION}s, /api/info timeout ${PROBE_TIMEOUT}s"
end=$(( $(date +%s) + DURATION ))
while [ "$(date +%s)" -lt "$end" ]; do sleep 30; log "provider: $(tail -1 "$ROOT/provider.log" | cut -d' ' -f2-); sessions started: $(wc -l < "$ROOT/sessions.txt" 2>/dev/null)"; done
kill "$LOADER" 2>/dev/null
log "service pid now $(reg pid) (started as $PID0); process alive: $(kill -0 "$PID0" 2>/dev/null && echo yes || echo no)"
echo "--- slowest /api/info answers (a stall over ~6 s = three 2 s client probes in a row = replacement) ---"
grep 'answered after' "$ROOT/probe.log" | sort -t' ' -k5 -g | tail -8
echo "answers slower than 6 s: $(grep 'answered after' "$ROOT/probe.log" | awk '{gsub("s$","",$5); if ($5 > 6) n++} END{print n+0}')"
"$OC" service stop >/dev/null 2>&1; kill "$PROBER" "$PROVIDER" 2>/dev/null
pkill -f "$ROOT/home/.cache/opencode/bin/opencode-pty" 2>/dev/null
log "sandbox kept in $ROOT (rm -rf it when done)"; exit 0
fi
# --- 7. One-shot clients, back to back, until one replaces the service ----------------------------
deadline=$(( $(date +%s) + DURATION )); calls=0; replaced=""
while [ "$(date +%s)" -lt "$deadline" ]; do
calls=$((calls + 1)); t0=$(date +%s%N)
"$OC" api get /api/session >/dev/null 2>&1
ms=$(( ($(date +%s%N) - t0) / 1000000 ))
now=$(reg pid)
[ $((calls % 20)) -eq 0 ] && log "client call #$calls took ${ms} ms; provider: $(tail -1 "$ROOT/provider.log" | cut -d' ' -f2-); $(grep -c 'in a row' "$ROOT/probe.log") probe timeout runs so far"
if [ -n "$now" ] && [ "$now" != "$PID0" ]; then replaced=$now; break; fi
done
# --- 8. Report --------------------------------------------------------------------------------------
kill "$LOADER" 2>/dev/null
echo
if [ -z "$replaced" ]; then
log "NOT REPRODUCED within ${DURATION}s ($calls client calls). Longest probe-timeout runs:"
grep 'in a row' "$ROOT/probe.log" | sort -t' ' -k6 -n | tail -5
else
log "REPRODUCED: client call #$calls (${ms} ms) replaced the service: pid $PID0 -> $replaced"
log "old service process still running: $(kill -0 "$PID0" 2>/dev/null && echo yes || echo no)"
echo "--- client/service log ---"
grep -E 'unresponsive|shutdown started|args="\[\\"(api|serve)' "$SLOG" | tail -5 | sed -E 's/ run=[0-9a-f]+//'
echo "--- independent /api/info probe (2 s, back to back) around the replacement ---"
tail -6 "$ROOT/probe.log"
echo "--- provider traffic around the replacement (model calls the service made every 5 s) ---"
tail -8 "$ROOT/provider.log"
sleep 20
echo "--- sessions told 'The server restarted while you were working' (sample of 10) ---"
n=0; for s in $(head -10 "$ROOT/sessions.txt"); do api "/api/session/$s/message?limit=5" "" 60 | grep -q '"notice":"restart"' && n=$((n + 1)); done
echo "$n/10 sampled sessions carry the restart notice (of $(wc -l < "$ROOT/sessions.txt") sessions running)"
fi
# --- 9. Cleanup (keeps $ROOT for inspection) -------------------------------------------------------
"$OC" service stop >/dev/null 2>&1
kill "$PROBER" "$PROVIDER" 2>/dev/null
pkill -f "$ROOT/home/.cache/opencode/bin/opencode-pty" 2>/dev/null
log "sandbox kept in $ROOT (rm -rf it when done)"Screenshot
No response
Operating System
Ubuntu 24.04 (kernel 7.0, x86_64, 8 vCPU VM); first seen on Windows 11
Terminal
Not relevant (scripted opencode api calls)
Activity
opencodejr commentedon Nov 16, 2026
Seen with a simulated model or a mock server, not a real provider; a maintainer should confirm.
Thanks for the detailed report and script. I reproduced this on 2.3.0 on Linux, and it still happens on the latest
dev(6a2300d).The setup was a sandboxed service, a local mock OpenAI-compatible provider where every turn is a tool call (the model was simulated), and about 1,200 tool-loop sessions started through the HTTP API. I ran
opencode api get /api/sessiononce a second. As load grew, calls took 6 to 16 s. Then one call loggedBackground service is unresponsive; recovery cannot preserve persistent terminals. The old service loggedshutdown started signal=SIGTERMabout 0.3 to 0.5 s later, and a new service was registered.As in your runs, an independent
/api/infoprobe got an answer from the old process between the warning and the SIGTERM:The script below is a shortened version of yours with the load step filled in. The load ramps faster, and
SFX=-localmakes it work when OpenCode runs from a source checkout. Related: #58111 (the same kill path, triggered by host sleep), and the open PRs #58130 and #63729, which change this recovery path.Reproduction script (repro.sh)
Run it:
bash repro.sh