Skip to content

service: /api/info takes seconds while the service is busy, so clients replace a live service and kill all runs #64274

Open

Description

@pascalandr

Description

When the background service is busy with many parallel agent sessions, GET /api/info often takes more than 2 s. It sometimes takes 10 to 28 s, while the service keeps working and then answers again by itself. Any client that calls Service.ensure() during one of these stalls (opencode api …, the TUI, opencode run) counts three 2 s probe timeouts. It then logs Background service is unresponsive; recovery cannot preserve persistent terminals, sends SIGTERM and then SIGKILL to the service, and starts a new one. Every in-flight turn is interrupted, and the new service resumes each session with "The server restarted while you were working…".

The old process was not hung. In each reproduction:

  • An independent /api/info probe got an answer from the old PID within a second of the client deciding to kill it (79 ms in the run with millisecond timestamps).
  • The old service logged shutdown started signal=SIGTERM 0.2 to 2.4 s after the warning, then answered /api/info with 503 while it shut down.
  • Right before the stall, it was making 20 to 30 model requests per second to the provider.

With the same load and no client, the service stayed up for the whole run. Its slowest /api/info answer took 9.7 s.

Expected: a client does not kill a live service that is doing work only because /api/info was slow for 6 s. Alternatively, /api/info keeps answering under load.
Actual: the service is replaced after about 6 s (3 × 2 s), and every running session and subagent is interrupted.

In #51216, the bot wrote: "Anything else that blocks the server for more than about 6s could still cause this restart." This report is that case on 2.3.0, reproducible on Linux, with no snapshot-heavy tree and no watcher churn. Related: #58111 (same kill path, triggered by host sleep) and #58130 (30 s window plus liveness check). The longest stall I measured lasted about 28 s (14 consecutive 2 s timeouts, after which the same PID answered), which is close to that window.

I first saw this on Windows 11 with a shared service running many parallel sessions and subagents (about 100 % CPU, 1.5 to 2 GB RSS). Opening the TUI logged the warning about 6 s after start. Later, a one-shot opencode api get /api/session/<id> did the same 6 s later. All in-flight runs were lost both times.

Measured on Linux (8 vCPU VM), with a fake provider where every turn is a tool call (read, grep, shell, write):

Load Main thread /api/info (fresh connection) Replaced by a client
200 sessions streaming text only not saturated ≤ 0.1 s no
150 sessions in tool loops ~100 % ≤ 0.8 s no
300–500 sessions in tool loops 100 % up to 2.7 s; 16 % of probes > 2 s no
900–1000 sessions in tool loops 100 % stalls of 6 to 28 s; 49 runs of ≥ 3 consecutive 2 s timeouts in 19 min yes, 4 times out of 4
same, no client (control, 60 s probe timeout) 100 % max 9.7 s; 31 answers > 2 s service stays up

Client logic in v2.3.0: packages/client/src/effect/service.ts → ensure(). After timeouts.count >= 3, it calls PtyHandoff.clear and terminate() (SIGTERM, 5 s, SIGKILL). requestTimeout: 2_000 is set in service-timing.ts.

Plugins

None

OpenCode version

2.3.0

Steps to reproduce

  1. On Linux x86_64 or arm64, with bash, curl, git, node 18 or later and npm, save the script below as repro.sh.
  2. Run bash repro.sh. It installs @opencode/cli-linux-<arch>@2.3.0 into a temporary sandbox with its own HOME and XDG_* directories, so it never touches an existing service. It also starts a local fake OpenAI-compatible provider (no API key), creates a small git repo, starts the sandbox service and launches 1000 sessions in waves of 100. It then runs opencode api get /api/session repeatedly until the service PID changes.
  3. Optional: CONTROL=1 DURATION=240 bash repro.sh runs the same load with no client and measures how long /api/info really takes.

Output from one run, trimmed (other runs reproduced after 45 s and after 19 min):

[14:37:38] service pid=2534678 url=http://127.0.0.1:49374
[14:40:27] client call #140 took 5828 ms; provider: provider_requests=8612 open_streams=4; 16 probe timeout runs so far
[14:40:58] REPRODUCED: client call #145 (12072 ms) replaced the service: pid 2534678 -> 2599794
--- client/service log ---
14:40:46.916Z INFO  cli starting args=["api","get","/api/session"]
14:40:52.975Z WARN  Background service is unresponsive; recovery cannot preserve persistent terminals
14:40:53.388Z INFO  shutdown started signal=SIGTERM          <- old service, still running its event loop
14:40:58.452Z INFO  cli starting args=["serve","--service"]
--- independent /api/info probe (2 s, back to back) ---
14:40:43 /api/info timed out (2s) 3 time(s) in a row, then answered in 1.384547s
14:40:53 /api/info timed out (2s) 3 time(s) in a row, then answered in 0.416798s   <- old PID answers again
--- provider traffic (model calls made by the service) ---
14:40:45Z provider_requests=9433
14:40:50Z provider_requests=9582
14:40:55Z provider_requests=9693
--- sessions told 'The server restarted while you were working' (sample of 10) ---
10/10 sampled sessions carry the restart notice (of 900 sessions running)
repro.sh
#!/usr/bin/env bash
# Reproduce: under heavy but healthy load, a one-shot `opencode api` client replaces the live
# background service (three 2 s /api/info probe timeouts), which interrupts every running session.
#
# Fully sandboxed: private HOME and XDG_* dirs under $ROOT, a local fake OpenAI-compatible provider
# (no API key, no network calls to model vendors), its own service on its own port. It never talks
# to a background service you may already run.
#
# Needs: bash, curl, git, node >= 18, npm. ~300 MB of disk for the binary, ~200 MB for data.
# Tested on Ubuntu 24.04 x86_64, 8 vCPU VM: replaced after 45 s, 3.3 min and 19 min in three runs.
# CONTROL=1 runs the same load with no client and a 60 s probe timeout: the service stays up.
# Keep ROOT outside your home: project config is also looked up in parent directories.
#
# Usage: bash repro.sh                 # defaults below
#        SESSIONS=1500 DURATION=2400 bash repro.sh
set -u
VERSION=${VERSION:-2.3.0}
SESSIONS=${SESSIONS:-1000}   # parallel agent sessions
WAVE=${WAVE:-100}            # sessions started per wave, one wave every WAVE_GAP seconds
WAVE_GAP=${WAVE_GAP:-20}
STEPS=${STEPS:-150}          # tool-call rounds per session before its final answer
DURATION=${DURATION:-2400}   # give up after this many seconds
PORT=${PORT:-4010}           # fake provider port
ROOT=${ROOT:-$(mktemp -d /tmp/oc-busy-service.XXXXXX)}
mkdir -p "$ROOT"; cd "$ROOT" || exit 1
log() { echo "[$(date -u +%H:%M:%S)] $*"; }

# --- 1. OpenCode, isolated -----------------------------------------------------------------------
case "$(uname -m)" in x86_64) ARCH=x64 ;; aarch64|arm64) ARCH=arm64 ;; *) echo "unsupported arch"; exit 1 ;; esac
if [ ! -x "$ROOT/npm/node_modules/@opencode/cli-linux-$ARCH/bin/opencode" ]; then
  log "installing @opencode/cli-linux-$ARCH@$VERSION into $ROOT/npm"
  npm install --silent --no-audit --no-fund --prefix "$ROOT/npm" "@opencode/cli-linux-$ARCH@$VERSION" || exit 1
fi
OC="$ROOT/npm/node_modules/@opencode/cli-linux-$ARCH/bin/opencode"
export HOME="$ROOT/home" XDG_CONFIG_HOME="$ROOT/home/.config" XDG_DATA_HOME="$ROOT/home/.local/share" \
  XDG_STATE_HOME="$ROOT/home/.local/state" XDG_CACHE_HOME="$ROOT/home/.cache" XDG_RUNTIME_DIR="$ROOT/run"
mkdir -p "$XDG_CONFIG_HOME/opencode" "$XDG_RUNTIME_DIR" && chmod 700 "$XDG_RUNTIME_DIR"
REG="$XDG_STATE_HOME/opencode/service.json"
SLOG="$XDG_DATA_HOME/opencode/log/opencode.log"
log "opencode $("$OC" --version)  sandbox=$ROOT"

cat > "$XDG_CONFIG_HOME/opencode/opencode.json" <<EOF
{
  "\$schema": "https://opencode.ai/config.json",
  "autoupdate": false,
  "share": "disabled",
  "model": "fake/slow",
  "small_model": "fake/slow",
  "provider": {
    "fake": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Fake provider",
      "options": { "baseURL": "http://127.0.0.1:$PORT/v1", "apiKey": "not-a-key" },
      "models": { "slow": { "name": "slow", "limit": { "context": 1000000, "output": 100000 } } }
    }
  }
}
EOF

# --- 2. Fake OpenAI-compatible provider: each turn is a tool call (read/grep/shell/write) ---------
cat > "$ROOT/fake-provider.mjs" <<'EOF'
import http from "node:http";
const [port, steps] = [Number(process.argv[2]), Number(process.argv[3])];
let served = 0, active = 0;
const sse = (res, o) => res.write(`data: ${JSON.stringify(o)}\n\n`);
const tool = (round, tag) => [
  ["read", { path: "big.txt", offset: round * 10 + 1, limit: 40 }],
  ["grep", { pattern: "line 1[0-9]*7 ", include: "*.txt", limit: 20 }],
  ["shell", { command: "seq 1 200" }],
  ["write", { path: `out/${tag}-${round}.txt`, content: `round ${round}\n`.repeat(50) }],
][round % 4];
http.createServer((req, res) => {
  let body = "";
  req.on("data", (d) => (body += d));
  req.on("end", () => {
    if (!req.url.endsWith("/chat/completions")) return res.writeHead(404).end();
    let p = {}; try { p = JSON.parse(body); } catch {}
    const msgs = p.messages ?? [], rounds = msgs.filter((m) => m.role === "tool").length;
    const first = msgs.find((m) => m.role === "user");
    const tag = String(typeof first?.content === "string" ? first.content : "s").replace(/[^a-z0-9]+/gi, "-").slice(0, 24);
    const base = { id: `c${++served}`, object: "chat.completion.chunk", created: 1, model: "slow" };
    active++; let done = false; const fin = () => { if (!done) { done = true; active--; } };
    res.on("close", fin);
    res.writeHead(200, { "content-type": "text/event-stream" });
    sse(res, { ...base, choices: [{ index: 0, delta: { role: "assistant", content: "" }, finish_reason: null }] });
    const end = (reason) => { sse(res, { ...base, choices: [{ index: 0, delta: {}, finish_reason: reason }], usage: { prompt_tokens: 10, completion_tokens: 10, total_tokens: 20 } }); res.end("data: [DONE]\n\n"); fin(); };
    if (p.tools?.length && rounds < steps) {
      const [name, args] = tool(rounds, tag);
      return setTimeout(() => { sse(res, { ...base, choices: [{ index: 0, delta: { tool_calls: [{ index: 0, id: `call_${served}`, type: "function", function: { name, arguments: JSON.stringify(args) } }] }, finish_reason: null }] }); end("tool_calls"); }, 50);
    }
    setTimeout(() => { sse(res, { ...base, choices: [{ index: 0, delta: { content: "done" }, finish_reason: null }] }); end("stop"); }, 50);
  });
}).listen(port, "127.0.0.1");
setInterval(() => console.log(`${new Date().toISOString()} provider_requests=${served} open_streams=${active}`), 5000);
EOF
node "$ROOT/fake-provider.mjs" "$PORT" "$STEPS" > "$ROOT/provider.log" 2>&1 &
PROVIDER=$!

# --- 3. A small git repository for the agents to work in ------------------------------------------
WORK="$ROOT/work"
if [ ! -d "$WORK/.git" ]; then
  mkdir -p "$WORK/out" && cd "$WORK" && seq 1 60000 | sed 's/^/line /' > big.txt
  for d in $(seq 1 50); do mkdir -p "src/m$d"; for f in $(seq 1 60); do seq "$f" 40 | sed "s/^/line m$d f$f /" > "src/m$d/f$f.txt"; done; done
  git init -q && git add -A && git -c user.name=r -c user.email=r@example.invalid commit -qm init; cd "$ROOT"
fi

# --- 4. Start the sandbox background service -------------------------------------------------------
"$OC" service start >/dev/null || exit 1
reg() { node -e 'try{const r=require(process.argv[1]);console.log(r[process.argv[2]])}catch{console.log("")}' "$REG" "$1"; }
URL=$(reg url); PW=$(reg password); PID0=$(reg pid)
api() { # api PATH [JSON body => POST] [max seconds]
  local method=GET; [ -n "${2:-}" ] && method=POST
  curl -s -m "${3:-30}" -u "opencode:$PW" -H 'content-type: application/json' -X "$method" ${2:+-d "$2"} "$URL$1"
}
log "service pid=$PID0 url=$URL"

# --- 5. Load: SESSIONS agent sessions, started in waves -------------------------------------------
start_session() {
  local id
  id=$(api /api/session "{\"location\":{\"directory\":\"$WORK\"},\"model\":{\"providerID\":\"fake\",\"id\":\"slow\"},\"permissions\":[{\"action\":\"*\",\"resource\":\"*\",\"effect\":\"allow\"}]}" | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{console.log(JSON.parse(s).data.id)}catch{}})')
  [ -n "$id" ] && api "/api/session/$id/prompt" "{\"text\":\"task $1\"}" >/dev/null && echo "$id" >> "$ROOT/sessions.txt"
}
( for ((i = 0; i < SESSIONS; i += WAVE)); do
    for ((j = i; j < i + WAVE && j < SESSIONS; j++)); do start_session "$j" & done; wait
    sleep "$WAVE_GAP"
  done ) > /dev/null 2>&1 &
LOADER=$!

# --- 6. Watch /api/info back to back. Default: 2 s timeout, like the client's probe.
#        CONTROL=1: no client at all and a 60 s timeout, to measure how long the busy service
#        really takes to answer when nobody kills it.
CONTROL=${CONTROL:-0}
[ "$CONTROL" = 1 ] && PROBE_TIMEOUT=${PROBE_TIMEOUT:-60} || PROBE_TIMEOUT=${PROBE_TIMEOUT:-2}
( streak=0; down=0
  while kill -0 "$PROVIDER" 2>/dev/null; do
    r=$(curl -s -o /dev/null -m "$PROBE_TIMEOUT" -u "opencode:$PW" -w '%{http_code} %{time_total}' "$URL/api/info"); rc=$?
    t=${r#* }
    if [ "$rc" = 28 ]; then streak=$((streak + 1))
    elif [ "$rc" = 0 ]; then
      [ "$streak" -gt 0 ] && echo "$(date -u +%T.%3N) /api/info timed out (${PROBE_TIMEOUT}s) $streak time(s) in a row, then answered in ${t}s"
      awk -v t="$t" 'BEGIN{exit !(t > 2)}' && echo "$(date -u +%T.%3N) /api/info answered after ${t}s"
      streak=0; down=0
    else
      [ "$down" = 0 ] && echo "$(date -u +%T.%3N) /api/info connection refused (curl exit $rc): service process gone"
      streak=0; down=1
    fi
    sleep 0.025
  done ) > "$ROOT/probe.log" 2>&1 &
PROBER=$!

if [ "$CONTROL" = 1 ]; then
  log "control run: no client for ${DURATION}s, /api/info timeout ${PROBE_TIMEOUT}s"
  end=$(( $(date +%s) + DURATION ))
  while [ "$(date +%s)" -lt "$end" ]; do sleep 30; log "provider: $(tail -1 "$ROOT/provider.log" | cut -d' ' -f2-); sessions started: $(wc -l < "$ROOT/sessions.txt" 2>/dev/null)"; done
  kill "$LOADER" 2>/dev/null
  log "service pid now $(reg pid) (started as $PID0); process alive: $(kill -0 "$PID0" 2>/dev/null && echo yes || echo no)"
  echo "--- slowest /api/info answers (a stall over ~6 s = three 2 s client probes in a row = replacement) ---"
  grep 'answered after' "$ROOT/probe.log" | sort -t' ' -k5 -g | tail -8
  echo "answers slower than 6 s: $(grep 'answered after' "$ROOT/probe.log" | awk '{gsub("s$","",$5); if ($5 > 6) n++} END{print n+0}')"
  "$OC" service stop >/dev/null 2>&1; kill "$PROBER" "$PROVIDER" 2>/dev/null
  pkill -f "$ROOT/home/.cache/opencode/bin/opencode-pty" 2>/dev/null
  log "sandbox kept in $ROOT (rm -rf it when done)"; exit 0
fi

# --- 7. One-shot clients, back to back, until one replaces the service ----------------------------
deadline=$(( $(date +%s) + DURATION )); calls=0; replaced=""
while [ "$(date +%s)" -lt "$deadline" ]; do
  calls=$((calls + 1)); t0=$(date +%s%N)
  "$OC" api get /api/session >/dev/null 2>&1
  ms=$(( ($(date +%s%N) - t0) / 1000000 ))
  now=$(reg pid)
  [ $((calls % 20)) -eq 0 ] && log "client call #$calls took ${ms} ms; provider: $(tail -1 "$ROOT/provider.log" | cut -d' ' -f2-); $(grep -c 'in a row' "$ROOT/probe.log") probe timeout runs so far"
  if [ -n "$now" ] && [ "$now" != "$PID0" ]; then replaced=$now; break; fi
done

# --- 8. Report --------------------------------------------------------------------------------------
kill "$LOADER" 2>/dev/null
echo
if [ -z "$replaced" ]; then
  log "NOT REPRODUCED within ${DURATION}s ($calls client calls). Longest probe-timeout runs:"
  grep 'in a row' "$ROOT/probe.log" | sort -t' ' -k6 -n | tail -5
else
  log "REPRODUCED: client call #$calls (${ms} ms) replaced the service: pid $PID0 -> $replaced"
  log "old service process still running: $(kill -0 "$PID0" 2>/dev/null && echo yes || echo no)"
  echo "--- client/service log ---"
  grep -E 'unresponsive|shutdown started|args="\[\\"(api|serve)' "$SLOG" | tail -5 | sed -E 's/ run=[0-9a-f]+//'
  echo "--- independent /api/info probe (2 s, back to back) around the replacement ---"
  tail -6 "$ROOT/probe.log"
  echo "--- provider traffic around the replacement (model calls the service made every 5 s) ---"
  tail -8 "$ROOT/provider.log"
  sleep 20
  echo "--- sessions told 'The server restarted while you were working' (sample of 10) ---"
  n=0; for s in $(head -10 "$ROOT/sessions.txt"); do api "/api/session/$s/message?limit=5" "" 60 | grep -q '"notice":"restart"' && n=$((n + 1)); done
  echo "$n/10 sampled sessions carry the restart notice (of $(wc -l < "$ROOT/sessions.txt") sessions running)"
fi

# --- 9. Cleanup (keeps $ROOT for inspection) -------------------------------------------------------
"$OC" service stop >/dev/null 2>&1
kill "$PROBER" "$PROVIDER" 2>/dev/null
pkill -f "$ROOT/home/.cache/opencode/bin/opencode-pty" 2>/dev/null
log "sandbox kept in $ROOT (rm -rf it when done)"

Screenshot

No response

Operating System

Ubuntu 24.04 (kernel 7.0, x86_64, 8 vCPU VM); first seen on Windows 11

Terminal

Not relevant (scripted opencode api calls)

Activity

  1. opencodejr commented on Nov 16, 2026

    @opencodejr
    Contributor

    Seen with a simulated model or a mock server, not a real provider; a maintainer should confirm.

    Thanks for the detailed report and script. I reproduced this on 2.3.0 on Linux, and it still happens on the latest dev (6a2300d).

    The setup was a sandboxed service, a local mock OpenAI-compatible provider where every turn is a tool call (the model was simulated), and about 1,200 tool-loop sessions started through the HTTP API. I ran opencode api get /api/session once a second. As load grew, calls took 6 to 16 s. Then one call logged Background service is unresponsive; recovery cannot preserve persistent terminals. The old service logged shutdown started signal=SIGTERM about 0.3 to 0.5 s later, and a new service was registered.

    As in your runs, an independent /api/info probe got an answer from the old process between the warning and the SIGTERM:

    probe: 18:15:00.368 000 (2.0s) / 02.377 000 (2.0s) / 04.386 000 (2.0s)
    18:15:05.054Z WARN  Background service is unresponsive; recovery cannot preserve persistent terminals
    probe: 18:15:05.229 200 (0.83s)   <- old service still answering
    18:15:05.312Z INFO  shutdown started signal=SIGTERM
    REPRODUCED: client call #90 (2207 ms) replaced the service: pid 76318 -> 254225
    

    The script below is a shortened version of yours with the load step filled in. The load ramps faster, and SFX=-local makes it work when OpenCode runs from a source checkout. Related: #58111 (the same kill path, triggered by host sleep), and the open PRs #58130 and #63729, which change this recovery path.

    Reproduction script (repro.sh)
    #!/usr/bin/env bash
    # Under heavy but healthy load, a one-shot `opencode api` client replaces the live background
    # service after three 2 s /api/info probe timeouts. Sandboxed (own HOME/XDG), local fake provider.
    # OC: command to run opencode (default: opencode from PATH).
    set -u
    SESSIONS=${SESSIONS:-300}; WAVE=${WAVE:-50}; WAVE_GAP=${WAVE_GAP:-5}; STEPS=${STEPS:-150}
    DURATION=${DURATION:-900}; PORT=${PORT:-4010}
    ROOT=${ROOT:-$(mktemp -d /tmp/oc-busy.XXXXXX)}; mkdir -p "$ROOT"; cd "$ROOT" || exit 1
    OC=${OC:-opencode}
    log() { echo "[$(date -u +%H:%M:%S)] $*"; }
    export HOME="$ROOT/home" XDG_CONFIG_HOME="$ROOT/home/.config" XDG_DATA_HOME="$ROOT/home/.local/share" \
      XDG_STATE_HOME="$ROOT/home/.local/state" XDG_CACHE_HOME="$ROOT/home/.cache" XDG_RUNTIME_DIR="$ROOT/run"
    mkdir -p "$XDG_CONFIG_HOME/opencode" "$XDG_RUNTIME_DIR" && chmod 700 "$XDG_RUNTIME_DIR"
    SFX=${SFX:-}  # set SFX=-local when OC runs a source checkout (its service registers as service-local.json)
    REG="$XDG_STATE_HOME/opencode/service$SFX.json"; SLOG="$XDG_DATA_HOME/opencode/log/opencode$SFX.log"
    log "opencode $($OC --version) sandbox=$ROOT"
    cat > "$XDG_CONFIG_HOME/opencode/opencode.json" <<EOF
    { "\$schema": "https://opencode.ai/config.json", "autoupdate": false, "share": "disabled",
      "model": "fake/slow", "small_model": "fake/slow",
      "permissions": [ { "action": "*", "resource": "*", "effect": "allow" } ],
      "provider": { "fake": { "npm": "@ai-sdk/openai-compatible", "name": "Fake",
        "options": { "baseURL": "http://127.0.0.1:$PORT/v1", "apiKey": "x" },
        "models": { "slow": { "name": "slow", "limit": { "context": 1000000, "output": 100000 } } } } } }
    EOF
    cat > "$ROOT/fake-provider.mjs" <<'EOF'
    import http from "node:http";
    const [port, steps] = [Number(process.argv[2]), Number(process.argv[3])];
    let served = 0, active = 0;
    const sse = (res, o) => res.write(`data: ${JSON.stringify(o)}\n\n`);
    const tool = (r, tag) => [["read", { path: "big.txt", offset: r * 10 + 1, limit: 40 }],
      ["grep", { pattern: "line 1[0-9]*7 ", include: "*.txt", limit: 20 }],
      ["shell", { command: "seq 1 200" }],
      ["write", { path: `out/${tag}-${r}.txt`, content: `round ${r}\n`.repeat(50) }]][r % 4];
    http.createServer((req, res) => { let body = ""; req.on("data", (d) => (body += d)); req.on("end", () => {
      if (!req.url.endsWith("/chat/completions")) return res.writeHead(404).end();
      let p = {}; try { p = JSON.parse(body); } catch {}
      const msgs = p.messages ?? [], rounds = msgs.filter((m) => m.role === "tool").length;
      const first = msgs.find((m) => m.role === "user");
      const tag = String(typeof first?.content === "string" ? first.content : "s").replace(/[^a-z0-9]+/gi, "-").slice(0, 24);
      const base = { id: `c${++served}`, object: "chat.completion.chunk", created: 1, model: "slow" };
      active++; let done = false; const fin = () => { if (!done) { done = true; active--; } }; res.on("close", fin);
      res.writeHead(200, { "content-type": "text/event-stream" });
      sse(res, { ...base, choices: [{ index: 0, delta: { role: "assistant", content: "" }, finish_reason: null }] });
      const end = (reason) => { sse(res, { ...base, choices: [{ index: 0, delta: {}, finish_reason: reason }], usage: { prompt_tokens: 10, completion_tokens: 10, total_tokens: 20 } }); res.end("data: [DONE]\n\n"); fin(); };
      if (p.tools?.length && rounds < steps) { const [name, args] = tool(rounds, tag);
        return setTimeout(() => { sse(res, { ...base, choices: [{ index: 0, delta: { tool_calls: [{ index: 0, id: `call_${served}`, type: "function", function: { name, arguments: JSON.stringify(args) } }] }, finish_reason: null }] }); end("tool_calls"); }, 50); }
      setTimeout(() => { sse(res, { ...base, choices: [{ index: 0, delta: { content: "done" }, finish_reason: null }] }); end("stop"); }, 50);
    }); }).listen(port, "127.0.0.1");
    setInterval(() => console.log(`${new Date().toISOString()} provider_requests=${served} open_streams=${active}`), 5000);
    EOF
    node "$ROOT/fake-provider.mjs" "$PORT" "$STEPS" > "$ROOT/provider.log" 2>&1 & PROVIDER=$!
    WORK="$ROOT/work"
    if [ ! -d "$WORK/.git" ]; then mkdir -p "$WORK/out" && cd "$WORK" && seq 1 60000 | sed 's/^/line /' > big.txt
      for d in $(seq 1 20); do mkdir -p "src/m$d"; for f in $(seq 1 30); do seq "$f" 40 | sed "s/^/line m$d f$f /" > "src/m$d/f$f.txt"; done; done
      git init -q && git add -A && git -c user.name=r -c user.email=r@example.invalid commit -qm init; cd "$ROOT"; fi
    $OC service start >/dev/null || exit 1
    reg() { node -e 'try{const r=require(process.argv[1]);console.log(r[process.argv[2]])}catch{console.log("")}' "$REG" "$1"; }
    URL=$(reg url); PW=$(reg password); PID0=$(reg pid)
    api() { local m=GET; [ -n "${2:-}" ] && m=POST; curl -s -m "${3:-30}" -u "opencode:$PW" -H 'content-type: application/json' -X "$m" ${2:+-d "$2"} "$URL$1"; }
    log "service pid=$PID0 url=$URL"
    # independent probe: 2 s timeouts, back to back
    ( while :; do o=$(curl -s -m 2 -u "opencode:$PW" -o /dev/null -w "%{http_code} t=%{time_total}" "$URL/api/info")
      echo "$(date -u +%H:%M:%S.%3N) code=$o"; done > "$ROOT/probe.log" 2>&1 ) & PROBE=$!
    start_session() { local i=$1 id
      id=$(api /api/session "{\"title\":\"s$i\",\"location\":{\"directory\":\"$WORK\"}}" | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{console.log(JSON.parse(s).data.id)}catch{console.log("")}})')
      [ -n "$id" ] && api "/api/session/$id/prompt" "{\"text\":\"task $i\"}" >/dev/null; echo "$id" >> "$ROOT/sessions.txt"; }
    ( n=0; while [ $n -lt "$SESSIONS" ]; do for _ in $(seq 1 "$WAVE"); do n=$((n+1)); start_session $n & done; wait; log "started $n sessions" >&2; sleep "$WAVE_GAP"; done ) &
    LOADER=$!
    T0=$(date +%s); i=0
    while [ $(( $(date +%s) - T0 )) -lt "$DURATION" ]; do
      i=$((i+1)); s=$(date +%s%3N); $OC api get /api/session >/dev/null 2>&1; e=$(date +%s%3N)
      PID=$(reg pid); [ $((e-s)) -gt 2000 ] && log "client call #$i took $((e-s)) ms; $(tail -1 "$ROOT/provider.log")"
      if [ -n "$PID" ] && [ "$PID" != "$PID0" ]; then
        log "REPRODUCED: client call #$i ($((e-s)) ms) replaced the service: pid $PID0 -> $PID"
        echo '--- log ---'; grep -E 'cli starting|unresponsive|shutdown started' "$SLOG" | tail -8
        echo '--- probe around ---'; awk '$2!="code=200" || $3+0>1.5' "$ROOT/probe.log" | tail -12
        break; fi
      sleep 1
    done
    kill $PROBE $LOADER $PROVIDER 2>/dev/null; $OC service stop >/dev/null 2>&1

    Run it: bash repro.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions