Skip to content

shell: processes a finished command left in its group are SIGKILLed 25 commands later, not at exit #64177

Closed

Description

@nkoynov

Description

Since 2.3.0, anything a shell command leaves running in its process group is SIGKILLed later, at a moment that has nothing to do with that command, instead of when the command exits (or never, as in 2.2.7).

The daemon keeps an exited command's leader unreaped so its group ID stays reserved. The server keeps the last 25 exited commands per directory (EXITED_LIMIT in packages/core/src/shell.ts), and when another one finishes it removes the oldest. remove calls terminatePipe, which ends in killpg(pgid, SIGKILL) (Process::kill in packages/process-daemon/src/service/pipe/unix.rs). The comment in remove says the group kill is intended ("This also kills what is left of its group, even after a successful exit").

So nohup cmd &, or a helper a plugin starts from a shell command, keeps running for a while and then dies when the 25th later command in the same directory finishes, from whichever session runs it. On our host, with several busy sessions per project, that was anywhere from 1 to 68 minutes after launch, and on one day 83 of 173 background jobs a plugin started this way were killed partway. It was hard to track down because each kill lined up with an unrelated command finishing in another session.

Killing the group's leftovers when the command exits, or leaving them alone, would both be predictable. A delayed kill tied to other commands isn't. #63976 and #64037 are changing how long exited commands are kept (a daemon-wide cap of 100 would tie it to commands in any directory), so it seems worth deciding there too.

Plugins

cursor-opencode-provider (its background-shell supervisor was the leftover process here; our fork now starts it with setsid)

OpenCode version

2.3.0 (the same code on dev cf112e7)

Steps to reproduce

  1. In one directory, run the shell command nohup sleep 600 >/dev/null 2>&1 & and note the sleep's PID.
  2. Run 25 more short commands (true) in the same directory, from any session or POST /api/shell.
  3. When the 25th of them finishes, the first command is evicted and the sleep gets SIGKILL.

Screenshot

n/a

Operating System

NixOS (Linux 6.18)

Terminal

headless opencode serve (driven by T3 Code)

Activity

  1. opencodejr commented on Nov 16, 2026

    @opencodejr
    Contributor

    Thanks for the precise report. I reproduced this on the unmodified 2.3.0 release, using only the HTTP API of a fresh opencode serve:

    1. In one git directory, POST /api/shell with nohup sleep 613 >/dev/null 2>&1 &.
    2. Send 26 more true commands to the same directory.
    3. After each one, check whether the background sleep is still alive.

    On 2.3.0 the sleep survives 24 later commands and disappears right after the 25th finishes, when the first command is evicted. The same script on 2.2.7 leaves it running after all 26, so this is a regression. It still happens on the latest dev (cf112e7), run from source.

    I only tested one directory, so I didn't check the cases with several sessions or other directories. The script below reproduces it.

    Related: #63976 and #63817 (zombies kept for the same retention of exited commands), and #64037, which would change how many exited commands are kept.

    Reproduction script (repro.sh)
    #!/usr/bin/env bash
    # Issue #64177: a process left in a finished shell command's process group is SIGKILLed
    # when that command is evicted (25 later commands in the same directory), not at exit.
    # Usage: OPENCODE="opencode" bash repro.sh
    set -u
    OPENCODE=${OPENCODE:-opencode}
    PORT=${PORT:-4198}
    ROOT=$(mktemp -d /tmp/opencode/evict.XXXX)
    export HOME=$ROOT/home XDG_DATA_HOME=$ROOT/home/.local/share XDG_STATE_HOME=$ROOT/home/.local/state XDG_CACHE_HOME=$ROOT/home/.cache XDG_CONFIG_HOME=$ROOT/home/.config
    mkdir -p "$HOME"
    export OPENCODE_SERVER_PASSWORD=repro-pass
    AUTH="opencode:repro-pass"
    $OPENCODE --version
    cd "$ROOT"; $OPENCODE serve --hostname 127.0.0.1 --port $PORT >"$ROOT/serve.log" 2>&1 &
    SERVER=$!
    cd - >/dev/null
    for i in $(seq 60); do curl -sf -u $AUTH -o /dev/null "http://127.0.0.1:$PORT/api/shell" && break; sleep 1; done
    
    dir=$ROOT/project; mkdir -p "$dir"; (cd "$dir" && git init -q)
    enc=$(python3 -c 'import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1]))' "$dir")
    run() {
      curl -sf -u $AUTH -X POST -H 'content-type: application/json' \
        "http://127.0.0.1:$PORT/api/shell?location%5Bdirectory%5D=$enc" -d "{\"command\":\"$1\"}"
      echo
    }
    run "nohup sleep 613 >/dev/null 2>&1 &" | head -c 300; echo
    sleep 2
    PID=$(pgrep -f "^sleep 613$" | head -1)
    echo "background sleep PID: ${PID:-none}"
    [ -z "$PID" ] && { cat "$ROOT/serve.log" | tail -20; kill $SERVER; exit 1; }
    for c in $(seq 1 26); do
      run "true" >/dev/null
      sleep 0.7
      if kill -0 "$PID" 2>/dev/null && [ "$(ps -o stat= -p $PID | cut -c1)" != Z ]; then st=alive; else st=GONE; fi
      echo "after $c later commands: sleep $PID $st"
    done
    ps -eo pid,ppid,stat,comm | grep -i pty | head -3
    kill $SERVER; wait $SERVER 2>/dev/null
    pkill -f "^sleep 613$" 2>/dev/null

    Run it: bash repro.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions