Skip to content

pty: shell commands leave zombies under opencode-pty daemon on Linux (2.3.0) #63817

Closed

Description

@lawzava

Summary

Since upgrading from 2.2.7 to 2.3.0, every finished shell command I've checked stays a zombie under opencode-pty daemon. The calls complete normally, with results and exit codes, but the processes are never reaped, while the same workload on 2.2.7 left none.

Environment

  • OpenCode 2.3.0, upgraded from 2.2.7 on 13 Nov 2026
  • Ubuntu 26.04.1 LTS, Linux x86_64, kernel 7.0.0-43-generic
  • opencode serve --hostname 127.0.0.1 --port 4096, running as a systemd user unit, not in a container or as PID 1
  • A client drives sessions over the HTTP API
  • PTY binary: ~/.cache/opencode/bin/opencode-pty/0.5.2-<hash>/opencode-pty

Steps to reproduce

  1. Start opencode serve 2.3.0 on Linux.

  2. Have a session run a few shell tool calls, such as git status and ls | head.

  3. After they finish, run:

    ps -eo pid,ppid,stat,comm | awk '$3 ~ /^Z/'
  4. Check the PPID of the remaining zombies. It points to opencode-pty daemon, a child of the server.

Expected

The daemon reaps finished children and any reparented descendants.

Actual

Finished commands and orphaned descendants accumulate as <defunct> processes with state Z.

After about 107 minutes, the daemon had 156 zombies, including 102 bash, 10 gh, 9 git, 8 sed, 7 grep and 4 go processes. Its /proc/<pid>/status showed SigCgt: 0000000100000440, which does not include SIGCHLD.

A separate reproduction with a scratch HOME and fresh config also left every finished command as a zombie for 10+ minutes on 2.3.0; 2.2.7 left none.

Restarting the server clears them, but until then they hold PIDs and count against the systemd unit's TasksMax, so a long-running server will eventually fail to fork (not reached here yet).

Not known

  • Whether the daemon sets PR_SET_CHILD_SUBREAPER. Grandchildren being reparented to it suggests this, but I haven't checked the source.
  • Whether persistent TUI terminals have the same problem. Only shell tool calls were checked.

Related

#53564 concerns orphaned grandchildren under container PID 1 on 2.0.24. Here the parent is opencode-pty daemon on a normal Linux host.

Activity

  1. opencodejr commented on Nov 13, 2026

    @opencodejr
    Contributor

    Thanks for the detailed report. I tried this on 2.3.0 on Linux with opencode serve and a fresh HOME. Commands came from the HTTP API (POST /api/shell) and from real shell tool calls in a session.

    What I saw:

    • Finished commands do stay as <defunct> children of opencode-pty daemon, and the daemon's SigCgt matches yours. 2.2.7 left none, and the latest dev (68fd7dd) behaves like 2.3.0.
    • The count never went past 25. The oldest ones were reaped as new commands finished. I tried 60 sequential commands, 4 projects with 30 commands each, 40 timed-out commands, 40 tool calls in one session, and several server restarts.
    • That cap looks intentional. The daemon keeps an exited command as a zombie until it is reaped, so its PID stays reserved, and the server keeps the 25 most recent exited commands available to look up. When a command is evicted from that list, it is reaped.
    • An orphaned grandchild (sh -c 'sleep 0.1 &') was reparented to PID 1, not to the daemon, so I saw no sign that the daemon is a subreaper.

    So I could reproduce a fixed number of zombies, but not the unbounded growth you saw (156 after about 107 minutes). That growth would be the real bug. Could you help narrow down what in your setup gets past the cap?

    • Does the zombie count keep rising past 25 under a single daemon PID? Output of ps -eo pid,ppid,stat,lstart,comm | awk '$3 ~ /^Z/' at two points in time would help.
    • How many opencode-pty daemon processes are running (pgrep -af 'opencode-pty daemon')?
    • What your client does: how many sessions and directories or worktrees at once, and whether it uses background shell commands, subagents, aborts or timeouts.
    • Anything in the server log around shell commands, e.g. warnings about the process daemon.

    You can reply here or edit the issue. If you edit it, please also leave a comment saying you've updated it.

  2. iehgit commented on Nov 13, 2026

    @iehgit

    Independent reproduction on the TUI path, with the count far past 25 (still running).

    • OpenCode v2.3.0 (npm latest), Linux, TUI (opencode serve --service →
      opencode-pty 0.5.2 daemon), not a container, no systemd unit.
    • Single daemon: pgrep -af 'opencode-pty daemon' → one PID (1564),
      started 17:26:58 local, 2h11m old at the samples below.
    • Five samples between 19:30:37 and 19:38:21 local (UTC+1) all showed the
      daemon with exactly 100 zombie children. Between samples I ran 10+ short
      shell tool calls; the newest zombie moved each time (latest 15093) and the
      total stayed at 100, so entries are recycled but the pool is 100, not 25.
    • The oldest cohort is never recycled: 25 zombies from 18:06:18–18:10:40,
      including PIDs 4975/4993/5028, were still present and unreaped 92 minutes
      later, across all five samples.
    • Current mix: 78 bash, 13 cat, 3 ls, 2 grep, 1 each sh/sed/mkdir/head.
    • Every shell tool call leaves at least one zombie: sh -c 'echo mypid=$$'
      printed 11367, and a second later ps -p 11367 showed Z with PPID 1564.
    • Daemon threads grew 99 → 116 → ~120 during observation.
    • SigCgt: 0000000100000440 (no SIGCHLD), matching the original report.
    • 2.2.7 left none (matches the report); restarting the daemon clears them.
  3. jotenakis2 commented on Nov 13, 2026

    @jotenakis2

    Third independent reproduction, this time on aarch64, and I can answer the follow-up questions: the count does go past 25 under a single daemon PID.

    Environment

    • OpenCode v2.3.0 (TUI, opencode serve --service → opencode-pty 0.5.2 daemon), upgraded from 2.2.7.
    • Arch Linux, kernel 7.2.11-14-1-ARCH, aarch64 (Omarchy/mac ARM).
    • zsh, TERM=xterm-256color.
    • 2.2.7 left no zombies in the same setup; the daemon binary is ~/.cache/opencode/bin/opencode-pty/0.5.2-6c313f2614a8329e/opencode-pty.

    One daemon, and the count keeps rising past 25

    pgrep -af 'opencode-pty daemon' → a single PID (3262238), child of opencode serve --service (2.3.0), started 22:52:29 local.

    # sample A, 23:16:28 -> 38 zombie children
    # sample B (2 min later, a few more shell tool calls) -> 40
    ps -eo ppid,pid,stat,lstart,comm | awk '$3 ~ /^Z/ && $1==3262238' | wc -l
    
    • Growth: 21 zombies after ~24 min of light use, 38 after ~50 min, 40 after ~52 min. It is monotonic, 1–3 children per shell tool call (zsh, plus the occasional helper: sed, ls, readlink).
    • The oldest one (3262248, started 22:52:29, right after the daemon started) is still there 25 minutes later and was never reaped. So no eviction: the pool is not recycled, it just grows. I have not reached 100 yet, so I can't confirm or refute the 100 cap from the other report.
    • Daemon /proc/3262238/status: SigCgt: 0000000100000440 (no SIGCHLD, same as yours), SigBlk: 0000000000004002, SigIgn: 0000000000001000. Threads grew 45 → 55 during observation, VmRSS 45 MB, 71 fds.

    What the client does (nothing exotic)

    • One TUI session, cwd = my home directory, not a git repo, no worktrees.
    • Only synchronous foreground shell tool calls. No background shell commands, no timeouts, no aborts, no subagents during the observation window.
    • The sessions in question were already long-lived: the 2.3.0 TUI was started while an older 2.2.7 TUI session was still open, so the shared serve --service daemon was restarted under 2.3.0 in the middle of the day. Once the new TUI was up, every shell call started leaving a zsh zombie.
    • Unrelated, for completeness: the only zombie not owned by the daemon is one zypak-sandbox from Vesktop (Electron), not opencode.

    Logs

    Nothing about the process daemon. The log has 63 level=WARN|ERROR lines in total, none matching pty|process daemon|process-daemon|reap|zombie. For 2.3.0 there is no message="spawning process" entry at all for shell tool calls (2.2.7 logged those), so shell spawns are not traceable in ~/.local/share/opencode/log/opencode.log anymore — that made correlating spawns with zombies harder than expected.

    Reproduction

    1. Start a 2.3.0 TUI on Linux/arm64 (opencode serve --service spawns the 0.5.2 daemon).
    2. Run a handful of shell tool calls.
    3. ps -eo ppid,pid,stat,lstart,comm | awk '$3 ~ /^Z/'
    4. Every finished call is a <defunct> child of the daemon, and the count only goes up.

    Impact is the same as reported: nothing visible for the user, but PIDs are held and it grows until the daemon/server restarts.

  4. opencodejr commented on Nov 14, 2026

    @opencodejr
    Contributor

    Closing this for now, since we haven't heard back. On 2.3.0 and the latest dev, finished shell commands did stay as <defunct> children of opencode-pty daemon. But the count stopped at 25 and the oldest were reaped as new commands finished, so I couldn't reproduce the unbounded growth you saw (156 after about 107 minutes). To take it further, we'd need to know what in your setup gets past that cap:

    • the zombie count under one daemon PID at two points in time
    • how many opencode-pty daemon processes are running
    • how your client uses sessions: directories or worktrees, background commands, subagents, aborts or timeouts

    If you can share any of that, or if it still happens on a newer version, reply here and the issue will reopen.

  5. nakatam1337 commented on Nov 14, 2026

    @nakatam1337

    Confirmed: unbounded zombie growth under a single opencode-pty daemon PID, well past 25, on OpenCode 2.3.0.

    Our observation

    • OpenCode CLI: opencode v2.3.0
    • opencode-pty daemon: 0.5.2-bc9cb0fa0ebb4e56, single PID 2674 (child of opencode serve --service, PPid 518)
    • OS: Debian GNU/Linux 13 (trixie), kernel 7.3.3-1-cachyos x86_64
    • Client: OpenCode TUI, one long-lived session, foreground shell tool calls only (no background commands, no subagents, no timeouts during observation)

    Zombie census under daemon PID 2674 (81 total children, 79 zombies)

    Count Command
    59 bash
    8 tmux: client
    6 ls
    1 tail
    1 ps
    1 openssl
    1 go
    1 git
    1 curl

    Growth is monotonic and unbounded

    Zombies span the full session lifetime and none are ever reaped:

    • Oldest: PID 2841 [curl] — 39,849s (~11.1 hours), still present
    • PID 3015 [bash] — 39,541s (~11.0 hours), still present
    • PID 3017 [ls] — 39,541s (~11.0 hours), still present
    • Newest: PID 72227 [bash] — 17s

    The oldest cohort (from daemon startup ~04:14 UTC) is still unreaped 11+ hours later, so the pool is not recycled — it just grows. Count observed at 79 and rising.

    Daemon state

    /proc/2674/status:
      State: S (sleeping)
      Tgid: 2674
      PPid: 518
      Threads: 97
      VmSize: 4659672 kB
      SigPnd: 0000000000000000
      SigBlk: 0000000000004002
      SigIgn: 0000000000001000
      SigCgt: 0000000100000440   (no SIGCHLD)
    

    Open fds: 118 (mostly pipes and unix sockets)

    Binary notes

    The opencode-pty binary contains waitpid, waitid, wait4, sigwait, posix_spawnp, pidfd_spawnp, and strings like process has been reaped and terminal child waiter, so reaping logic exists but is not applied to direct daemon children.

    Comparison with 2.2.7

    2.2.7 left no zombies in the same setup; the leak appeared with 2.3.0 / opencode-pty 0.5.2. Restarting the daemon clears them.

    This matches what iehgit (100 zombies, 2h11m, single daemon) and jotenakis2 (aarch64, monotonic growth past 25, oldest unreaped 25 min) reported. The contributor's cap-of-25 reproduction was not seen here — under one daemon PID we exceed 25 by a large margin with no eviction.

    I'm happy to provide the full per-PID zombie list (ages 17s to 11h) if useful.

  6. nakatam1337 commented on Nov 14, 2026

    @nakatam1337

    Requesting reopen: we provided the missing data that was blocking this issue.

    The original author of #63817 never replied with the follow-up details (zombie count under one daemon PID at two points in time, etc.), so it was closed for inactivity. We have now provided exactly that data on this thread:

    • Single opencode-pty daemon PID (2674)
    • 79 zombie children — well past the 25 cap the contributor could not reproduce
    • Oldest zombie ~11 hours old (39,849s) and still unreaped → pool is not recycled, it grows monotonically
    • Full per-PID zombie list (ages 17s to 11h)
    • Daemon /proc state: SigCgt has no SIGCHLD, Threads: 97, VmSize: 4.6 GB
    • Reproduced on OpenCode 2.3.0 / opencode-pty 0.5.2-bc9cb0fa0ebb4e56, Debian 13
    • 2.2.7 left no zombies; restarting the daemon clears them

    This matches the independent reproductions by iehgit (100 zombies, 2h11m) and jotenakis2 (aarch64, monotonic past 25). The contributor's cap-of-25 reproduction was not seen in our setup.

    With this data, the unbounded-growth bug is confirmed under a single daemon PID. Please reopen #63817 so it can be tracked and fixed rather than closed as a duplicate of a closed issue.

  7. nakatam1337 commented on Nov 14, 2026

    @nakatam1337

    @opencodejr — could you please reopen #63817?

    We provided all the data you asked for on our duplicate #63948 (which is closed as a duplicate of this one):

    • Single opencode-pty daemon PID with 79 zombie children — well past the 25 cap you couldn't reproduce
    • Oldest zombie ~11 hours old and still unreaped → the pool is not recycled, it grows monotonically
    • Full per-PID zombie list (ages 17s to 11h), daemon /proc state, signal masks (SigCgt has no SIGCHLD)
    • Reproduced on OpenCode 2.3.0 / opencode-pty 0.5.2-bc9cb0fa0ebb4e56, Debian 13
    • 2.2.7 left no zombies; restarting the daemon clears them

    This matches the independent reproductions by iehgit (100 zombies, 2h11m) and jotenakis2 (aarch64, monotonic past 25).

    Since the original author never replied and this issue closed for inactivity, could you reopen it so the unbounded zombie growth can be tracked and fixed? Our data is right here on this thread.

  8. lawzava commented on Nov 15, 2026

    @lawzava
    Author

    Sorry for the slow reply. It's still happening here, past 25 under one daemon, so could you reopen this?

    Checked on 15 Nov, all times EET. OpenCode 2.3.0, opencode-pty 0.5.2-bc9cb0fa0ebb4e56, Ubuntu 26.04.1, kernel 7.0.0-43-generic, x86_64, with opencode serve running as a systemd user unit, not in a container.

    1. Zombie count under the same daemon PID:

      Sample Time Since daemon start Zombies
      A 02:31:27 14 min 34 s 58
      B 02:42:00 25 min 7 s 64

      Some do get reaped, but not oldest first. Between A and B, 26 zombies started between 02:22 and 02:31 were reaped. The 6 oldest, started between 02:16:53 and 02:19, are still there, and the first of them started in the same second as the daemon.

      Counted only zombie children of that daemon ($DAEMON is its PID):

      ps -eo ppid,pid,stat,lstart,comm | awk -v d=$DAEMON '$3 ~ /^Z/ && $1==d'

      Sample A was mostly bash (35), plus git, grep, gh, sed, ls and a few other short commands. The original report reached 156 after about 107 minutes on the same setup.

    2. Exactly one opencode-pty daemon, checked with pgrep -af 'opencode-pty daemon'. The server restarted at 02:16:10 and the daemon started at 02:16:53.

    3. A client drives the server over the HTTP API, with about 4 to 6 concurrent sessions across several directories, including git worktrees. Sessions also start child task sessions, which run in the background in 2.3.0. Shell calls are mostly foreground, aborts happen when a session is interrupted, and I haven't checked timeouts in this window.

    4. No WARN or ERROR lines mentioning pty, process daemon or reap in the server log. The daemon had 66 threads, about 25 MB RSS, 77 open fds and SigCgt: 0000000100000440 (no SIGCHLD).

  9. opencodejr commented on Nov 15, 2026

    @opencodejr
    Contributor

    Thanks for the extra data, and to @iehgit and @jotenakis2 for theirs. It made the difference: I've now reproduced this on 2.3.0, and it still happens on the latest dev (38bd737).

    What I found:

    • A finished shell command stays a zombie under opencode-pty daemon until the server evicts it from its list of recent exited commands. That list keeps the newest 25 per directory, not per daemon. My earlier test ran everything in one directory, which is why I only ever saw 25.
    • With 30 commands in each of 6 git projects on one opencode serve, the single daemon went 25 → 50 → 75 → 100 → 125 → 150 zombies. The count stayed at 150 after a minute with no new commands.
    • Zombies in a directory are only reaped when a newer command finishes in that same directory. A directory or worktree that goes idle keeps its 25 until the server restarts. That matches what you saw: totals well past 25, and the oldest ones never reaped.
    • The same happens with real shell tool calls in a session. 2.2.7 did not run commands in the daemon and left none.

    The script below reproduces it over the HTTP API, with no model needed. I haven't yet tested the single-directory growth one commenter reported, so that may have an additional cause. Restarting the server clears the zombies in the meantime.

    Reproduction script (repro.sh)
    #!/usr/bin/env bash
    # #63817: on `opencode serve` 2.3.0, finished shell commands stay <defunct> children of `opencode-pty daemon`.
    # Only the newest 25 exited commands *per directory* are ever reaped, so the total grows with every
    # directory (or git worktree) the server runs commands in, and nothing is reaped for directories that go idle.
    # Runs PER commands in each of DIRS git projects over the HTTP API (POST /api/shell, no model needed),
    # counting zombie children of the single daemon after each directory.
    # Usage: ./repro.sh                                              (opencode from PATH)
    #        OPENCODE="bun ... packages/cli/src/index.ts" ./repro.sh   (source checkout)
    # Needs curl and jq.
    set -uo pipefail
    OPENCODE=${OPENCODE:-opencode}
    PORT=${PORT:-4310}
    DIRS=${DIRS:-6}
    PER=${PER:-30}
    mkdir -p /tmp/opencode
    ROOT=$(mktemp -d /tmp/opencode/zombie.XXXXXX)
    mkdir -p "$ROOT/home"
    export HOME=$ROOT/home XDG_CONFIG_HOME=$ROOT/home/.config XDG_DATA_HOME=$ROOT/home/.local/share \
      XDG_CACHE_HOME=$ROOT/home/.cache XDG_STATE_HOME=$ROOT/home/.local/state
    for d in $(seq "$DIRS"); do
      mkdir -p "$ROOT/p$d" && git -C "$ROOT/p$d" init -q &&
        git -C "$ROOT/p$d" -c user.email=a@b -c user.name=a commit -q --allow-empty -m "init $d"
    done
    $OPENCODE --version
    (cd "$ROOT/p1" && exec $OPENCODE serve --hostname 127.0.0.1 --port "$PORT") > "$ROOT/serve.log" 2>&1 &
    SERVER=$!
    trap 'kill $SERVER 2>/dev/null || true' EXIT
    for _ in $(seq 60); do grep -q "server password" "$ROOT/serve.log" 2>/dev/null && break; sleep 1; done
    PASS=$(sed -n 's/^server password //p' "$ROOT/serve.log")
    
    run() { # directory command
      curl -s -g -m 30 -o /dev/null -u "opencode:$PASS" -H 'content-type: application/json' -X POST \
        "http://127.0.0.1:$PORT/api/shell?location[directory]=$1" -d "$(jq -n --arg c "$2" '{command:$c}')"
    }
    count() { ps -eo ppid,stat | awk -v d="$1" '$1==d && $2 ~ /^Z/' | wc -l; }
    
    for d in $(seq "$DIRS"); do
      for i in $(seq "$PER"); do run "$ROOT/p$d" "git status --short; ls | head -1"; done
      sleep 3
      DAEMON=$(pgrep -f "^/.*opencode-pty daemon --runtime-dir $ROOT")
      echo "after $PER commands in each of $d directories: daemons [$DAEMON], zombie children of daemon: $(count "$DAEMON")"
    done
    echo "waiting 60s with no new commands..."
    sleep 60
    echo "60s later: zombie children of daemon: $(count "$DAEMON")"
    echo "--- 5 oldest zombie children ---"
    ps -eo ppid,pid,stat,lstart,comm --sort=start_time | awk -v d="$DAEMON" '$3 ~ /^Z/ && $1==d' | head -5
    echo "daemon $DAEMON $(grep SigCgt /proc/$DAEMON/status)"

    Run it: bash repro.sh

  10. alex-yorik commented on Nov 16, 2026

    @alex-yorik

    Confirmed on v2.3.0, Ubuntu 24.04 (kernel 6.8.0-149), opencode-pty daemon 0.5.2-bc9cb0fa0ebb4e56.

    Repro size: in a single live session, ~20 shell tool calls left 25 zombie processes ([bash], [grep], [w], [nproc], [swapon], [who]) — roughly one unreaped child per shell tool invocation. Parent is the opencode-pty daemon; children exit but are never wait()ed.

    Proc evidence during the session:

    • zombie count grows linearly with the number of shell tool calls, does not plateau within the session
    • no ps-visible cleanup after commands finish, no eviction observed in ~10 minutes of idle

    Environment: root shell via opencode TUI on pts/0, server role running as opencode serve --service.

    This matches the behavior described here and in #63976 / #53564 — same accumulation pattern, Linux-only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions