Skip to content

opencode serve --service event loop deadlock on shared-uid k8s container (overlayfs) — native file watcher subscribe blocks the main thread; only OPENCODE_DISABLE_FILEWATCHER=1 recovers #62687

Open

Description

@sunoldman

opencode serve --service event loop deadlock on shared-uid k8s container (overlayfs) — native file watcher subscribe blocks the main thread; only OPENCODE_DISABLE_FILEWATCHER=1 recovers

Description

opencode serve --service (v2) intermittently becomes permanently unresponsive: the process stays alive and keeps LISTENing on its port, but never answers HTTP again. Live diagnosis of the hung process:

  • Main thread parked in an infinite futex wait: ps -o wchan = futex_wait_queue_me, %CPU = 0.0
  • curl -m8 http://127.0.0.1:<port>/api/info → TCP connects but read times out (no response)
  • No inotify fds held by the process at the time of the hang (/proc/<pid>/fdinfo)
  • The v2 watchdog (opencode-serve-watchdog-v2) health-checks every 10s, gets Read timed out, restarts the service 3 times, then gives up: Auto-restart suspended: service failed repeatedly after 3 restarts — so from the user's perspective the service "cannot start".

Root-cause isolation (A/B/A control on the affected machine, opencode v2.2.5):

state wchan /api/info watchdog
OPENCODE_DISABLE_FILEWATCHER=1 set ep_poll 200 healthy
variable removed, service restarted immediately futex_wait_queue_me timeout Read timed out → restart → suspended loop
variable restored, service restarted ep_poll 200 healthy

So the serve-side file watcher being enabled is both necessary and sufficient for the hang — exactly the pattern reported for v1 in #37111, and matching the inotify-exhaustion root-cause domain of #16610.

Environment specifics that matter: this runs in a shared-uid (uid 1000) Kubernetes dev container with overlayfs, where (per #16610 comments) inotify_add_watch() can fail with ENOSPC even far below max_user_watches, and where the history shows EMFILE: too many open files, watch '<path>' failures. The service hangs shortly after the watcher starts subscribing.

Why the existing guard does not help (serve side)

The serve-side watcher is packages/core/src/filesystem/watcher.ts, which subscribes through the native @parcel/watcher binding (ParcelWatcher, linux backend = inotify):

  • packages/server/src/routes.ts:131-132 — Watcher.node.replace(Watcher.configured({ enabled: options.fs?.filewatcher })) + FileWatch.node.replace(...)
  • packages/core/src/filesystem/watcher.ts:82-88 — layer({ enabled: false }) returns subscribe: () => Effect.succeed(Stream.empty) (fully disables inotify); the native subscribeDirectory(...) path (:246-282) is used otherwise
  • packages/cli/src/server-process.ts:139 — the switch the server reads: OPENCODE_FILEWATCHER_DISABLE ?? OPENCODE_DISABLE_FILEWATCHER

The native subscribe() performs its backend setup synchronously on the calling (main JS) thread before a promise materializes; when that setup wedges (inotify exhaustion / overlayfs ENOSPC / shared-uid instance-pool pressure), the Effect.timeout(SUBSCRIBE_TIMEOUT_MS) never gets a chance to fire — the event loop is already gone (same mechanism as #37111). PR #51536 (fix(tui): degrade gracefully when fs.watch fails) added a safeWatch() degrade path for the TUI (node:fs watch), but the serve side has no equivalent failure tolerance — a wedged native subscribe kills the whole server instead of degrading to "no watcher for that directory".

Plugins

None required — reproduce with --pure/no plugins; happens with the default install (metric/behavior plugins not involved).

OpenCode version

2.2.5 (also reproducible conceptually on 2.2.6+; upstream has no fix in packages/core/src/filesystem/watcher.ts as of the v2.2.5 line — the only later commits are #60955 "stop logging errors when optional watched folders are missing" and #59081 "watch files open in the side panel").

Steps to reproduce

Environment: Linux dev container in a shared-uid k8s pod (uid 1000) on overlayfs; host kernel 3.10 (container userspace Ubuntu 20.04). Low inotify headroom and/or a history of EMFILE: too many open files, watch ... in ~/.local/share/opencode/log/opencode.log.

  1. Start the background service: opencode serve --service (or opencode service start).
  2. Leave the file watcher enabled (no OPENCODE_DISABLE_FILEWATCHER).
  3. Wait — the serve process transitions to wchan=futex_wait_queue_me, HTTP stops responding (curl to /api/info times out), watchdog logs Health check error: ... Read timed out (read timeout=10.0) and eventually Auto-restart suspended.
  4. Set export OPENCODE_DISABLE_FILEWATCHER=1, restart the service → healthy (this is the confirmed workaround; A/B/A verified).

Screenshot

No response

Operating System

Linux (k8s container, shared uid 1000, overlayfs; container userland Ubuntu 20.04.3, host kernel 3.10)

Terminal

code-server web terminal (background service; not TUI — TUI works fine, see PR #51536 scope)

Activity

  1. opencodejr commented on Nov 8, 2026

    @opencodejr
    Contributor

    Thanks for the detailed report and the A/B/A check. I can reproduce this.

    To stand in for a shared-uid container with no inotify instances left, I set fs.inotify.max_user_instances=8 on Linux and had a helper process use up all 8 instances. Then I started opencode serve (2.2.5) and sent requests for a git project:

    • /api/info and /api/session return 200.
    • The first /api/model request never answers, and from then on every request, /api/info included, times out.
    • The main thread sits in a futex wait at 0% CPU, and the process even ignores SIGTERM.

    The same steps work normally with free inotify instances, or with OPENCODE_DISABLE_FILEWATCHER=1 while the instances are used up. That matches your result. It still happens on the latest dev (5a47777) run from source.

    The server seems to get stuck inside the native @parcel/watcher subscribe call. If that call never returns, the timeout around it can't fire. The script below reproduces the hang. Until there's a fix, OPENCODE_DISABLE_FILEWATCHER=1 is the workaround.

    Related reports, most relevant first: #53291, #37111 and #16610 describe the same deadlock in v1. #48366 and #32930 were earlier fix attempts that were closed without being merged.

    Reproduction script (repro.sh)
    #!/usr/bin/env bash
    # Reproduce: opencode serve hangs when no inotify instance is available (Linux, needs root/writable sysctl).
    # Usage: OPENCODE="opencode" bash repro.sh   (or OPENCODE="bun ... packages/cli/src/index.ts")
    set -u
    OPENCODE=${OPENCODE:-opencode}
    PORT=${PORT:-$((20000 + RANDOM % 20000))}
    export OPENCODE_PASSWORD=repro
    AUTH="-u opencode:repro"
    PROJ=$(mktemp -d /tmp/opencode/proj.XXXX)
    cd "$PROJ" && git init -q && echo hi > a.txt && git add . && git -c user.email=a@b -c user.name=a commit -qm init
    ORIG=$(cat /proc/sys/fs/inotify/max_user_instances)
    # Exhaust the per-uid inotify instance limit (like a shared-uid container would)
    EXHAUST=${EXHAUST:-1}
    [ "$EXHAUST" = 1 ] && echo 8 > /proc/sys/fs/inotify/max_user_instances
    [ "$EXHAUST" = 1 ] || echo 0 > /tmp/opencode/hog.ready
    [ "$EXHAUST" = 1 ] && python3 - >/dev/null 2>&1 <<'PY' &
    import ctypes, time, os
    libc = ctypes.CDLL(None, use_errno=True)
    fds=[]
    while True:
        fd = libc.inotify_init1(0)
        if fd < 0: break
        fds.append(fd)
    open('/tmp/opencode/hog.pid','w').write(str(os.getpid()))
    open('/tmp/opencode/hog.ready','w').write(str(len(fds)))
    time.sleep(600)
    PY
    HOG=$!
    while [ ! -f /tmp/opencode/hog.ready ]; do sleep 0.2; done; echo "hog holds $(cat /tmp/opencode/hog.ready) inotify instances"
    $OPENCODE serve --log-level debug --port $PORT --hostname 127.0.0.1 > /tmp/opencode/serve.log 2>&1 &
    SRV=$!
    for i in $(seq 1 15); do
      code=$(curl $AUTH -s -m 8 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$PORT/api/info" -H "x-opencode-directory: $PROJ")
      echo "t=$i /api/info -> $code"
      [ "$code" = "200" ] && break; sleep 1
    done
    # touch an instance-scoped endpoint for the project directory so watchers subscribe
    for i in 1 2 3; do
      for ep in session model agent config; do
        code=$(curl $AUTH -s -m 8 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$PORT/api/$ep" -H "x-opencode-directory: $PROJ")
        echo "/api/$ep (dir=$PROJ) -> $code"
      done
      code=$(curl $AUTH -s -m 8 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$PORT/api/info")
      echo "/api/info -> $code"
      sleep 2
    done
    SPID=$(pgrep -P $SRV | head -1); SPID=${SPID:-$SRV}
    for p in $SRV $(pgrep -P $SRV); do echo "pid $p wchan=$(cat /proc/$p/wchan) state=$(awk '{print $3}' /proc/$p/stat)"; done
    kill -9 $SRV $HOG $(cat /tmp/opencode/hog.pid 2>/dev/null) 2>/dev/null; rm -f /tmp/opencode/hog.pid; rm -f /tmp/opencode/hog.ready
    echo $ORIG > /proc/sys/fs/inotify/max_user_instances
    tail -5 /tmp/opencode/serve.log

    Run it: bash repro.sh

  2. r266-tech commented on Nov 8, 2026

    @r266-tech
    Contributor

    A dependency bump alone does not resolve this startup failure: I reproduced a synchronous subscribe() hang with both the pinned @parcel/watcher@2.5.1 and latest 2.6.0, using their unmodified Linux arm64 glibc prebuilds with Bun 1.4.4.

    Native dependency Normal control Descriptor exhaustion
    2.5.1 create event received; unsubscribe completed subscribe() never returned
    2.6.0 create event received; unsubscribe completed subscribe() never returned

    The isolated child process had RLIMIT_NOFILE=96. I filled its descriptors, then freed two for the backend's startup pipe. In each failing case, neither the statement after subscribe() nor a 500 ms JS timer ran before the parent killed the child at 5 seconds. No system sysctl was changed. This tests failed native startup, not the reported Kubernetes/overlayfs environment.

    The 2.6.0 backend still waits for mStartedSignal, while initialization can throw before signaling it. This is tracked by parcel-bundler/watcher#159; the proposed startup-error fix parcel-bundler/watcher#246 is now closed, unmerged. The existing disable-watcher workaround avoids that boundary; increasing a JS timeout cannot unblock it.

    AI-assisted investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions