Repository navigation
Description
opencode serve --service event loop deadlock on shared-uid k8s container (overlayfs) — native file watcher subscribe blocks the main thread; only OPENCODE_DISABLE_FILEWATCHER=1 recovers
Description
opencode serve --service (v2) intermittently becomes permanently unresponsive: the process stays alive and keeps LISTENing on its port, but never answers HTTP again. Live diagnosis of the hung process:
- Main thread parked in an infinite futex wait:
ps -o wchan=futex_wait_queue_me,%CPU = 0.0 curl -m8 http://127.0.0.1:<port>/api/info→ TCP connects but read times out (no response)- No inotify fds held by the process at the time of the hang (
/proc/<pid>/fdinfo) - The v2 watchdog (
opencode-serve-watchdog-v2) health-checks every 10s, getsRead timed out, restarts the service 3 times, then gives up:Auto-restart suspended: service failed repeatedly after 3 restarts— so from the user's perspective the service "cannot start".
Root-cause isolation (A/B/A control on the affected machine, opencode v2.2.5):
| state | wchan |
/api/info |
watchdog |
|---|---|---|---|
OPENCODE_DISABLE_FILEWATCHER=1 set |
ep_poll |
200 | healthy |
| variable removed, service restarted | immediately futex_wait_queue_me |
timeout | Read timed out → restart → suspended loop |
| variable restored, service restarted | ep_poll |
200 | healthy |
So the serve-side file watcher being enabled is both necessary and sufficient for the hang — exactly the pattern reported for v1 in #37111, and matching the inotify-exhaustion root-cause domain of #16610.
Environment specifics that matter: this runs in a shared-uid (uid 1000) Kubernetes dev container with overlayfs, where (per #16610 comments) inotify_add_watch() can fail with ENOSPC even far below max_user_watches, and where the history shows EMFILE: too many open files, watch '<path>' failures. The service hangs shortly after the watcher starts subscribing.
Why the existing guard does not help (serve side)
The serve-side watcher is packages/core/src/filesystem/watcher.ts, which subscribes through the native @parcel/watcher binding (ParcelWatcher, linux backend = inotify):
packages/server/src/routes.ts:131-132—Watcher.node.replace(Watcher.configured({ enabled: options.fs?.filewatcher }))+FileWatch.node.replace(...)packages/core/src/filesystem/watcher.ts:82-88—layer({ enabled: false })returnssubscribe: () => Effect.succeed(Stream.empty)(fully disables inotify); the nativesubscribeDirectory(...)path (:246-282) is used otherwisepackages/cli/src/server-process.ts:139— the switch the server reads:OPENCODE_FILEWATCHER_DISABLE ?? OPENCODE_DISABLE_FILEWATCHER
The native subscribe() performs its backend setup synchronously on the calling (main JS) thread before a promise materializes; when that setup wedges (inotify exhaustion / overlayfs ENOSPC / shared-uid instance-pool pressure), the Effect.timeout(SUBSCRIBE_TIMEOUT_MS) never gets a chance to fire — the event loop is already gone (same mechanism as #37111). PR #51536 (fix(tui): degrade gracefully when fs.watch fails) added a safeWatch() degrade path for the TUI (node:fs watch), but the serve side has no equivalent failure tolerance — a wedged native subscribe kills the whole server instead of degrading to "no watcher for that directory".
Plugins
None required — reproduce with --pure/no plugins; happens with the default install (metric/behavior plugins not involved).
OpenCode version
2.2.5 (also reproducible conceptually on 2.2.6+; upstream has no fix in packages/core/src/filesystem/watcher.ts as of the v2.2.5 line — the only later commits are #60955 "stop logging errors when optional watched folders are missing" and #59081 "watch files open in the side panel").
Steps to reproduce
Environment: Linux dev container in a shared-uid k8s pod (uid 1000) on overlayfs; host kernel 3.10 (container userspace Ubuntu 20.04). Low inotify headroom and/or a history of EMFILE: too many open files, watch ... in ~/.local/share/opencode/log/opencode.log.
- Start the background service:
opencode serve --service(oropencode service start). - Leave the file watcher enabled (no
OPENCODE_DISABLE_FILEWATCHER). - Wait — the serve process transitions to
wchan=futex_wait_queue_me, HTTP stops responding (curl to/api/infotimes out), watchdog logsHealth check error: ... Read timed out (read timeout=10.0)and eventuallyAuto-restart suspended. - Set
export OPENCODE_DISABLE_FILEWATCHER=1, restart the service → healthy (this is the confirmed workaround; A/B/A verified).
Screenshot
No response
Operating System
Linux (k8s container, shared uid 1000, overlayfs; container userland Ubuntu 20.04.3, host kernel 3.10)
Terminal
code-server web terminal (background service; not TUI — TUI works fine, see PR #51536 scope)
Activity
opencodejr commentedon Nov 8, 2026
Thanks for the detailed report and the A/B/A check. I can reproduce this.
To stand in for a shared-uid container with no inotify instances left, I set
fs.inotify.max_user_instances=8on Linux and had a helper process use up all 8 instances. Then I startedopencode serve(2.2.5) and sent requests for a git project:/api/infoand/api/sessionreturn 200./api/modelrequest never answers, and from then on every request,/api/infoincluded, times out.The same steps work normally with free inotify instances, or with
OPENCODE_DISABLE_FILEWATCHER=1while the instances are used up. That matches your result. It still happens on the latestdev(5a47777) run from source.The server seems to get stuck inside the native
@parcel/watchersubscribe call. If that call never returns, the timeout around it can't fire. The script below reproduces the hang. Until there's a fix,OPENCODE_DISABLE_FILEWATCHER=1is the workaround.Related reports, most relevant first: #53291, #37111 and #16610 describe the same deadlock in v1. #48366 and #32930 were earlier fix attempts that were closed without being merged.
Reproduction script (repro.sh)
Run it:
bash repro.shr266-tech commentedon Nov 8, 2026
A dependency bump alone does not resolve this startup failure: I reproduced a synchronous
subscribe()hang with both the pinned@parcel/watcher@2.5.1and latest2.6.0, using their unmodified Linux arm64 glibc prebuilds with Bun 1.4.4.subscribe()never returnedsubscribe()never returnedThe isolated child process had
RLIMIT_NOFILE=96. I filled its descriptors, then freed two for the backend's startup pipe. In each failing case, neither the statement aftersubscribe()nor a 500 ms JS timer ran before the parent killed the child at 5 seconds. No system sysctl was changed. This tests failed native startup, not the reported Kubernetes/overlayfs environment.The 2.6.0 backend still waits for
mStartedSignal, while initialization can throw before signaling it. This is tracked by parcel-bundler/watcher#159; the proposed startup-error fix parcel-bundler/watcher#246 is now closed, unmerged. The existing disable-watcher workaround avoids that boundary; increasing a JS timeout cannot unblock it.AI-assisted investigation.