Skip to content

Runaway FileWatch host-event loop freezes TUI and grows debug log to 13 GB #4612

Closed
Closed
Runaway FileWatch host-event loop freezes TUI and grows debug log to 13 GB#4612

Description

@tdihp
tdihp
opened on Aug 26, 2026

Describe the bug

A long-running/resumed Copilot CLI session can enter a tight loop that emits:

[DEBUG] [rust:copilot_runtime::protocol::jsonrpc::engine] No connection accepted a host event {"kind":"FileWatch"}

Once the loop becomes continuous, the terminal UI stops responding while the
process consumes about two CPU cores and writes hundreds of KB/s to its debug
log.

In the affected process:

  • The final 10,000 log records were all the identical discarded FileWatch
    event.
  • Current CPU measured over five seconds was approximately 200%.
  • RSS was approximately 1.1 GB.
  • The process log had reached 13 GB and was growing at approximately 460 KB/s.
  • The process remained alive; this was not an OS OOM kill or terminal flow
    control pause.
  • The host had ample free memory, disk space, and inodes.
  • USE_TGREP=false was already effective. Startup telemetry contained
    disabled_reason: "use_tgrep_false", and no tgrep child was running.

This appears related to IDE integration, but I cannot yet prove that opening
VS Code is the sole trigger. The affected session logged IDE lock-file watcher
startup and was running in a workspace open in VS Code. Disabling IDE
auto-connect prevented IDE discovery and the continuing event storm in a
controlled fresh-session test.

This differs from #3701: there was no repeated MCP server spawning or leaked
MCP process tree. The runaway activity was inside the Copilot process's
FileWatch/JSON-RPC event path.

Affected version

GitHub Copilot CLI 1.0.81-9

The no-IDE control test was also run after auto-update installed 1.0.81-11.

Steps to reproduce the behavior

The exact transition into the continuous loop is not yet deterministic, but
this is the observed setup:

  1. Open a large multi-repository workspace in VS Code.
  2. Start Copilot CLI from that workspace root with debug logging enabled.
  3. Resume and continue using a long-lived session.
  4. Leave IDE auto-connect enabled (the default).
  5. The session eventually becomes unresponsive while its log continuously
    prints the discarded FileWatch event shown above.

Useful checks while the failure is active:

ps -p <pid> -o pid,pcpu,pmem,rss,nlwp,stat,wchan,cmd
tail -n 10000 ~/.copilot/logs/process-*.log |
  sed -E 's/^[0-9TZ:.-]+ \[[A-Z]+\] \[[^]]+\] //' |
  sort | uniq -c | sort -nr | head

The affected process showed:

10000 No connection accepted a host event {"kind":"FileWatch"}

Controlled comparison

I configured:

{
  "ide": {
    "autoConnect": false
  }
}

Then started a fresh CLI session from the same workspace while VS Code remained
open. In that session:

  • There were no IDE lock watcher, IDE discovery, or ide_connected records.
  • The process had no inotify watch on ~/.copilot/ide.
  • Only five FileWatch records appeared during startup.
  • Log growth then stopped, current CPU dropped to approximately 1%, and the TUI
    remained responsive.

For comparison, using the invalid dotted JSON key
"ide.autoConnect": false produced a settings warning, started the IDE lock
watcher, detected VS Code, and connected to the IDE. The nested object shown
above is required.

Expected behavior

  • FileWatch events should be consumed, coalesced, debounced, or dropped without
    entering an unbounded CPU/logging loop.
  • A disconnected/uninterested host-event consumer should not cause every
    filesystem event to be logged indefinitely.
  • IDE lock-file activity should not starve terminal input or rendering.
  • Debug logging should be rate-limited for repeated identical events.

Additional context

  • OS: WSL2 Linux 6.6.87.2-microsoft-standard-WSL2
  • Architecture: ARM64
  • Terminal: VS Code integrated terminal
  • Workspace root contained multiple repositories.
  • Session indexing reported SESSION_INDEXING=true.
  • tgrep was disabled with USE_TGREP=false.
  • The affected process had two inotify descriptors resolving to the resumed
    session's own ~/.copilot/session-state/<session>/ and research/
    directories. The first FileWatch records also appeared before the log line
    that started the IDE lock-file watcher, so IDE auto-connect may be an
    amplifier or trigger rather than the underlying event source.
  • Another session from the same environment previously accumulated a 2 GB log
    and approximately 4 GB RSS with large bursts of the same FileWatch message,
    but later became idle instead of remaining in the tight loop.

Workaround:

{
  "ide": {
    "autoConnect": false
  }
}

Restart Copilot CLI after applying the setting. Manual /ide connection
remains available.

Activity

  1. github-actions
    added
    triage
    on Aug 26, 2026

  2. tdihp commented on Aug 26, 2026

    @tdihp
    tdihp
    on Aug 26, 2026
    Author

    New evidence materially narrows this down: IDE auto-connect is not required.

    After terminating the first runaway process, I resumed the same local session.
    The replacement process immediately reproduced the failure even though its
    telemetry consistently reports:

    "config_ide_autoConnect": "false"

    There are no Starting IDE lock file watcher, Connected to IDE, or
    ide_connected records in the replacement process log. Its only inotify
    descriptors resolve to the resumed session's own:

    ~/.copilot/session-state/<session>/
~/.copilot/session-state/<session>/research/

    Observed timeline:

    05:50:04.158 Session indexing debug: SESSION_INDEXING=true
05:50:06.344 first discarded FileWatch event

    The storm then continued from the first minute onward:

    05:51  116,443 events/min
05:52  125,341 events/min
05:53  106,727 events/min
...
06:15  136,025 events/min

    Current replacement-process measurements:

    CPU:             ~239%
log growth:      ~339 KB/s
RSS:             ~1.15 GB
dominant tail:   9,691 / 10,000 records are discarded FileWatch events

    The watched session directory was not continuously changing during a
    three-second sample. Its persistent footprint is also modest (105 files,
    ~56 MB), while another non-spinning session has 3,080 files and ~1.97 GB of
    state. This does not look like a simple threshold on session file count.

    A brand-new session with ide.autoConnect=false remains stable. Resuming this
    specific existing session reproduces the storm immediately. The likely trigger
    is therefore persisted/resumed session watcher or event state, rather than VS
    Code lock-file activity itself. The IDE-disabled workaround prevents one input
    path but does not recover an already affected session; avoiding resume and
    starting a new session is currently the reliable workaround.

  3. tdihp commented on Aug 26, 2026

    @tdihp
    tdihp
    on Aug 26, 2026
    Author

    Further tracing of the shipped app.js identifies the watcher ownership and
    explains why only two watches can still exhaust the process.

    The interactive TUI unconditionally mounts two React hooks for local sessions:

    1. A plan-status hook watches dirname(plan.md), which is the session root.
    2. A research-status hook watches <session>/research. If that directory does
      not exist, it temporarily watches the session root until research/
      appears.

    These are exactly the two inotify descriptors observed in the runaway process:

    ~/.copilot/session-state/<session>/
~/.copilot/session-state/<session>/research/

    They are non-recursive directory watches. The file count and number of watches
    are therefore not the source of exhaustion.

    The relevant bundled-code path is:

    plan/research React hook
  -> wz(path, { persistent: false }, callback)
  -> supportWatchPathWeak(path)
  -> async `await handle.next()` pump
  -> native Rust FileWatch host-event dispatch

    Watching the directory is reasonable because plan.md, research reports, or
    the research/ directory may be created or atomically replaced. The
    JavaScript callbacks filter for plan.md or research .md files. However,
    that filtering occurs only after the native watcher/event layer processes the
    raw directory event.

    The runaway process had only these two watches, but generated approximately
    2,000-3,000 rejected events per second:

    No connection accepted a host event {"kind":"FileWatch"}

    Each rejection is dispatched and debug-logged individually. There is no
    visible coalescing, rate limit, backoff, or automatic shutdown of a watcher
    whose events have no consumer. That is sufficient to consume multiple CPU
    cores and grow the log by hundreds of KB/s despite having only two inotify
    watches.

    Subagents appear to be an amplifier/trigger rather than a continuing source.
    They create bursts of task transitions, session-state writes, rewind
    snapshots, research output, and native session/host-channel lifecycle churn.
    The evidence is consistent with a teardown/restore race leaving a weak watcher
    producer detached from its consuming connection. Once the session reaches that
    state, resuming it recreates the loop immediately and it continues even while
    the watched directories are not changing.

    In short, this is not inotify watch exhaustion:

    two normal non-recursive watches
  -> watcher consumer becomes detached
  -> unbounded rejected FileWatch dispatch/log loop
  -> CPU and log exhaustion

    There is currently no documented setting to disable these plan/research
    watchers. ide.autoConnect=false disables only the separate IDE lock-file
    watcher.

  4. tdihp commented on Aug 26, 2026

    @tdihp
    tdihp
    on Aug 26, 2026
    Author

    Correction after further investigation: the continuous FileWatch process
    initially examined was not the exact terminal instance reported as frozen.

    The correct process also exhibited a substantial rejected-FileWatch event
    burst. However, its event ordering indicates that this burst may be a symptom
    or amplifier after the TUI event consumer disconnects, rather than the initial
    cause of the freeze:

    subagent starts
  -> TUI/session-host event consumer stops receiving timeline events
  -> runtime and subagents continue independently
  -> FileWatch events have no consumer and are repeatedly rejected
  -> CPU, logging, and memory pressure increase
  -> FileWatch activity may subside while the TUI remains frozen

    This resembles #4533, where runtime/subagent activity continues after the
    terminal stops updating. Concurrent ownership of a resumed session was also
    present and may be relevant, but that relationship is not yet established.

    Please treat the FileWatch storm as a confirmed resource-amplification problem,
    while causality for the initial TUI detachment remains under investigation.

  5. MilkyWay008 commented on Aug 27, 2026

    @MilkyWay008
    MilkyWay008
    on Aug 27, 2026

    That FileWatch host-event flood at 2-3k rejections/sec is the TUI React watcher hooks firing into a detached jsonrpc engine, and the inotify watches on the two session-state dirs are just the trigger surface, so disabling auto-connect wouldn't stop it. As a stopgap, avoid resuming sessions that already looped, and truncate the debug log before relaunching so the 13GB doesn't come back instantly. There's a related report about the same class of watcher loop, worth cross-linking in case maintainers can pool the traces. WSL2 is an unusual enough combo that the reproduce details you posted will carry the issue. oh also, btw, I built a portable Hermes OTG and I run it on USB as my rescue agent; super handy doing quick diagnose & fix, check it, imo it'll help you. https://github.com/MilkyWay008/Hermes-OTG

  6. peacenix commented on Aug 27, 2026

    @peacenix
    peacenix
    on Aug 27, 2026

    The discriminator is plan.md in the session root, combined with a regression after 1.0.81-1.

    Same session, same machine, plan.md present throughout:

    build run rejected FileWatch events
    1.0.81-1 Aug 18-25, 7 days 0
    1.0.81-9 Aug 26 758,629 / 4,920,665
    1.0.81-14 Aug 27 6,018,006
    1.0.81 Aug 27 16,634,832

    Two other local sessions with no plan.md -- one with a larger 92 MB events.jsonl and 96 subagent starts -- stay flat on -9 and on GA. Both conditions are needed: the file, and a build after -1.

    This fits your hook analysis. The plan-status hook watches dirname(plan.md), which is the session root, and events.jsonl is appended there on every event. First agent activity starts the loop and it then self-sustains -- one process held ~2,350 rejections/sec for 18 h overnight at a constant rate with the session idle. Your three-second sample was right: once started it does not need ongoing directory changes.

    Moving plan.md out of the session root fixed it in place, no restart: 277% -> flat at idle. An affected session does not have to be abandoned.

  7. cwedgwood commented on Aug 27, 2026

    @cwedgwood
    cwedgwood
    on Aug 27, 2026

    Confirming this can be fatal, not only a hang.

    Same conditions (GA 1.0.81, plan.md in the session root). The process aborted roughly ten minutes after start:

    node::OOMErrorHandler
v8::internal::V8::FatalProcessOutOfMemory
v8::internal::Heap::CollectGarbage
v8::internal::MinorGCJob::Task::RunInternal

    V8 reached its own heap limit. This was not an OS OOM kill — the cgroup had headroom and recorded no kill.

  8. cwedgwood commented on Aug 28, 2026

    @cwedgwood
    cwedgwood
    on Aug 28, 2026

    Still present in 1.0.82-0.

    Same conditions (plan.md in the session root). Rejected FileWatch events, measured over a short session:

    build rate
    1.0.81 ~530/s
    1.0.82-0 ~540/s

    Unchanged.

  9. tdihp commented on Aug 31, 2026

    @tdihp
    tdihp
    on Aug 31, 2026
    Author

    changing log to default doesn't seem to help. This pretty much renders plan mode unusable :(.

  10. spoojaryFungible commented on Sep 8, 2026

    @spoojaryFungible
    spoojaryFungible
    on Sep 8, 2026

    Reproduced on 1.0.84-0 (reporter saw 1.0.81-9/-11), with data that I think narrows this down.

    No IDE was connected

    The report suspects IDE integration. In my case the storm ran with no IDE attached at all:

    • ~/.copilot/ide is empty (no lock files)
    • no vscode-server / code-server process on the host
    • the CLI runs in plain tmux over SSH, never launched from VS Code

    So an open IDE is not required to trigger it. The ~/.copilot/ide watch is registered regardless, and "no connection accepted" is simply the steady state when nothing is attached — the events are generated and logged even though there is provably no consumer.

    The three inotify watches

    Resolved from /proc/<pid>/fdinfo (mask 0xfee = IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_OPEN|IN_MOVED_*|IN_CREATE|IN_DELETE|...):

    inode path
    1c61d7 ~/.copilot/session-state/<session-id>/
    1c61dc ~/.copilot/session-state/<session-id>/research
    20dee2 ~/.copilot/ide

    The first one contains events.jsonl, session.db and session.db-wal — files the session itself writes on every event. The process is watching a directory it continuously writes to, so it generates its own event stream.

    Measured amplification

    Over a 20 s window with the session idle:

    events.jsonl:  +14,746 bytes  (6 events: turn_start/user.message/tool.execution_*/turn_end)
debug log:     +3,575 KB      (~26,000 identical FileWatch lines)

    ≈ 4,300 discarded log lines per session event, ≈ 248x byte amplification. Sustained 15 GB/day while idle; measured 700 KB/s (~60 GB/day) shortly after a truncate, vs the ~460 KB/s in the report.

    It is per-session, not per-config

    Same host, 8 concurrent long-lived sessions, all with the identical 3-watch layout above:

    session age FileWatch lines log size
    3.7 days (affected) 1,601,347 47 GB + 36 GB
    14 days 3,020 178 MB
    13 days 4,380 113 MB
    11 days 2,596 48 MB
    6 days 9 43 MB

    A 14-day session emitted 3,020 of these; the affected 3.7-day session emitted 1.6 million. Identical watch configuration, so the watch setup alone is not the trigger — something puts one session into the tight loop and it never leaves.

    Additional failure mode: it fills the disk

    The report notes 13 GB. Here two logs in the same process tree reached 47 GB and 36 GB, filling a 247 GB root filesystem to 0 bytes free. That surfaced as:

    Failed to persist session events: GenericFailure, Failed to append to JSONL file
  .../events.jsonl: I/O error: No space left on device (os error 28)
Execution failed: Request session.send failed ...

    so the session could no longer persist events, and events.jsonl was left with a partially written trailing record. Worth noting the log is written at DEBUG with no rotation or rate limiting, so a single looping session can take down unrelated work on the same machine.

    Note for anyone hitting this

    rm does not help while the session is alive — the file is held open, so the blocks are not released until the process exits. Truncate instead:

    : > ~/.copilot/logs/process-<ts>-<pid>.log

    Happy to pull anything else from the affected process while it is still running.

  11. patniko commented on Oct 26, 2026

    @patniko
    patniko
    on Oct 26, 2026
    Collaborator

    Thanks for the very thorough investigation here, including the plan.md and session-directory analysis. Copilot CLI v1.0.85 changed the file watchers to ignore non-mutating (read/access) file-system events, so reading files in the session directory no longer feeds back into the watcher and triggers the endless FileWatch event stream, CPU spin and runaway debug log described here; this is the same fix that resolved #4807. The separate problem of the terminal UI no longer processing events while subagents keep running is tracked in #4533. If you still see the FileWatch storm on the latest version, please reopen with your version and a log excerpt.

    Triaged as part of a Copilot SDK backlog review, with help from GitHub Copilot.

  12. patniko
    closed this as completedon Oct 26, 2026
  13. github-actions
    removed
    triage
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions