fix(council): Perplexity research outage — port collision, invisible failures, keeper that never survived

[intellegix]

Second machine-wide Perplexity research outage in 24h (2026-08-08). Filed evidence:
council-automation/PERPLEXITY-SESSION-FAILURE-EVIDENCE-2026-08-08.md.

Was yesterday's fix a regression?

No. PR #71's identity gate worked exactly as designed — it correctly refused the
foreign Chrome on 9222 seconds before each failure. Three separate defects were found.

1. The identity gate was applied at one call site, not its twin

_start_via_cdp refused the relay's Chrome and fell back to a local launch. That
fallback runs _ensure_fresh_session("local-launch"), which still probed 9222 with a
bare urlopen and concluded "the keeper is alive" — the same reachability-is-not-identity
mistake, un-fixed. It then fired a keeper refresh that could never land, waited 120s, and
submitted on expired cookies.

Compounding it, PerplexitySessionKeeper had been Disabled since 2026-07-30, so
Start-ScheduledTask was a silent no-op and that 120s could only ever time out.

Fixes: both call sites share _keeper_cdp_alive(); _keeper_task_state() checks the task
before waiting on it; the keeper moves to a dedicated CDP port 9223 so it can never
contend with /takeover on 9222 again (the collision port-registry flagged as "latent,
still unresolved"); session_keeper.py refuses to adopt a foreign Chrome on its own port.

2. Failures were invisible — arguably the bigger bug

The queue logged started and never error, so errors_today read 0 through a total
outage. acquire_slot's error path cannot run when the process is killed outright, and
that is the normal end for a research run: the MCP layer calls execFileAsync with a 540s
timeout, and Node's timeout kill on Windows is TerminateProcess — no signal, no
unwinding. Queue-based monitoring was structurally blind to the dominant failure mode,
which is why both outages were found by a session tripping over them.

gc_dead_tickets() is the one place another process can observe the death, so it now logs
a dropped event there (once, by the winner of the unlink race). errors_today counts it,
the snapshot carries last_failure, queue_monitor alerts on it, and the MCP layer
returns RUNNER_KILLED instead of an opaque "Task failed".

3. The keeper never survived its launcher — found while verifying

DETACHED_PROCESS detaches the console but does not leave the parent's job object, and
every realistic launcher supplies one. Chrome died ~60s after each keeper invocation,
cleanly enough that it removed its own DevToolsActivePort file. Every run then found no
CDP and fell back to the local launch — i.e. this is why traffic reached the broken path
at all. Fixed with CREATE_BREAKAWAY_FROM_JOB (with fallback).

Two follow-ons: session_keeper.cdp now records Chrome's pid, not the one-shot keeper
python's; and the identity gates are reordered so the weakest signal (a recorded pid)
cannot veto positive proof of ownership. Previously that veto rejected every healthy
keeper. The foreign-Chrome protection is unchanged and still tested.

Correction (commit 4). I first claimed the breakaway flag made Chrome survive past
t+120s. That measurement came from a launch that had not picked up the new code, so it
does not support the claim — corrected rather than left standing. Measured reality:
[keeper 07:27:14] Job breakaway not permitted ([WinError 5] Access is denied).
Breakaway needs the containing job to grant JOB_OBJECT_LIMIT_BREAKAWAY_OK and Task
Scheduler's does not — the same constraint already proved with IsProcessInJob for the
Santee app. Under the task, Chrome lives on the order of minutes and the task repetition
re-establishes it; from an interactive shell the flag works and Chrome persists, so the
code stays. The proper fix (make the long-lived process the task's own process) needs
a new scheduled task and schtasks /Create is denied without elevation here — left as a
documented open item. This changes nothing about the outage fix or the monitoring fix:
nothing depends on the keeper being continuously up, and the verified 32.9s query ran on
the local-launch path, which now has a correct freshness guard and a working refresher.

Also: keeper-timeout-stale-proceed no longer proceeds

Hard-expired cookies raise SessionStaleError before submit (SESSION_STALE to the
caller) rather than burning a 5–8 minute slot on a doomed run. Cookies that are merely
expiring soon still proceed — that is what the old always-proceed rationale was about.

Verified by doing, not asserting

• Real research_query succeeded in 32.9s (was: two identical ~9-minute deaths).
• Keeper on 9223 and /takeover relay on 9222 coexisting; neither displaced.
• Keeper Chrome alive past t+120s with its launcher long exited (before: gone by t+60s).
• Forced a failure: os._exit inside a live queue slot — same semantics as the
  TerminateProcess Node issues. Produced enqueued/started/dropped, errors_today
  0 → 1, last_failure populated, and the monitor daemon logged ALERT [critical].
  That exact sequence was completely silent before this PR.
• 95 tests pass (17 new in tests/test_pplx_failure_visibility.py; 10 of the first 11
  fail against pre-fix code).

Ops changes already applied to the live machine

PerplexitySessionKeeper and PerplexityQueueMonitor re-enabled and running; keeper
rebound to 9223. ~/.claude/council-automation and mcp-servers/browser-bridge/server.js
updated in place (the MCP server executes them directly, so the fix had to be live to be
verifiable). server.js changes take effect for newly started sessions.

Note: the first commit snapshots two improvements that were live in the working tree but
unmerged, so the fixes apply on top of what actually runs.

🤖 Generated with Claude Code

https://claude.ai/code/session_013jYxW6be3KvPW4Sm8zdW5q

---

Ready for review (undrafted 2026-08-09) — and the case for merging got concrete today

Undrafted after verifying: 95 tests pass, branch clean and fully pushed, and every fix has been
running in production since 2026-08-08 07:26.

Today's incident is the argument. At 10:23:31 a repo-wide git restore under ~/.claude
reverted every modified-vs-HEAD tracked file — which silently un-did four of the files in this PR.
council_browser.py:67 imports start_lock_heartbeat from a submission_lock.py that the reverted
copy does not define, so council_browser became un-importable and research was down ~50 minutes
while errors_today stayed 0 and last_failure stayed null (the break was upstream of the
queue, so the counters read affirmatively green). Restored by hand at 11:20 from this branch.

Two things that matters for:

1. These fixes currently exist only in a live working tree plus this unmerged branch. A working
   tree is exactly what got reverted. Merging converts "restored by hand" into "cannot silently
   regress."
2. The near-miss was worse than the outage. Had the import not broken, research would have come
   back up without the keeper-port fix, the identity-gate twin fix and the fail-fast — i.e. the
   2026-08-08 machine-wide outage would have been possible again, with every metric green. The crash
   was the lucky part.

Also validated in production since opening: the dropped event fired on its first real failure
at 14:13 (a runner process death that left no other trace), moving errors_today 0 → 1 and
surfacing within one monitoring cycle. Before this PR that run would have been invisible.

One item from the original description is now closed by evidence rather than assertion: the
cross-process cookie-guard hypothesis I flagged as unproven has been resolved against itself —
run() admits at most one run machine-wide, making the mechanism structurally impossible, and the
refined variant was exercised 22 times in one day with zero corrupted extractions. No code change
followed, which is the point of having owed a reproduction.