Lifecycle event seam + bounded post-mortem surface for supervised work (companion to #1/#2)

[iceteaSA]

Contract proposal, two halves of one seam: durable session-lifecycle events + a bounded post-mortem surface. Companion to #1 (watchers) and #2 (schedules) — this is about subc as the observability substrate for supervised work.

What we run today, harness-side

Eleven passive observer plugins ride inside our harness process and re-derive session state from its SQLite DB on every tool call. The two relevant here:

• A runaway detector classifying live child sessions into: dead-dispatch (errored-only child), no-activity stall, truncation-wedge (finish=length with zero tool parts), and p90-relative runaway per agent class (tools/runaway-tap/index.ts:250-325 in our config repo).
• A post-mortem tool reconstructing a finished session from durable rows — terminal state, bounded recent errors, final text, cost — with the heuristic that a missing result envelope is normal completion and finish=length is not (tools/autopsy.ts:44-67).

Both work, but the architecture is wrong: they die with the harness process, poll instead of subscribing, and force every consumer (dashboard, notifier, score logger) to re-implement the same state derivation.

Offer

• The terminal-reason / liveness signal vocabulary above, field-tested for months (the truncation-wedge signature alone has caught a class of provider-clamp failures nothing else surfaced).
• A consumer-side reference implementation once a seam exists — we would port our detectors to it immediately and delete the polling versions.

Asks

1. Should subc core standardize a lifecycle event stream for supervised work (module/session started, heartbeat/progress, terminal with reason taxonomy), with policy (reminders, kills, scoring) left entirely to consumers?
2. Same question for a bounded inspect-style post-mortem op: terminal reason, bounded error tail, bounded final output. Should subc own that contract, or only expose raw state for a higher layer to shape?
3. If both are out of scope for core — is the blessed pattern a dedicated observer module, and if so does the module protocol already carry enough (event subscription across modules) to build it without core changes?

[iceteaSA]

Taking your three asks in order, with the caveat that the third one turns out to gate the first two.

Ask 1 — lifecycle event stream

The direction you propose is already the declared architecture, though the doc frames it as transport rather than as an event stream. docs/subc-core-architecture.md:307 specifies PUSH forwarding as a core service — "async frames (completions, status changes) delivered on the owning channel" — and the control protocol reserves a watch. op prefix (crates/subc-control/src/lib.rs) described in docs/subc-control-protocol.md:171 as "watch.* for subc-owned events".

So a subc-owned event vocabulary is reserved, and the policy-stays-with-consumers split you propose matches how the rest of the surface is drawn. Supervision already tracks most of the state such a stream would carry: ModuleState, restart counts, ExitReport{kind, code, signal}, and health transitions all exist inside crates/subc-core/src/supervise.rs. What's missing is emission, not observation.

Your terminal-reason taxonomy is the part subc doesn't have and can't derive. The daemon knows how a process ended — exit code, signal, crash vs clean. It has no notion of dead-dispatch, truncation-wedge, or p90-relative runaway, because those are properties of the work, not the process. That's a real boundary and I'd argue it's the right one: core emits observed process and route facts, consumers classify. Your truncation-wedge signature (finish=length with zero tool parts) is a good example of why — it's meaningful only to something that knows what a tool part is.

Ask 2 — bounded post-mortem surface

Partly built, and there's a live parallel worth knowing about. SupervisorEntry already carries last_exit_code and last_exit_signal, surviving respawn so a running module still reports what killed its previous incarnation. Issue #3 is currently working the same seam from the operator side, asking for restart counts and bounded stderr on the status row.

The stderr half is architecturally blocked in a way that bears directly on your ask. crates/subc-core/src/supervise.rs:2437 documents that child stdio is deliberately inherited — every supervised child shares the daemon's fd, which is what puts all module output in one log without a per-child reader task. The comment records the structural alternative (per-child pipe with a line-atomic writer) as consciously declined, with a measurement: 212 of 3000 lines came out spliced under an incremental emitter.

So "bounded error tail" is the expensive field in your post-mortem contract, and it's expensive for a reason that has nothing to do with the contract's shape. Worth designing around rather than assuming.

Ask 3 — and this is the one that gates the others

You ask whether a dedicated observer module could build this without core changes, given the module protocol. The blocker is the same one #1 and #2 hit, and I don't think any of the three can move until it's resolved.

PUSH as specified is live-connection semantics: frames delivered on the owning channel. Every one of these issues needs delivery to a consumer that isn't currently holding a route:

• Where should restart-surviving condition watchers live? (scheduler scope vs module home) #1 wants a watcher to notify a session that has since restarted.
• Time-triggered work (scheduled agent runs): does subc core want a schedule surface? (companion to #1) #2 wants a scheduler to wake a harness session at a scheduled time.
• Lifecycle event seam + bounded post-mortem surface for supervised work (companion to #1/#2) #4 wants an observer to receive lifecycle events it subscribed to in a previous process.

Same missing contract three times: can subc address a session that holds no current route? If yes, all three are variations on one delivery mechanism and the event stream you're asking for is mostly plumbing. If no, an observer module can't subscribe durably either — it would face the identical problem one layer out, and you'd have relocated the polling rather than removed it.

That also answers your "is the blessed pattern a dedicated observer module" as posed: the module protocol carries enough for a live observer today (route to a module, receive pushes on that route). It does not carry durable subscription, so an observer module would die with its connection the same way your in-process detectors die with the harness.

Suggestion

These four issues have been treated as four scope questions. Three of them are one design question wearing different clothes, and it's the one nobody has specified: durable, session-addressed delivery. I'd rather see that designed once — with lifecycle events, scheduled wake, and condition-watcher notification as its three consumers — than answered three times inconsistently.

Your offer of a consumer-side reference implementation is more valuable against that contract than against the event vocabulary, since the vocabulary is the easy half.

[iceteaSA]

Cross-reference, because these two threads can be answered independently and one of them can silently break the other.

#6 documents that error codes have no authenticated provenance — a module can emit a daemon-reserved code and subc relays it verbatim, so clients branching on codes to decide retry safety are doing so unsoundly. One candidate remediation is a daemon-only channel-0 push, whose whole value is that a module cannot forge it.

That property holds today, and I verified why: there is no module→client push relay at all. ModuleControlPush has a single variant (RouteStatus), handle_status_update consumes it into a cache and returns no frames, and unknown module push ops are ignored rather than forwarded. Channel-0 client-bound frames are daemon-generated.

It holds by absence, not by rule. Nothing forbids a relay — there simply isn't one.

This issue is asking for the feature that would add one. A lifecycle event stream carrying module-originated events to subscribed observers is, mechanically, a module→client push relay. Build it without care and any provenance guarantee resting on daemon-only pushes quietly evaporates.

Both are still buildable together. The constraint is narrow: an observer-bound event relay must not let a module choose the op identity or the origin of what lands on a client's channel 0 — relay module events under a distinct daemon-stamped op, don't forward module push frames through.

Flagging it here rather than only in #6 so the guard survives these being read in either order.

[ualtinok]

Answering all three, and the answer splits your question along a line the issue treats as one thing: "supervised work" is two different subjects with two different owners.

Module lifecycle is mine. The daemon spawns, supervises, health-probes and restarts modules, so module-scoped events — started, registered, health verdict, terminal with exit code and signal — are subc's to emit and it already holds all of them. Today they are readable only by polling ck module list / ck module status, which is the gap you are pointing at.

Session lifecycle is not mine, and this is settled rather than my preference: it belongs to prefrontal, the module that owns sessions. Everything in your ask that concerns a session — started, heartbeat/progress, terminal reason taxonomy, the post-mortem surface — lands there. Same ruling as #2: core stays a supervisor, modules own the work.

So, to your three asks directly:

1. A lifecycle event stream for supervised work — split. Module events: in scope for core, and worth doing for the reason you give (consumers should not each re-derive state, and observers should not die with a harness process). Session events: prefrontal.
2. A bounded post-mortem op — not core. Terminal reason, error tail and final output are properties of a run, and core does not know what a run is. It has no session model and I do not want to give it one; the daemon is deliberately state-free, and a post-mortem surface needs durable state.
3. The blessed pattern is a module, and the protocol already carries enough. Modules can hold a request open and stream StreamData frames to a subscriber — that is how event subscription works today, and a consumer subscribes rather than polls. An observer module can subscribe to whatever a producing module exposes without any core change. What core would add is the module-scoped half of (1), so an observer can see module terminal states it cannot otherwise reach.

Your terminal-reason vocabulary is the most valuable part of the offer and it should not land here. Field-tested taxonomy — dead-dispatch, no-activity stall, truncation-wedge, p90-relative runaway — is exactly the substance of the contract, and the truncation-wedge signature in particular (finish=length with zero tool parts) is the kind of signal that only shows up in months of real traffic. It belongs against prefrontal and the runner, where it will be implemented. I would rather it be discussed against the module that will carry it than accumulate here where the answer can only be "not this layer".

One thing worth stating plainly, since it bears on how much core should take: three of your five issues here (#3, #5, this one) are the same shape — the daemon knows the answer and its operator surface does not carry it. That is a pattern in my code rather than three papercuts, and the module-scoped half of (1) is the general fix rather than another field. It is the reason I am taking that half rather than deferring the whole issue.

[ualtinok]

Same correction as on #3 and #5: I answered the issue body without reading your comments first. Your analysis changes my answer in one place and sharpens it in another, and the gating question you raise is the right one.

Verified at source rather than taken:

• The watch.* prefix is genuinely reserved — crates/subc-control/src/lib.rs:28 (pub const WATCH: &str = "watch.") and docs/subc-control-protocol.md:171 naming it "for subc-owned events". You found a reservation in my own protocol that I had forgotten was there. That does make the module-scoped half of your ask declared architecture rather than a new proposal, and I was answering as though it were new.
• scheduler.* is reserved in the same list. That is now settled the other way and I have answered Time-triggered work (scheduled agent runs): does subc core want a schedule surface? (companion to #1) #2: scheduling will not be in core, it goes to prefrontal. The reservation predates the decision; I would rather retire the prefix than leave a reserved name implying a plan that no longer exists.

Your boundary is right and better stated than mine. Core emits observed process and route facts — ModuleState, restart counts, ExitReport{kind, code, signal}, health transitions, all of which supervision already tracks. Consumers classify. Your truncation-wedge example is exactly why: it is meaningful only to something that knows what a tool part is, and the daemon does not and should not. "What's missing is emission, not observation" is the accurate summary of my side.

On ask 3, which you say gates the other two — you are right that it is one contract, and I can answer it: today, no. The daemon cannot address a session that holds no current route. PUSH is live-connection semantics: frames go on the owning channel, and if there is no channel there is no addressee. Route death is expressed by GOODBYE precisely because the client is still connected to receive it.

So your three-way collapse is correct, and I would add that it explains why the answer to each issue felt different depending on which one you asked first.

Where that leaves each piece:

• Durable delivery to a session with no route is not something I want in core, and for the same reason as scheduling: it needs durable state — a queue, delivery tracking, a retention policy — and the daemon is deliberately state-free. Adding a store behind the supervisor puts all fifteen modules behind one migration.
• prefrontal already solves this problem for its own domain: it addresses sessions that are not currently connected, and delivers when they next are. That is the mechanism your Where should restart-surviving condition watchers live? (scheduler scope vs module home) #1, Time-triggered work (scheduled agent runs): does subc core want a schedule surface? (companion to #1) #2 and Lifecycle event seam + bounded post-mortem surface for supervised work (companion to #1/#2) #4 all need, and it exists.
• The module-scoped watch.* half remains mine and remains worth doing — live subscribers, no durability, no addressing of absent sessions. An observer module holding an open subscription is a live connection by definition. That covers "consumers should not each re-derive module state" without core growing a mailbox.

So the split is: core emits, live only; prefrontal owns durable delivery to sessions; consumers classify. An observer module that wants to survive its own restart subscribes on start and backfills from supervisor.list — which is exactly the shape you would build anyway, since a durable stream still needs a resume point.

On the stderr blocker you flag for ask 2: that decision is being reversed, see #7. Your read of supervise.rs is accurate — I wrote that comment, and it is correct about interleaving and silent about durability, which is the axis that matters for your post-mortem contract.

[iceteaSA]

Split accepted — core emits module facts live-only, prefrontal owns durable session delivery, consumers classify. And "what's missing is emission, not observation" is the right summary; I'll take the terminal-reason taxonomy to prefrontal where it'll be implemented rather than accumulating it here.

Two things before this closes.

The backfill story has a gap, and it's the one my post-mortem ask was actually about

You say an observer subscribes on start and backfills from supervisor.list, "which is exactly the shape you would build anyway, since a durable stream still needs a resume point". Agreed on the shape. But supervisor.list can't serve as that resume point for the case that matters.

SupervisorEntry carries: module_id, state, enabled, live, health, last_probe_ms, last_exit_code, last_exit_signal, and restart_count once the patch on #3 lands. That's a current-state snapshot with one slot for exit information.

So an observer that misses events while restarting can recover that a module exited and how it exited most recently — but not:

• How many terminal events it missed. restart_count tells you respawns happened; it doesn't let you reconstruct them, and it resets on a clean start.
• Any terminal state older than the latest. Two crashes with different causes while the observer was down leave one last_exit_code. The first is unrecoverable.
• When any of it happened. There's no timestamp on last_exit. last_probe_ms is when health was last collected, which is unrelated. So an observer can't order a recovered exit against events it did receive, or tell a crash thirty seconds ago from one three hours ago.

That last one is the sharpest, because it's the same defect your own comment on last_probe_ms warns about: "an old value and an absent one call for opposite readings, so do not render them alike." last_exit has no value to be old.

Not asking for a durable event store — that's the thing you're correctly refusing. But if supervisor.list is the designated resume point, it needs enough to be one: at minimum a timestamp on the last exit, so a backfilled terminal state can be ordered against the live stream. Without it an observer can't tell whether what it read is news.

The pattern you named is worth one more instance

You wrote that three of five issues are the same shape — the daemon knows the answer and the operator surface doesn't carry it — and that the module-scoped watch.* half is the general fix rather than another field.

The gap above is a fourth instance, and it's slightly different in a way that matters for that fix: here the daemon knows the answer at the moment it happens and then overwrites it. last_exit is a one-slot buffer. An event stream fixes this properly — the event carries the terminal state when it occurs, so nothing has to be reconstructed. Which is an argument that the watch.* half is load-bearing rather than convenient: without it, every consumer is reduced to sampling a surface that forgets.

Filed #6 and #7 since these comments, both from the same seam. #7 you've already accepted; #6 is a protocol question about error provenance that touches how much a consumer can trust any code it branches on. Neither needs anything here.