Skip to content

Beta stability slice: device-loss recovery, truthful lifecycle, encoder probe, output supervisor, take telemetry - #420

Merged
iamfatness merged 26 commits into
mainfrom
codex/beta-stability-slice
Sep 10, 2026
Merged

Beta stability slice: device-loss recovery, truthful lifecycle, encoder probe, output supervisor, take telemetry#420
iamfatness merged 26 commits into
mainfrom
codex/beta-stability-slice

Conversation

@iamfatness

@iamfatness iamfatness commented Sep 10, 2026

Copy link
Copy Markdown
Owner

The beta-stability half of #419, split out so it can merge on its own. #419 stays open as the architecture draft; nothing from its Wave 1/2 foundations is included here.

What's in it (26 commits, cherry-picked in order from #419)

Live-show fixes (2026-09-08/09)

  • Live routing transitions and stale frames
  • Each SRT ingest pipe gets a single owning reader thread
  • ISO video submitted on the video tick (arrival-driven), audio sample loss published
  • Orphaned egress encoders are killed with the core process (job object)

Beta slice (docs/production-realtime-completion-plan.md §7 on #419)

  • D3D device loss is recovered by generation with a bounded ladder, never permanent CPU fallback
  • Destination lifecycle is truthful: Stop no longer claims completion before finalize
  • Fault-injection seams for device loss, blocked present and blocked monitor render
  • Encoder capacity is probed per machine; software spill is loud; ISO admission refuses before the show
  • Output destination supervisor: health from accepted units, 5-10-20-40-60s restart ladder, loud give-up, NDI in-process risk published

Observability

  • GET /snapshot serves the core's own sessionState (49 nodes), redacted, with staleness
  • Worker evidence counters (realtimeEvidence), lifecycle contracts, and the Wave 0 qualification judge input
  • Rendered scene id attribution is honest (was stuck on the pre-take scene)
  • One structured record per Take, and per-source Zoom subscription churn
  • Fake engine delivers exactly one video stream per participant; drill names evidence instead of asserting a cause

Tiles

  • A settled wall is cut to, not rebuilt, on Program (partial: the full fix is the persistent-source redesign in codex/persistent-sources)

Verification

  • Native: 781/781 (MonitorRenderFaultInjection.* are real-GPU timing tests, opt-in)
  • Shell: Control 74/74, MediaCore 2158/2158, WinUI 1289/1289
  • Live: ran on the 2026-09-09 8-source show and this morning's 7-participant test meeting from the Establish production realtime architecture foundations #419 branch (Program 59.7 fps + 8 ISO, encoder probe measured 7 sessions on the 4090)

Split notes

  • One conflict resolution: the take-telemetry commit's ZoomEngineRuntime hunk was bundled with Establish production realtime architecture foundations #419's source-authority code; only the subscription-churn parts are kept here.
  • The evidence/contracts commit (85dd7d8) is included because the beta fixes depend on its worker counters and lifecycle enum. It wires no architecture classes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SAB8v62BEV8ihmaeGhhtY9

iamfatness and others added 26 commits September 10, 2026 08:14
@iamfatness
Fix live routing transitions and stale frames
d7d1ec4 
(cherry picked from commit 658a3d1)
@iamfatness
Document live show remediation plan
a773463 
(cherry picked from commit dbf6edc)
@iamfatness
Establish production evidence and identity contracts
4b5ed68 
(cherry picked from commit 85dd7d8)
@iamfatness @claude
Give each SRT ingest pipe a single owning reader thread
0b0227b 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 13545fb)
@iamfatness @claude
Submit ISO video on the video tick and publish audio sample loss
2da55a3 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 10a1a99)
@iamfatness @claude
Kill orphaned egress encoders with the core process
d3f7046 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 523a295)
@iamfatness @claude
Feed the qualification judge and make the drill name evidence
8109c8e 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit f06d6fb)
@iamfatness @claude
Keep node:test goldens and blocking timer test out of pool scheduling
6e73f45 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 2881bb7)
@iamfatness @claude
Recover from D3D device loss instead of degrading until restart
875f93c 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 4527e1f)
@iamfatness @claude
Give each fake participant exactly one video stream
2dc8c0f 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 8a53ac8)
@iamfatness @claude
Drive ISO video off frame arrival and separate startup shedding
25e03a7 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit aa17c10)
@iamfatness @claude
Stop claiming a destination completed before it did
8ec0452 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit c044320)
@iamfatness @claude
Add fault-injection seams for device loss, present and monitor render
bc600bd 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit e6ac97a)
@iamfatness @claude
Probe encoder capacity instead of assuming eight sessions
84dcc1c 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit a268fee)
@iamfatness @claude
Judge monitor-stall recovery against the fault, not an idle baseline
2b88a21 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit a54e7b3)
@iamfatness @claude
Retry a starved baseline before failing a monitor timing test
dbaa949 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 5f7b0e5)
@iamfatness @claude
Make real-GPU timing measurements opt-in, not part of the default suite
aeada32 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit b6b3a5c)
@iamfatness @claude
Cut to a settled Tiles wall instead of rebuilding it on program
7142480 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit a305391)
@iamfatness @claude
Write the morning handoff for the live-show findings
4dc38be 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 62e19da)
@iamfatness @claude
Serve the core's own snapshot for live observation
646929d 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit 0056b0f)
@iamfatness @claude
Note the meeting ended and record the snapshot endpoint
577b75b 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YASnsbX3oQRZRxUNroqjY5
(cherry picked from commit b437b61)
@iamfatness @claude
Make the rendered scene id honest and trace every take
ff51093 
programFrame.sceneId stuck on the pre-take scene because the buffered
render path attributed it from a peek of the latest delivered frame, with
no else branch when nothing new came back. Attribution now keys on the
delivery sequence and publishes live/holding/unknown. Each take writes one
structured record (scene, plan, layers, wall adoption, background
continuity, subscription churn), and Zoom subscription churn is counted
per source with a reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SAB8v62BEV8ihmaeGhhtY9
(cherry picked from commit 329e33c)
@iamfatness @claude
Supervise each output destination with a bounded restart ladder
6710ca6 
A supervisor sits outside each protocol's async writer, judges health from
accepted units advancing, restarts on the house 5-10-20-40-60s ladder, gives
up loudly after five failures, and bypasses the ladder for terminal
configuration errors. NDI's in-process residual risk is published, not hidden.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SAB8v62BEV8ihmaeGhhtY9
(cherry picked from commit ceb334c)
@iamfatness @claude
Record the post-reboot state in the handoff
ff0e93a 
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SAB8v62BEV8ihmaeGhhtY9
(cherry picked from commit 46b07ff)
@iamfatness @claude
Rename the churn node so the shell can parse core snapshots again
bdd5df9 
The core published subscription churn as "zoomSubscriptions", an object,
but the shell already binds that name to a list of spine subscriptions.
Every snapshot failed to deserialize, leaving the shell blind to the core
(engine status showed the JSON error, program frame count stuck at 0).
Caught on the first live launch; the core suite cannot see shell parsing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SAB8v62BEV8ihmaeGhhtY9
(cherry picked from commit 1d164b7)
@iamfatness @claude
Restore the churn test helper and point plan links at PR #419
e9d17f7 
Splitting the beta slice off the architecture branch dropped the
findChurnSource helper along with the authority tests that shared its
conflict hunk, and left two doc links pointing at plans that only exist
on the #419 branch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SAB8v62BEV8ihmaeGhhtY9
@iamfatness
iamfatness merged commit 52e061c into main Sep 10, 2026
18 of 19 checks passed
@iamfatness iamfatness mentioned this pull request Sep 10, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@iamfatness