Skip to content

CONTRACT: feat(ui-server): add GPT-Live asynchronous client delegation - #2070

Merged
schlessera merged 6 commits into
mainfrom
feat/959-gpt-live-42413928
Oct 17, 2026
Merged

schlessera merged 6 commits into
mainfrom
feat/959-gpt-live-42413928

Conversation

@schlessera

@schlessera schlessera commented Oct 16, 2026 •

Copy link
Copy Markdown
Owner

Closes #959

What changed

Adds createGptLiveConversationProvider(apiKey) for explicit registration beside Gemini in this voice stack. The primary Live API v1 WebSocket uses server Bearer authentication, gpt-live-1, PCM16/24 kHz, marin, client delegation and store:false. Native delegation remains metadata; the host owns semantic commit, its independently selected backend, tools, permissions and ordinary chat progress/questions. Later input and output continue while backend work genuinely awaits.

Correlation and lifecycle

Audio commands retain their own event identities. Delayed transcript/delegation events use echoed client_event_id; ambiguous or unknown explicit correlation closes visibly. Historical zero-duration text never becomes microphone input. Host results retain native delegation_id, use bounded quiet thinking chunks and at most one commentary append, and reject stale/consumed/replaced/aborted handles. Native redirects are refused before Bearer credentials can be forwarded. session.closed acknowledges whole-session close only; no output-done or cancellation acknowledgement is invented.

Proof

Current head 3dc08df98e155d08b9feda38ceb05ce68e82415f, repaired main base 7cc86931e763f0eca636b89abd78b3b5371ec027, tree 040dcebe25ead4aecb7cc22f32a99e3119185e7a:

  • Local preflight passes: 542 invariant tests / 3635 assertions, then 25 affected feedback tests / 105 assertions.

  • Both async adapters and diagram input binding pass 52 focused cases / 172 assertions. GPT alone has 23 cases / 72 assertions; both adapters have 42 / 123. The real host holds work pending while later speech, chat progress/tool activity and an ordinary interactive question continue. The named real-host restart regression also passes (1 case / 7 assertions) with the delivered fixture repair.

  • Hosted confirmation 39032664018: all fourteen product jobs pass, including 4,706 unit cases with no failures/skips. Every selected job and private unit-worker checkout matches f9889eb78e00c38f1f94b2013242176d079adb72. The final aggregate rejects freshness after main advancement; the whole workflow is failed and is not relabelled green. Assessed main integration and separate affected-runtime/diagnostic receipts are recorded in the merge comment.

  • Retained failing-first receipt: after completing 16 inputs, a 17th was incorrectly rejected by the host active-utterance limit. Historical identities are now bounded separately (4096), preserving the host's 16 active slots; the named regression and later input pass.

  • Retained mutations: removing the epoch check makes stale publication resolve; borrowing the latest input identity mislabels a delayed utterance; removing the history guard turns replay text into microphone input. Each failed its named assertion and was restored. These feature commits compare unchanged after base integration.

  • No live provider session or paid experiment ran.

Stack and release

Merged normally as 18a65206d4ed5f6fc4790728293ac2ce72858026, parent bc9ee2b25eb0687fa66b01536a6e15aae9c010cd, tree b671d5deeceb2b7b82418599aab567c72147808a, matching the computed merge tree. Ancestry on main and #959 closure are verified. Automatic main-push CI39034501043 is separate and remains pending. Gemini #2063 is also merged; UI #2142 and permission #2145 remain above this layer in native stack #2135. The diff adds only the GPT-Live slice and its tests/docs/minor ui-server changeset. Hosted proof retains its actual parent/head identity; unrelated later main advancement is assessed without repeatedly refreshing. Milestone 0.44.0 remains assigned to the PR and #959, with final readback and actual squash reconciliation required at merge.

Checklist

  • All fourteen selected product jobs passed and actual checkout identities are reconciled; the aggregate freshness failure and assessed integration remain explicit.
  • Acceptance criteria implemented with the limits below.
  • PR/issue 0.44.0 milestones read back before merge and after issue closure.
  • Additive contract, API report and minor changeset included.
  • Phase-1 dictation and host authority preserved.

Anything left open

#961 owns separately authorized live handshake, device/audio, latency and safety qualification. Transcript deltas lack a finalization event and origin/certainty remain unknown, so the client must require manual review. Exact permission speech is unsupported; echo isolation and word alignment remain unproven. store:false does not certify zero retention. Client capture/playback and permission-specific behavior are separate stack layers.

@schlessera schlessera added this to the 0.44.0 milestone Oct 16, 2026
@schlessera schlessera added type: feat A new capability or a change in user-visible behaviour. area: ui-sdk Wire protocol, runtime schemas, the server and client SDK surfaces. area: ui-server Hono app factory: turn coordinator, auth, session catalog, routes, cron. area: backends AgentBackend implementations and the seam they share. contract Touches the integration contract: CONTRACT: commit, contract doc updated in the same commit. labels Oct 16, 2026
@schlessera
schlessera force-pushed the feat/959-gpt-live-42413928 branch from e525be3 to 5d1d977 Compare October 16, 2026 12:25
@schlessera
schlessera changed the base branch from main to feat/958-gemini-live-42413928 October 16, 2026 12:25
@schlessera
schlessera added this pull request to stack #2135 October 16, 2026 12:25
@schlessera
schlessera marked this pull request as ready for review October 16, 2026 12:27
@schlessera

Copy link
Copy Markdown
Owner Author

Additional reproduction from voice stack #2135, without another main refresh:

  • Gemini PR CONTRACT: feat(ui-server): add Gemini Live async conversation adapter #2063 head f98cf7bbd33f2bac0205105b774be9e63834076a has a complete passing confirmation 38900599104, including the aggregate.
  • GPT-Live PR CONTRACT: feat(ui-server): add GPT-Live asynchronous client delegation #2070 head 5d1d977d62ae72d8a6de00251ec8a2753eafee67 declares live parent f98cf7bbd33f2bac0205105b774be9e63834076a. All fourteen selected test/verification jobs passed in 38900601821; its aggregate rejected the requested/actual merge-parent identity. This is not a passing aggregate or merge receipt.
  • That run's metadata checkout is 83080eb5a06d8d0e56d86780fa576662888fb378, tree b3e713bde91ef5e01f805e27df7a02bdbddfdcca, parents 5b349e2b03e8b449cd3c6de346f3e1c765ee632f and 5d1d977d62ae72d8a6de00251ec8a2753eafee67.
  • Independent immutable Git commit API readback identifies 5b349e2b03e8b449cd3c6de346f3e1c765ee632f as the lower synthetic merge, parents 3fc6fea5e22304df2810348074c32b1983fbea46 and f98cf7bbd33f2bac0205105b774be9e63834076a, tree 0fa659d7bc15cabe033cb74d0f57889b172595a3.

This matches #2141's native merge-chain shape. Metadata/immutable-object evidence is recorded here; no independent per-job identity reconciliation or replacement validation policy is claimed. The tested heads remain intact. No unrelated-main rebase or aggregate bypass is being used.

@schlessera

Copy link
Copy Markdown
Owner Author

Gemini PR #2063 is merged as cd5e8bb04b0d0e24f1a6c4d4f73e17f1b3fe4908. GitHub rebased this native member onto it and retargeted it to main. Current #2070 head: da85ded4c25367a086fbee50e12ad99b5bc95249. Range-diff verifies all six GPT-Live commits unchanged. Current local loopback proof passes both adapters: 42 tests / 123 assertions, including host async work with continuing chat progress and interactive questions; source citations are anchored.

Current-head confirmation 38933150766 is pending with its first job waiting for a hosted runner. Contract run 38933095529 is queued. The earlier upper-layer confirmation's fourteen selected jobs passed, but its native parent-identity aggregate failed under #2141; that historical run is not substituted for this new candidate. The PR remains ready, milestone 0.44.0, awaiting current hosted qualification before the next dependency-order native merge.

@schlessera

Copy link
Copy Markdown
Owner Author

Continuing delivery on 2026-10-17. Prior confirmation 38933150766 completed with unit failures and a stale-base aggregate; it is no longer described as queued or passing.

The failed profile/model, cancellation, bridge snapshot and admission expectations already have repairs on main (#2183, #2193, #2199 and #2251). I reproduced the four stale profile assertions before integration, then rebased the six GPT commits onto fixed base 8c4d191e761bec31f3267d3b612a77712692f54a. Range-diff verifies all six commits unchanged. Current pushed head: 1a6cb2cefa0326f4e65421898e94fc3c4eb02122.

Local preflight now passes completely: 541 fast invariant tests, then 25 import-graph feedback tests. The targeted eleven-file regression run passes 149 cases, including both async adapters, registry/model expectations, cancellation, runtime records, admission, snapshots and isolated unit-worker controls. Two interactive policy-probe cases failed because the fixture selected a host-home Python unavailable inside the worker; unchanged main reproduces both. Selecting system Python through a command-scoped PATH passes the entire 19-case policy boundary suite / 587 assertions with all namespaces, mounts, nonempty attack checks and offline guards intact. The fixture portability issue is filed separately as #2304; no product guard or test assertion was changed here.

Fresh confirmation 39023395726 passed metadata/lint/retention and has selected the full categories. Its metadata job 117369650156 actually checked out e3da8cef9ce9a4f4cbdd713bbd85aceadee792fa, parents fixed base 8c4d191e761bec31f3267d3b612a77712692f54a / current head, tree 0c5ecb958f1a4f0a9c0352fe70208d581ca9bcb4. This identifies its tested candidate; other main advancement is not substituted into that receipt. Complete hosted proof remains in progress, not green.

UI #2142 and permission #2145 have also been rebased in their own layers and pushed, resolving the API-report and documentation conflicts while preserving main's Mermaid signature and current contracts. The top layer again passes 39 pinned real-browser scenarios and both native approval-recovery cases, whose setup completes the current package build. Those layers remain draft for dependency-order delivery; no native aggregate waiver is implied. All existing release milestones remain 0.44.0.

@schlessera

Copy link
Copy Markdown
Owner Author

Current voice adapter confirmation run 39023395726 at head 1a6cb2cefa0326f4e65421898e94fc3c4eb02122 exposes the same diagram binding prerequisite already recorded by #2293. Actual checkout e3da8cef9ce9a4f4cbdd713bbd85aceadee792fa has parents 8c4d191e761bec31f3267d3b612a77712692f54a and that voice head.

Unit (3), job 117369944061, fails the exact diagram binding test at tests/bound-browser-documents.test.ts:40: the declaration has no binding, so reading its values throws. The unchanged focused suite reproduces locally: eight pass, one fails on that same expression. Both actual diagram owner/caller files are byte-identical to the tested main base. Their SHA-256 values are respectively d7280cf23ba24bd636f7737d8552801e37f8c03c1e773aee660c9e07ded524a9 and dbac044e90a3b7b84e308a96cca8396874afb9b96d8cc23b2fddb4293d85706c; neither matches its declaration. The voice diff changes neither the checker nor these producers.

Unit (4), job 117369944070, separately fails the real-host restart case because its fixture tries to create an existing inode-pin directory. This is the exact helper line repaired by merged #2305. A controlled run applying only that delivered helper passes the named real-host restart test (one pass, seven assertions); the original helper was then restored byte for byte. This is a scoped upstream-repair control, not a new current-head whole-run receipt. Other completed selected jobs currently pass, including both browser shards, runtime, native host, endurance, verify and packaging; remaining jobs are still in progress. This is not a whole-run success.

Resume #2070 by incorporating the delivered restart repair and #2293 after it lands, verifying both named failures and the voice contracts, then requesting current selected confirmation. Keep the diagram proof correction with its existing owner; no gate weakening or unchanged failing-job rerun is proposed.

@schlessera
schlessera marked this pull request as draft October 17, 2026 11:00
@schlessera
schlessera force-pushed the feat/959-gpt-live-42413928 branch from 1a6cb2c to 6c06f6b Compare October 17, 2026 11:05
@schlessera

Copy link
Copy Markdown
Owner Author

Pushed the prepared voice stack in one atomic update on fixed base 9d5cd00b7646ffee951c7a29a38717989f7fd868:

Range-diff verifies every feature commit unchanged. Delivered fixture repairs #2305 and #2317 are incorporated. Current GPT local preflight passes 541 invariant tests plus 25 affected tests; both loopback adapters pass 42 tests / 123 assertions, including async host work with continuing chat progress and interactive questions. The broader scope command's unexplained SIGKILL is recorded under #2074; it is not a full-suite pass.

Historical full confirmation 39023395726 finished with thirteen selected jobs passing, the two diagnosed main fixture failures, and a stale-base aggregate failure. All fifteen actual selected checkout logs independently agree on e3da8cef9ce9a4f4cbdd713bbd85aceadee792fa; these receipts remain historical, not current-head success.

The deterministic diagram binding now has a smaller repair, #2322 / #2314. It remains draft behind existing discovery failure #2153; no need to wait for the larger #2293 retention change. These three voice PRs are draft, with milestones 0.44.0, to avoid another full run against the known-broken checker.

Current recovery setup hits its existing 180-second package-build deadline before assertions; #1600 now contains named phase evidence (ui-react begins at 173.8 seconds). The queued browser attempt then skips all 39 cases because that interrupted build left styles.css absent. Neither is passing coverage. A separate full bun run build then passes. With that prerequisite restored, the current top layer freshly passes all 39 pinned Chromium conversation cases in 29.54 seconds. This resolves the missing CSS prerequisite for browser execution; it does not repair the recovery hook or count as executing its two scenarios. Latest draft metadata/contract gates pass; intentionally skipped full categories are not merge qualification.

Resume once the reviewed source-binding repair is delivered: incorporate that specific repair, verify the named binding/restart cases and current async behavior, obtain current selected hosted proof, then merge the eligible native prefix in dependency order. Keep actual tested identities; unrelated main advancement alone does not require refreshing. No guard, timeout or CI-policy relaxation is included.

Use the primary Live v1 connection and server Bearer auth, preserving host-owned commit, backend execution and interactive chat progress. Correlate original transcript fragments by echoed audio command IDs and reject uncorrelatable multi-utterance input; no complete-turn or per-output cancellation signal is fabricated.

Proof: 19 real loopback adapter/host tests and 49 assertions pass. Epoch and delayed-input identity mutations fail their intended rejection/misattribution assertions, then pass after restoration. Focused preflight passed 448 tests / 2793 assertions; strict types passed before final resync fixture addition. No paid or live provider calls.
…phone input

Ignore zero-duration appended text and require command correlation after a historical resync. Removing the zero-duration guard makes the named history test fail by adding Historical context to the fresh input; restored suite passes 20 cases and 50 assertions. This tightens the original-input and resync criteria of #959.
…ities

Apply the same explicit audio identity validation to delegation metadata as to transcript deltas. An unknown supplied command id must not fall back to implicit host binding. The 21-case real loopback suite passes 51 assertions.
…nstalled types

Use the installed OpenAI Live event shapes for nonempty session.started, input transcript and client delegation fixtures. The session expiry uses the pinned fictional fixture date. The 21-case loopback suite remains green.
…redentials

Exercise native Bun redirect refusal against two loopback servers, preserve complete WebSocket descriptors, and make history/foreign-identity assertions wait for their actual event evidence. The focused suite passes 22 tests and 54 assertions.
…ission

The provider must retain historical command identities without treating settled utterances as live host slots. Use the existing bounded history capacity; the host still enforces active utterance limits. A 17-request loopback regression first failed at the expected resolves assertion because input 17 rejected with identity capacity exceeded; after the fix the suite passes 23 cases and 72 assertions. Audited other historical/pending ledger limits in the adapter.
@schlessera
schlessera force-pushed the feat/959-gpt-live-42413928 branch from 6c06f6b to 3dc08df Compare October 17, 2026 12:28
@schlessera
schlessera marked this pull request as ready for review October 17, 2026 12:28
@schlessera

Copy link
Copy Markdown
Owner Author

Delivery resumed after #2326 landed the diagram binding repair. Current pushed heads: #2070 3dc08df98e155d08b9feda38ceb05ce68e82415f, #2142 3eae038d092c7ea8f498f5b02e0c164bedb385cd, #2145 97ab5adaf5a5956179f0ba846615daa3a3be69b6, base 7cc86931e763f0eca636b89abd78b3b5371ec027. GPT and permission feature commits compare unchanged; the UI range-diff differs only in an upstream backend citation offset, verified against the source.

Current focused adapters/binding proof passes 52 cases / 172 assertions; the named restart case also passes. GPT complete local preflight passes 542 invariant tests plus 25 affected feedback tests. Top-layer actual Chrome recovery passes both cases / 16 assertions (163.71 seconds), and all 39 pinned Chromium conversation cases pass (28.93 seconds). Delivered forward-closure builds complete under the unchanged setup budget; no guard or timeout change is included here. Upper-layer local preflights remain in progress, with UI fast invariants already passing 724 cases.

GPT is ready and confirmation 39032664018 is running all fourteen selected product categories. Contract and metadata pass. Actual checkout f9889eb78e00c38f1f94b2013242176d079adb72 has parents recorded base/current GPT head, tree 040dcebe25ead4aecb7cc22f32a99e3119185e7a; immutable API and fetched Git object agree. This identifies its candidate, not a completed whole-run result. Upper layers remain draft for dependency-order delivery. The obsolete #2314 blocker has been removed from #959.

@schlessera

Copy link
Copy Markdown
Owner Author

All fourteen selected product jobs in 39032664018 pass. Actual checkout f9889eb78e00c38f1f94b2013242176d079adb72, parents 7cc86931e763f0eca636b89abd78b3b5371ec027 / inspected head 3dc08df98e155d08b9feda38ceb05ce68e82415f, tree 040dcebe25ead4aecb7cc22f32a99e3119185e7a, is independently confirmed by all fifteen selected job logs/resources and all eight private unit-worker checkouts. Unit reports total 4,706 passes, zero failures/skips. The final aggregate fails only Candidate became stale after main advanced; this workflow is not reported as successful.

Main additions were assessed: #1870 changes HTTP fixtures/documentation; #2335 changes restricted relay IPC validation. The composed current-main candidate uses base 9c52e8c020aca6d8c4dcea0a2afd1640bc1bf60b, inspected voice head, tree b21832683705f66cd9d9cd664ed0112d518b070b. All GPT feature paths remain byte-identical. This actual combined tree freshly passes 112 adapter, binding, launcher/runtime-placement/sandbox tests / 666 assertions, including real restricted workers, with guards intact.

The additional native qualification first reproduces the existing #2315 SDK whole-profile parent timeout: 24 identity plus 13 egress controls pass before its unchanged 25-second caller exits143. This is not a passing full native receipt. A separately labelled diagnostic changes only that caller to #2315's selected four-turn composed allowance; all existing inner turn/cancellation deadlines and every assertion remain. It completes both SDK profiles, all 28 restricted-relay checks, actual managed probes, prepared native consumers and cleanup-failure controls, with zero owned resources and complete loop/mount cleanup. The fixture is then restored byte-for-byte and the combined tree is clean. This is controlled affected-runtime evidence, not clean-source whole-suite qualification or delivery of #2315.

Following the maintainer instruction to push tested work and avoid repeated full refreshes for main advancement, retain these actual identities and use the normal native prefix merge with final head/main/milestone readback. No old passing run is relabelled as the newer combined tree; the freshness failure and diagnostic limitation remain explicit.

@schlessera
schlessera merged commit 18a6520 into main Oct 17, 2026
17 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: backends AgentBackend implementations and the seam they share. area: ui-sdk Wire protocol, runtime schemas, the server and client SDK surfaces. area: ui-server Hono app factory: turn coordinator, auth, session catalog, routes, cron. contract Touches the integration contract: CONTRACT: commit, contract doc updated in the same commit. type: feat A new capability or a change in user-visible behaviour.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ui-server: adapt GPT-Live client delegation to existing host backends

1 participant