Skip to content

P2: screen-scraped question replaces the authoritative transcript prompt — multi-question payload is order-dependent and frozen #730

Description

@RonenMars

Successor to #721 (whose stated scope — multi-select detection and glyph stripping — is met and verified on v1.70.4). Found by the Group C probe, phase D, 2026-08-28.

What

The same multi-question AskUserQuestion form, same session, same tool call, is published either as the correct 2-question payload or as a degraded 1-question scrape of the visible tab (with the TUI affordances Type something / Chat about this taken as options).
Across 11 distinct prompts: 6 degraded, 5 correct, and the correlation with the producer is exact — every degraded payload has provenance: {source: "screen", confidence: "inferred"}, every correct one {source: "transcript", confidence: "authoritative"}.
Whichever payload lands is frozen for the prompt's life: the mapper never re-scrapes and revision counts state transitions only, so a client that catches the degraded parse cannot recover (one run: 310 s).

Why (origin/main @ 04d6760, src/api/handlers/sessions.handlers.ts)

Two producers feed one prompt slot and the last writer wins, with no regard for authority:

  • handleJsonlQuestion (:1185–1220): if the content key matches the prior prompt it upgrades it in place to transcript; if the lossy scrape's key differs, it cancels the prior as replaced and opens a new transcript prompt. Screen-then-transcript therefore ends correct.
  • handleLiveQuestion (:1162–1183): replaces any prior open prompt as replaced and opens a screen prompt without checking whether the prior is the authoritative transcript one. Transcript-then-screen therefore ends degraded.

The order between the two is not fixed (observed gaps from below capture resolution to ~6 s in either direction), so the outcome is a race.

Why P2, not P1

Safety holds on v1.70.4: both variants publish inputMode: "multi", the app renders a disabled card with no tappable rows, and a well-formed direct answer is refused 400 unsupported_prompt_shape with zero bytes.
Fidelity does not: half the time the client is shown one question of two, with two non-options presented as options, permanently for that prompt.

Done looks like

  • An authoritative (transcript) prompt is never replaced by an inferred (screen) one for the same question: handleLiveQuestion keeps the transcript prompt when one is open (or upgrades in place is already handled by the JSONL side); screen-then-transcript keeps upgrading in place / replacing as today.
  • No schema change is needed — the discriminator (provenance.confidence) is already on the wire.
  • Test on the real path with both orderings (transcript first, screen first) asserting the surviving prompt is the transcript one in both cases and that exactly one open prompt exists; mutation: restore unconditional replacement → the transcript-first case red.
  • Related: P3: synthesized legacy toolUseId keys on content-key length, so different questions can collide #727 (the screen path's synthesized toolUseId keys on key.length).

Evidence: ai-investigation-claude/tracks/C-opus5-medium/evidence-scrubbed/ (phase D, D-row7-tap, D-row7-after-dismiss, D-recon-tap3; provenance table in PHASE-D-CLOSEOUT.md).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions