Skip to content

Let the candidate control when agent takes the turn - #192

Merged
jserv merged 2 commits into
mainfrom
turn-taking
Oct 1, 2026
Merged

jserv merged 2 commits into
mainfrom
turn-taking

Conversation

@jserv

@jserv jserv commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Jim answered the pause a candidate took to find the next word, and kept talking after a candidate asked for time to think. The first commit raises the default silence window from one second to three, tells the interviewer a pause within a sentence is not a finished answer, and adds "Your turn is done" (a button, Alt+Enter, or a click on a countdown ring that fills through the window while the microphone is quiet), which ends the audio stream so Gemini answers without waiting it out. The second adds a thinking hold: a Thinking button or an explicit spoken request in English ("let me think", "can you give me a minute") keeps the floor, generated replies are dropped, nudges wait, hints and endings are refused, and evidence and the deadline carry on. Speaking again, Continue or yielding ends it, and so do the five-minute warning, the round change and one check-in after two silent minutes.

The trade worth a reviewer's eye is the longer window: three seconds would slow every reply, which is why the yield exists, and it only helps if ending the audio stream really ends the turn. A probe against the live API (9 sessions, gemini-3.1-flash-live-preview, 3s window) measured the first reply audio at a median 0.76s after the last word with audioStreamEnd against 3.36s without; the next utterance after it was detected normally, and 1 turn in 12 ignored the end and fell back to the full window. The same probe showed inputTranscription.finished is never sent and the transcript arrives about 0.3s after the stream ends, which is what the pre-click transcript window in the hold is sized from. Spoken requests are matched by phrase, so recognition is English only and deliberately strict: short phrases such as "one second" or "hold on" count only as a whole sentence.

Verified with scripts/test.sh at each commit (all gates pass; at the tip 1061 Rust tests and 647 browser tests, with the Playwright lane skipped because Chromium is unsupported on this Ubuntu 20.04 host). Before the rebase onto current main: with Chromium 130 linked in by hand the browser suite passed 753 of 757, the 4 failures being PDF tests that fail identically on main, and credentialed scripts/browser-check.sh with the rust agent passed the voice, barge and report flows, and a scripted run against a real room clicked Thinking, saw Jim stay listening for the whole hold, released it by speaking and by Continue, and yielded with Alt+Enter. scripts/gemini-check.sh gets setupComplete with the new window. cargo mutants --in-diff over the change found 10 missed and 4 timeouts; after new tests, and with the three functions that only write to the room added to the exclusion list with the reason, a re-run over the affected functions caught all 63 viable mutants. It was not re-run after the final cleanup. The prompt golden changes on purpose: contract bundle 22, live prompt 14.

Closes #140
Closes #172


Summary by cubic

Lets candidates control when Jim takes or gives up the turn. The default silence window increases from one to three seconds so a pause mid-sentence is not treated as a finished answer, while Your turn is done (a button, Alt+Enter, or the countdown ring) ends the audio turn immediately.

  • Adds a Thinking hold started by the Thinking button or an explicit spoken request in English; replies generated during a hold are dropped and hints, nudges, and endings wait.
  • Ends the hold when the candidate speaks, chooses Continue, yields the turn, or after the five-minute warning, the round transition, or a single check-in after two silent minutes.
  • Keeps the editor, microphone, evidence, and deadline active throughout, and records thinking lifecycle events in evidence and replay views.
  • Covers review gaps: failed cold briefings no longer drop their reply, spoken holds no longer inherit the button's release cooldown, the turn ring stops when the mic fails or no interviewer is present, and system events can end a hold.
  • Bumps the contract bundle to 25 with live prompt 17. Closes Jim often cuts in before I finish speaking #140 and Interviewer keeps talking even I asks for time to think #172.

Written for commit 271e939. Summary will update on new commits.

Review in cubic

@jserv
jserv requested review from ColtenOuO and alanhc September 30, 2026 13:20
@jserv jserv changed the title Let the candidate control when Jim takes the turn Let the candidate control when agent takes the turn Sep 30, 2026
@jserv
jserv force-pushed the turn-taking branch 3 times, most recently from 629cb84 to 5727b3d Compare September 30, 2026 21:58
@sysprog21 sysprog21 deleted a comment from cubic-dev-ai Bot Oct 1, 2026
cubic-dev-ai[bot]

This comment was marked as resolved.

Comment thread src/agent/turn_taking.rs
Comment thread src/agent/turn_taking.rs Outdated
@jserv
jserv force-pushed the turn-taking branch 2 times, most recently from 0c6c00d to 80b9c52 Compare October 1, 2026 20:39
jserv added 2 commits October 2, 2026 05:42
Jim answered the pause a candidate took to find the next word: Gemini
ended the turn after one second of silence, and a second-language
speaker pauses longer than that mid-sentence. The default window is now
three seconds, and the live prompt says a pause within a sentence is not
a finished answer. A longer window alone would slow every reply, so the
page adds "Your turn is done" (Alt+Enter, or a click on the ring), which
ends the audio stream so Gemini answers at once: in a measured session
the first audio came about 0.8s after the last word instead of 3.4s. A
ring beside the status line fills through the window the agent publishes
as a participant attribute while the microphone is quiet, and resets on
speech, so the moment Jim takes the turn is visible before it comes. The
prompt golden changes on purpose: contract bundle 25, live prompt 17.

Close #140
Jim kept talking after a candidate asked for time to think: "let me
think" was just another turn to answer, and silence alerts, hints and
editor reviews filled the gap within seconds. A Thinking button, or an
explicit spoken request in English, now holds the floor. Replies the
model generates meanwhile are dropped, nudges and editor reviews wait,
and hints and endings are refused, while editor, microphone and test
evidence stay live and the deadline does not move. Speaking again,
Continue or "Your turn is done" ends the hold, and so do the five-minute
warning, the round change and one check-in after two silent minutes. A
spoken request stays provisional until its utterance ends, so reasoning
that merely begins with "let me think" is not a hold, and the transcript
of speech from before a click cannot end the hold it was clicked into.
The model is told when something it said during a hold went unheard,
test runs reach it as context it does not answer, a reconnection during
a hold is briefed without being asked to speak, and a Continue whose
reply never comes is asked for after four seconds. The prompt golden
changes on purpose within bundle 25.

Close #172
@jserv
jserv merged commit b87121c into main Oct 1, 2026
15 checks passed
@jserv
jserv deleted the turn-taking branch October 1, 2026 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Interviewer keeps talking even I asks for time to think Jim often cuts in before I finish speaking

2 participants