feat(runtime): ask an unattended turn to prove the task's requirements

CONTEXT.md

@@ -0,0 +1,78 @@
+# Woopcode
+
+A terminal-native coding agent: a streaming agent loop driving a React Ink
+interface, and the same loop driving a headless single-prompt path. This file is
+the glossary — what the words mean here. Mechanism lives in the code, and the
+reasoning behind a hard-to-reverse choice lives in `docs/adr/`.
+
+## Language
+
+### A unit of work
+
+**Turn**:
+One stretch of work answering one user message, from the loop being entered to
+it returning an answer. A turn survives being asked to continue at the iteration
+ceiling; it is still the same turn.
+_Avoid_: request, session, conversation
+
+**Iteration**:
+One provider response and the tool calls it carried. The unit both budgets and
+every counter are measured in.
+_Avoid_: loop, cycle, round
+
+**Step**:
+An iteration, when counted against what remains rather than what has happened.
+The wall budget is converted into steps at the rate the turn has been running
+at, so one warning can serve both budgets.
+
+**Turn-initiating message**:
+The message that started the turn — the last real conversation turn present when
+the loop was entered. In a headless run it is the task statement; in the
+interface it is what the user just typed. Pinned into the window for the life of
+the turn, because the loop pushes messages of its own that would otherwise crowd
+it out.
+_Avoid_: the prompt, the first message, the task
+
+**Window**:
+The tail of the transcript actually sent to the provider, counted in
+conversation turns rather than messages. Distinct from the transcript, which is
+everything the turn has accumulated. Its turn ceiling bounds the tail, not the
+whole window: a pinned turn-initiating message rides outside it, so a window can
+hold one turn more than the ceiling names, and never two.
+
+### Ending a turn
+
+**Finish gate**:
+A check that runs when the model responds without calling a tool, deciding
+whether the turn may end or must go round once more. There are two, and both may
+answer at once, in which case the turn receives a single message.
+
+**Verification reminder**:
+The finish gate that fires on unverified edits. Evidence the loop can see.
+
+**Requirement gate**:
+The finish gate that fires on an unattended turn stopping early with budget in
+hand, asking it to prove each stated requirement with command output. Aimed at
+what the loop cannot see: work that was checked thoroughly against the wrong
+property.
+
+**Unverified edits**:
+A turn that changed the workspace and ran no shell command afterwards. Order
+within an iteration is what decides it, so this is counted in tool executions
+rather than iterations.
+
+**Unattended**:
+Nobody is reading the answer as it arrives, so a wrong one stands. True of the
+headless path, false of the interface — where a person can correct a turn for
+the cost of one sentence.
+_Avoid_: headless, non-interactive, automated
+
+**Wind-down warning**:
+The message telling the model its turn is nearly over and to start nothing new.
+Derived from a rate measured on the turn itself, so it can clear again when the
+rate recovers.
+
+**Budget**:
+What bounds a turn. Two of them — iterations and wall-clock seconds — and a turn
+stops on whichever binds first. Neither is a quota: the provider enforces that
+itself.

commands/agent.tsx

@@ -205,6 +205,11 @@ async function runHeadless(
 
   const selectedModel = await resolveModel(options.model);
   store.setSelectedModel(selectedModel);
+  // Half of "nobody is watching this run". This half is the approval path's,
+  // read through the store; the other half is `controller.setUnattended` below,
+  // which is the loop's. They are not merged because `runtime/loop.ts` must not
+  // import the interface's store — so the one fact is stated twice, on purpose,
+  // and each site names the other.
   store.setNonInteractive({ autoApprove });
 
   const log = createEventLog(options.events);
@@ -302,6 +307,10 @@ async function runHeadless(
   };
 
   const controller = new AgentController(provider, apiKey, selectedModel, callbacks, baseUrl);
+  // The other half of the fact stated at `store.setNonInteractive` above:
+  // nobody is reading the answer as it streams, so the loop's finish gates
+  // apply to this run.
+  controller.setUnattended(true);
   await controller.initialize(options.session);
 
   // On stderr, not stdout: stdout is the agent's answer and a caller pipes it.

commands/agentController.ts

@@ -103,6 +103,8 @@ export class AgentController {
    * first edit of the next session.
    */
   private sessionMode: SessionMode = "build";
+  /** See setUnattended. False for the TUI, true for `-p`. */
+  private unattended = false;
   /**
    * The session this turn belongs to. Null until `initialize`, and null for the
    * lifetime of a run started with persistence off.
@@ -176,6 +178,19 @@ export class AgentController {
     return demo ? new Error(demo) : error;
   }
 
+  /**
+   * Declares that nobody is watching this session's turns.
+   *
+   * Set by the headless path only, beside the store's own non-interactive flag.
+   * A property rather than an argument to `run`, because it is true of the
+   * session rather than of one prompt — and deliberately not inferred inside
+   * the loop from a missing callback, which would extend the behaviour to every
+   * embedder that happens not to pass one.
+   */
+  setUnattended(unattended: boolean) {
+    this.unattended = unattended;
+  }
+
   getSessionMode() {
     return this.sessionMode;
   }
@@ -291,7 +306,7 @@ export class AgentController {
         !conversational,
         // Snapshotted as the turn starts, so a Tab pressed while it runs applies
         // to the next turn rather than changing the rules underneath this one.
-        { planMode: this.isPlanMode() },
+        { planMode: this.isPlanMode(), unattended: this.unattended },
       );
 
       const assistantText = response || this.pendingAssistantText;

config/config.ts

@@ -491,9 +491,51 @@ function isConversationTurn(message: Message | undefined): boolean {
   return message?.role === "user" && !message.images?.length;
 }
 
+/**
+ * Where the message that started this turn sits in the transcript.
+ *
+ * The last real conversation turn at the moment the loop is entered: in a
+ * headless run that is the task statement, and in the TUI it is what the user
+ * just typed. Captured as an index rather than a reference because the array
+ * only ever grows by pushing, so the index stays true for the whole turn while
+ * an identity check would rest on nothing written down.
+ *
+ * Undefined for a transcript with no conversation turn in it at all, which is
+ * read as "nothing to pin" rather than defaulting to the first message.
+ */
+export function turnInitiatingIndex(messages: Message[]): number | undefined {
+  for (let i = messages.length - 1; i >= 0; i--) {
+    if (isConversationTurn(messages[i])) return i;
+  }
+  return undefined;
+}
+
+/**
+ * The window sent to the provider: the last `maxTurns` conversation turns, plus
+ * the message that started the turn wherever that has fallen out of them.
+ *
+ * **`maxTurns` is a bound on the tail, not on the window.** A pinned request
+ * carries `maxTurns + 1` conversation turns, and never more — the one exception
+ * in a budget that is otherwise a hard boundary, and the only place in this
+ * file where the number of turns sent exceeds the number asked for. It is worth
+ * stating because every other context decision treats that ceiling as absolute:
+ * a reader sizing a prompt from `MAX_TURNS` alone will be one message short.
+ *
+ * The pin exists because the loop itself pushes user messages — the wind-down
+ * warning, the finish gates, a truncated-stream resume — and every one of them
+ * counts as a turn here. Six of those and the window no longer holds the
+ * question being answered: a benchmark trial ran 200 iterations off a single
+ * prompt, and the gate that asks a model to re-read its task would otherwise be
+ * naming something the model can no longer see.
+ *
+ * The extra message is the cheapest in the window — one prompt, no tool results
+ * — and it is only ever prepended when it is genuinely outside the tail, so a
+ * short conversation assembles exactly as it did before this existed.
+ */
 export function recentMessages(
   message: Message[],
   maxTurns: number,
+  pinnedIndex?: number,
 ): Message[] {
   if (maxTurns <= 0 || message.length === 0) {
     return [];
@@ -513,5 +555,16 @@ export function recentMessages(
     }
   }
 
-  return message.slice(startIndex);
+  const window = message.slice(startIndex);
+
+  if (
+    pinnedIndex === undefined ||
+    pinnedIndex >= startIndex ||
+    pinnedIndex < 0 ||
+    pinnedIndex >= message.length
+  ) {
+    return window;
+  }
+
+  return [message[pinnedIndex]!, ...window];
 }

config/types.ts

@@ -123,6 +123,17 @@ export interface TurnSummary {
   salvagedIterations: number;
   /** Times the turn was asked to check its own edits before finishing. */
   verificationReminders: number;
+  /** Times the turn was asked to re-check the task's requirements before finishing. */
+  requirementReminders: number;
+  /**
+   * Whether a tool ran after the requirement gate fired.
+   *
+   * Absent when the gate never fired, which is a different thing from firing to
+   * no effect: the failure this gate is aimed at is a model that answers the
+   * question in prose, from memory, and runs nothing — and in the score alone
+   * that is indistinguishable from a run where the gate never mattered.
+   */
+  requirementGateActedOn?: boolean;
   toolCalls: number;
   /**
    * Index of the last tool execution that changed the workspace, counting from

docs/adr/0002-finish-gates-for-an-unattended-turn.md

@@ -0,0 +1,145 @@
+---
+title: Finish gates for an unattended turn
+type: concept
+summary: Why a turn that stops early is asked to prove the task's requirements, why the task statement is pinned into the window, and which alternatives were rejected.
+prerequisites: []
+related:
+  - /docs/adr/0001-wall-clock-budget-for-the-agent-loop
+since: 1.1.0
+---
+
+# Finish gates for an unattended turn
+
+Status: accepted
+
+A turn ends when the model responds without calling a tool. Until now one thing
+could overrule that: a turn that had changed files and run nothing to check
+them was asked, once, to verify. That gate fires on evidence the loop can see.
+The failure it cannot see is a turn that verified something thoroughly, and
+verified the wrong thing.
+
+## The measurement
+
+Two of the three failed trials in the `jobs/tb2-post-1.1` run ended that way,
+early and confident, with most of both budgets unspent.
+
+| task | stopped at | of ceiling | wall used | what it did |
+| --- | --- | --- | --- | --- |
+| overfull-hbox | iteration 59 | 200 | 26% | ran its chosen check three times, declared success |
+| video-processing | iteration 83 | 200 | 7% | confident, wrong |
+
+`overfull-hbox` is the instructive one. It ran `pdflatex` and a search for
+overfull boxes three times over, so the unverified-edits gate had nothing to
+say — the edits *were* checked. The task also constrained which wording was
+permitted, and nothing it ran tested that. CLAUDE.md already recorded the
+pattern from an earlier run: two trials that reported success with accurate
+self-verification still scored zero, because they verified the wrong property.
+
+These are the cheapest points on the board. The agent had roughly 70% of both
+budgets left and chose to stop.
+
+## What was decided, and the alternatives
+
+**A second finish gate, asked once, headless only.** When an unattended turn
+responds with no tool calls and has budget left to act, the loop injects one
+user message: enumerate every requirement the task states, including
+constraints on what is *not* allowed, and quote the command output that proves
+each one — with recollection explicitly refused as evidence.
+
+Interactive turns are excluded because a person is reading the answer and can
+correct it for the cost of one sentence, and because a conversational turn has
+no requirements to enumerate. The loop learns this from an explicit
+`unattended` option rather than by inferring it from a missing optional
+callback, which would have handed the behaviour to every embedder that happened
+not to pass one.
+
+Rejected: **strengthening the system prompt.** Free, and it is where the
+instruction to verify already lives — which is the argument against it. The
+model is told to verify today and did, three times, against the wrong property.
+A prompt is read once at the start of a turn that goes on to run for hundreds
+of iterations; a loop mechanic fires at the moment the mistake is being made.
+
+Rejected: **firing only on turns that wrote something.** Narrower, but it
+misses a task whose deliverable is a written answer, and the observed failure is
+independent of whether files changed.
+
+**The gate needs ten steps, and no wind-down warning outstanding.** Ten is twice
+the wind-down threshold. The gate asks for work, and inside that zone the loop
+is telling the model the opposite — finish what you started, begin nothing new.
+The flag is read as well as the count because the count is derived from a
+measured rate that moves: it can recover past the floor while the model is still
+under a warning issued earlier.
+
+**Both gates answer with one message.** They are cheap in round trips and
+expensive in window: every message the loop pushes costs one of the six
+conversation turns the window keeps, and losing the window is what this gate
+exists to correct.
+
+**The task statement is pinned into the window.** The gate tells the model to go
+back to the task statement above, and that has to be true. The window counts
+user messages, and the loop pushes user messages of its own — the wind-down
+warning, the finish gates, a truncated-stream resume — so six of them and the
+question being worked on has left the request. The loop captures the
+turn-initiating message when it is entered and `recentMessages` carries it back
+in when the window has moved past it.
+
+This makes the turn ceiling a bound on the *tail* rather than on the window: a
+pinned request carries one conversation turn more than `MAX_TURNS` names, and
+never two. That is the only exception in a budget every other context decision
+treats as absolute, so it is written down here as well as at `recentMessages` —
+the extra message is one prompt, carrying no tool results, and it is prepended
+only when it has genuinely fallen out of the tail.
+
+Rejected: **quoting the task into the gate's message instead.** Self-contained
+and needs no context change, but a long turn still argues from a question it
+cannot see. Rejected: **pinning `messages[0]`.** Wrong for an interactive
+session, where the first message is usually a greeting, and wrong under
+`--resume`, where it came off disk after trimming.
+
+**The duplicate threshold is cleared when the gate fires.** The two collide head
+on: the check a turn most needs to re-run is usually the one it has already run
+twice, where the loop answers that the result is already in the conversation —
+pointing at output the window dropped long ago. An amnesty rather than an
+exemption, since the gate fires once and only above the step floor.
+
+The reset is wholesale — every previously exhausted call may run again, not only
+the one the gate is asking about — so what bounds it is that it happens once and
+that suppression resumes immediately: the threshold counts again from zero, and
+a third identical call after the gate is refused exactly as it would have been
+before. A narrower amnesty, scoped to calls that classified as verification,
+was considered and rejected: `TurnState` does not record a classification per
+key, and adding one buys a distinction the step floor already pays for.
+
+## How it will be judged
+
+`TurnSummary` gains `requirementReminders` and `requirementGateActedOn`, both
+written to `run_end` in the events JSONL. They exist to separate three outcomes
+that a score alone collapses into one: the gate never fired, the gate fired and
+the model went and ran commands, and the gate fired and the model answered in
+prose from memory. The last is this mechanism's likeliest failure, and without
+the flag it is invisible.
+
+`requirementGateActedOn` compares tool executions against a snapshot taken as
+the gate fires. That reads as "afterwards" rather than "at some point" because
+the count never decreases and the gate fires from a response that called no
+tool, so nothing can move it in between.
+
+Every mechanism here was proved by reverting it and watching its test go red,
+because a regression test that has never failed proves nothing. Each revert was
+confirmed applied before the suite ran:
+
+| reverted | what went red |
+| --- | --- |
+| the clock guard on the verification gate | `WallBudgetExhaustedError` in place of the finished answer |
+| the pin | the task absent from every request after the sixth injection |
+| the requirement gate's `unattended` condition | seven tests, while the three negative ones stayed green |
+| the duplicate amnesty | two executions of the repeated check instead of three |
+| the composed status line | the merged notice naming only the verification gate |
+| the trial metadata keys | `KeyError` in the harness tests |
+
+The replay harness cannot speak to any of this. Its recordings hold one
+conversation turn each — the loop's injected messages were never written to the
+event log — so the pin never fires there and the baseline is unchanged by
+construction. What settles it is the benchmark: `overfull-hbox` and
+`video-processing` first, then the five-task job to check the three passing
+tasks did not regress.

harbor_woopcode/agent.py

@@ -894,4 +894,13 @@ def populate_context_post_run(self, context: AgentContext) -> None:
             # anything afterwards. Absent on a run from a CLI that predates the
             # summary, which is not the same as False.
             "woopcode_unverified_edits": summary.get("unverifiedEdits"),
+            # The requirement gate, in two parts, because they answer different
+            # questions of a job: how often a trial tried to stop early, and
+            # how often being asked actually sent it back to run something.
+            # A gate that fires and is answered in prose changes nothing, and
+            # in the score alone that is invisible.
+            "woopcode_requirement_reminders": summary.get("requirementReminders"),
+            "woopcode_requirement_gate_acted_on": summary.get(
+                "requirementGateActedOn"
+            ),
         }