feat(runtime): make the wall budget bind, in the tools and through Harbor

CLAUDE.md

@@ -107,9 +107,21 @@ PYTHONPATH=. harbor run -d terminal-bench/terminal-bench-2 \
 
 `-i` needs the **fully qualified** task name. A bare `overfull-hbox` fails at config validation — which is cheap, because it fails before any container or API call.
 
+A command line without `-c job.yaml` reads none of it, so both budgets fall back to `agent.py`'s defaults — `max_iterations` 200 and no wall budget. Add `--ak max_iterations=1000 --ak agent_timeout_sec=<the task's task.toml value>` to a single-task run, or it is capped at 200 steps and says nothing about the clock.
+
 Four things that are easy to get wrong:
 
-- **Wall clock is the binding budget, not iterations.** `job.yaml` sets `max_iterations: 200`, but Harbor enforces a per-task agent timeout from the task package and raises `AgentTimeoutError` — 750s for `overfull-hbox`. The baseline run used 191s of that for 59 iterations, so 200 iterations is only reachable if each averages under ~3.7s. Nothing in the loop knows about this budget; the iteration counter is the only one it can see.
+- **Both budgets bind, and at `max_iterations: 200` iterations bound first on every task measured.** Harbor enforces a per-task agent timeout from the task package's `task.toml` and raises `AgentTimeoutError`; the loop enforces `WOOPCODE_MAX_WALL_SEC` and stops on whichever binds first. Timeouts vary by 16× across five tasks, so a rule read off any one of them does not generalise — against `jobs/tb2-post-1.1`:
+
+  | task | timeout | wall used | unused | iterations | what stopped it |
+  | --- | --- | --- | --- | --- | --- |
+  | build-pov-ray | 12000s | 287s | 98% | 79 | model chose to |
+  | circuit-fibsqrt | 3600s | 400s | 89% | 165 | model chose to |
+  | make-mips-interpreter | 1800s | 406s | **77%** | **200** | **our ceiling** |
+  | overfull-hbox | 750s | 192s | 74% | 59 | model chose to |
+  | video-processing | 3600s | 261s | 93% | 83 | model chose to |
+
+  `make-mips-interpreter` was killed by its own 200th iteration at 406s of 1800, mid-work, with `exception_info: null` proving Harbor's timeout never fired — so `job.yaml` sets `max_iterations: 1000` and the ceiling reverts to guarding a pathological loop. Harbor does **not** hand the agent its timeout: `AgentContext` has no such field and `Trial` holds `timeout_sec`, so an operator supplies it as `agent_timeout_sec`, which `agent.py` forwards verbatim. `docs/adr/0001-wall-clock-budget-for-the-agent-loop.md` has the measurements and the rejected alternatives.
 - **`agent_timeout_sec: null` in a job's `lock.json` does not mean there is no timeout.** It means that run never hit one. The value only appears in `result.json`'s `exception_info` after it fires.
 - **Judge a change on the recorded `durationMs`, not on wall clock.** The loop stamps it around the provider request only, so it is the comparable number; total trial time includes container setup, tool execution and the verifier. Confusing the two once turned a 1.5s baseline into a reported 11s.
 - **Provider latency varies enormously and will masquerade as a regression.** The same request shape has measured 1,519ms median across 59 iterations on one day and ~63s on another, and within a single 15-request probe the same configuration ranged from 1,742ms to 90,002ms depending on position in the sequence. Before blaming a code change, check whether latency tracks position rather than the change, and whether the effect is anti-correlated with what you think causes it.
@@ -118,7 +130,7 @@ Reading a trajectory, `run_end`'s `ok: true` means the loop finished, not that t
 
 ## Environment variables
 
-`WOOPCODE_API_KEY`, `WOOPCODE_PROVIDER`, `WOOPCODE_MAX_ITERATIONS`, `WOOPCODE_MAX_ATTEMPTS` (retry), `WOOPCODE_TOOL_HISTORY_BUDGET`, `WOOPCODE_THINKING_BUDGET`, `WOOPCODE_NON_INTERACTIVE`, `WOOPCODE_DEMO_URL`. Bun loads `.env` automatically — no `dotenv`.
+`WOOPCODE_API_KEY`, `WOOPCODE_PROVIDER`, `WOOPCODE_MAX_ITERATIONS`, `WOOPCODE_MAX_WALL_SEC`, `WOOPCODE_MAX_ATTEMPTS` (retry), `WOOPCODE_TOOL_HISTORY_BUDGET`, `WOOPCODE_THINKING_BUDGET`, `WOOPCODE_NON_INTERACTIVE`, `WOOPCODE_DEMO_URL`. Bun loads `.env` automatically — no `dotenv`.
 
 Sandboxing: `E2B_API_KEY`, `WOOPCODE_SANDBOX_TEMPLATE`, `WOOPCODE_SANDBOX_TIMEOUT_MS`, `WOOPCODE_SANDBOX_MAX_FILE_BYTES`, `WOOPCODE_SANDBOX_NETWORK`, `WOOPCODE_SANDBOX_ENV`, `WOOPCODE_SANDBOX_SETUP`.
 

commands/agent.tsx

@@ -216,8 +216,7 @@ async function runHeadless(
     prompt,
   });
 
-  let failed = false;
-  let budgetExhausted = false;
+  const outcome = new HeadlessOutcome();
   let summary: TurnSummary | undefined;
 
   const callbacks: AgentCallbacks = {
@@ -296,8 +295,7 @@ async function runHeadless(
       process.stdout.write(text);
     },
     onError(error) {
-      failed = true;
-      if (error instanceof BudgetExhaustedError) budgetExhausted = true;
+      outcome.record(error);
       log.write({ type: "error", ts: now(), message: error.message });
       process.stderr.write(`✖ ${error.message}\n`);
     },
@@ -319,8 +317,7 @@ async function runHeadless(
   try {
     await controller.run(prompt);
   } catch (error) {
-    failed = true;
-    if (error instanceof BudgetExhaustedError) budgetExhausted = true;
+    outcome.record(error);
     const message = error instanceof Error ? error.message : String(error);
     log.write({ type: "error", ts: now(), message });
     process.stderr.write(`✖ ${message}\n`);
@@ -329,22 +326,61 @@ async function runHeadless(
     await controller.dispose();
   }
 
-  log.write({ type: "run_end", ts: now(), ok: !failed, summary });
+  log.write({ type: "run_end", ts: now(), ok: !outcome.failed, summary });
   process.stdout.write("\n");
-  // Exit codes are a contract with automated callers:
-  //   0 - the turn completed
-  //   2 - the loop ran out of budget, of iterations or of wall-clock time; work
-  //       may be partially done, and the caller should judge the result rather
-  //       than treat this as a crash. One code for both: a distinct one for the
-  //       deadline would be booked as an exception by any harness not yet
-  //       updated to know it, dropping those trials from the mean.
-  //   1 - anything else went wrong
-  process.exit(failed ? (budgetExhausted ? EXIT_BUDGET_EXHAUSTED : 1) : 0);
+  process.exit(outcome.exitCode());
 }
 
-/** See the exit-code contract in `runHeadless`. */
+/** See the exit-code contract in `HeadlessOutcome.exitCode`. */
 export const EXIT_BUDGET_EXHAUSTED = 2;
 
+/**
+ * How a headless run ended, accumulated across the two places it can fail.
+ *
+ * A type rather than the pair of booleans it replaces, because those were the
+ * same type and one of their four combinations — exhausted but not failed — has
+ * no meaning. `record` is the only way to set either, and it always sets
+ * `failed`, so that combination is now unconstructible rather than merely
+ * untested. `tools/timeoutBudget.ts` makes the same argument for `BudgetedTimeout`.
+ *
+ * It also gives the classification one home. Inline at both failure sites, the
+ * `instanceof` could only be checked by a test that copied it — which asserts
+ * against the copy, and goes on passing when the original changes.
+ */
+export class HeadlessOutcome {
+  /** Whether anything went wrong at all. Drives `run_end`'s `ok`. */
+  failed = false;
+
+  /** Whether what went wrong was a spent budget rather than a fault. */
+  budgetExhausted = false;
+
+  /** Records a failure. The line `runHeadless` runs at both of its failure sites. */
+  record(error: unknown): void {
+    this.failed = true;
+    if (error instanceof BudgetExhaustedError) this.budgetExhausted = true;
+  }
+
+  /**
+   * The exit code this run reports.
+   *
+   * Exit codes are a contract with automated callers:
+   *   0 - the turn completed
+   *   2 - the loop ran out of budget, of iterations or of wall-clock time; work
+   *       may be partially done, and the caller should judge the result rather
+   *       than treat this as a crash. One code for both: a distinct one for the
+   *       deadline would be booked as an exception by any harness not yet
+   *       updated to know it, dropping those trials from the mean.
+   *   1 - anything else went wrong
+   *
+   * Reachable from a test, unlike the expression it replaces — that sat inline
+   * in a `process.exit` beside a live provider and a real session, and the
+   * contract has a second party: `harbor_woopcode/agent.py` maps 2 to success.
+   */
+  exitCode(): number {
+    return this.failed ? (this.budgetExhausted ? EXIT_BUDGET_EXHAUSTED : 1) : 0;
+  }
+}
+
 /** Runs the interactive TUI agent. */
 async function runInteractive(
   modelOverride?: string,

docs/adr/0001-wall-clock-budget-for-the-agent-loop.md

@@ -69,6 +69,18 @@ fraction as a 12000s one.
 is advisory: `run_terminal` defaults to 300s and the model may ask for more, so
 one command started just inside the budget outlives it by minutes — on
 `overfull-hbox`, a single default-timeout call is 40% of the entire budget.
+`run_terminal`, `run_tests` and `repl` clamp; `process_start` deliberately does
+not, because a background process does not hold the loop and so cannot overshoot
+the deadline. The clamp is read after approval rather than at the top of
+`execute`, since the clock runs while a human decides.
+
+**A clamped kill is explained by the clock, not by the timeout.** The standing
+advice for a timeout is to run it again with a larger one, which is exactly
+wrong when the budget rather than the number ended the call — the model would
+spend its last seconds reaching the same end. Rejected: leaving the existing
+messages and relying on the wind-down warning to have set the context, which
+puts two paragraphs an unknown number of tool calls apart and asks the model to
+connect them.
 
 **The deadline lives in module state (`runtime/deadline.ts`), not on
 `Tool.execute`.** `runtime/sandbox/registry.ts` argues this case in its own
@@ -95,6 +107,27 @@ one message and one flag. A constant expressed in seconds was rejected as the
 wrong shape across this task set — 120s is 16% of `overfull-hbox`'s budget and
 1% of `build-pov-ray`'s.
 
+The rate that conversion runs on is measured on the turn itself, so it is
+unreliable exactly when there is least of it. `meanStepMs` after one step *is*
+that step, and provider latency has measured 1,742ms to 90,002ms inside a single
+probe — so one slow opening request made a turn with 690s of budget read as five
+steps from the end, and a latched flag would have left the model winding down
+for the rest of it. That is this document's own failure reached from the other
+side, so the rate is ignored until `MIN_RATE_SAMPLES` steps have gone into it
+(the mean recovers by the fourth), and the flag re-arms if the estimate comes
+back above twice the threshold.
+
+## What it costs the prompt
+
+`bun run replay:baseline` over the ten fixtures in
+`packages/tests/fixtures/replay`, before and after: **byte-identical**, peak
+prompt characters unchanged on every fixture (mean 126,563, max 219,570).
+
+Expected, and worth stating rather than assuming. Nothing here rewrites
+history — the wind-down adds at most one short user message to a turn, and only
+to turns that reach it, which no fixture does. The harness measures characters
+and cannot speak to cache rates; it says so itself.
+
 ## Consequences
 
 - `WOOPCODE_MAX_WALL_SEC` and the exit-code behaviour become a contract with

harbor_woopcode/README.md

@@ -120,9 +120,21 @@ Pass with `--ak key=value`, or under `kwargs:` in `job.yaml`.
 | --- | --- | --- |
 | `source_dir` | — | Install from a local checkout instead of npm. Required until the CLI changes above are published |
 | `version` | `latest` | npm version to install. Pin for reproducible numbers |
-| `max_iterations` | `200` | Loop budget per task |
+| `max_iterations` | `200` | Loop budget per task, in steps |
+| `agent_timeout_sec` | — | Loop budget per task, in wall seconds, forwarded verbatim as `WOOPCODE_MAX_WALL_SEC`. Unset leaves the loop bounded by iterations alone |
 | `auto_approve` | `True` | Must stay on; there is no human to approve edits |
 
+Harbor does not hand the agent its own timeout — `AgentContext` has no such
+field, and `task.config.agent.timeout_sec` is held by `Trial`
+(`harbor/trial/trial.py:_compute_agent_timeout_sec`). So `agent_timeout_sec` has
+to be supplied by whoever starts the run, which is what Harbor's own Cline agent
+does under this same name. Read the number from the task package's `task.toml`
+and pass it whole; the reserve is subtracted inside the loop
+(`runtime/deadline.ts`), so a published number traces back to `task.toml` with
+no arithmetic in between. Because the kwarg applies to every task in a job,
+`job.yaml` carries the shortest timeout in its five-task set;
+`docs/adr/0001-wall-clock-budget-for-the-agent-loop.md` has the per-task table.
+
 ## Running
 
 ```bash
@@ -136,6 +148,22 @@ export GEMINI_API_KEY=...
 PYTHONPATH=. harbor run -d terminal-bench/terminal-bench-2 -a harbor_woopcode:WoopCode -m google/gemini-3.5-flash-lite --ak source_dir=$(pwd) -l 5 --agent-setup-timeout-multiplier 2 --max-retries 2 --retry-include ApiRateLimitError
 ```
 
+A command line without `-c job.yaml` reads **none** of that file, so both
+budgets fall back to the defaults in `agent.py` — `max_iterations` 200, and no
+wall budget at all. Pass them when running a single task, or the run is capped
+at 200 steps and cannot demonstrate anything about the clock:
+
+```bash
+# One task, with its own task.toml budget
+PYTHONPATH=. harbor run -d terminal-bench/terminal-bench-2 \
+  -a harbor_woopcode:WoopCode -m google/gemini-3.5-flash-lite \
+  --ak source_dir=$(pwd) --ak max_iterations=1000 --ak agent_timeout_sec=1800 \
+  -i terminal-bench/make-mips-interpreter -n 1
+```
+
+`-i` needs the fully qualified task name; a bare `make-mips-interpreter` fails
+at config validation, before any container or API call.
+
 ### Two flags worth knowing about
 
 **`--agent-setup-timeout-multiplier 2`.** Harbor allows 360s for agent setup.

harbor_woopcode/agent.py

@@ -103,6 +103,44 @@
 _ALWAYS_FORWARDED = ["WOOPCODE_API_KEY", "WOOPCODE_PROVIDER", "GEMINI_API_KEY"]
 
 
+def _whole_seconds(value: int | float | str | None) -> int | None:
+    """Coerce a wall-clock budget to whole seconds, or reject it loudly.
+
+    Three types reach here. ``job.yaml`` gives an int; ``--ak key=value`` runs
+    the value through ``json.loads``, so ``=1800`` is an int, ``=1800.0`` a
+    float, and anything JSON cannot read stays a str. Harbor's own Cline agent
+    accepts the same three for the same reason.
+
+    Raising beats forwarding a bad value: the CLI ignores a
+    ``WOOPCODE_MAX_WALL_SEC`` it cannot parse and runs unbudgeted, warning on a
+    stderr stream that is inside the container and buried in the trial log. The
+    operator would then read a run that ignored their budget as evidence about
+    that budget. A ``ValueError`` here fails at config validation instead --
+    before any container starts or any token is spent.
+    """
+    if value is None:
+        return None
+
+    try:
+        # OverflowError as well as the obvious two: `json.loads` reads `1e400`
+        # and `Infinity` as `inf`, and `int(inf)` raises neither TypeError nor
+        # ValueError -- so without it those two are the one bad input that
+        # escapes as a traceback instead of the message below.
+        seconds = int(float(value))
+    except (TypeError, ValueError, OverflowError):
+        raise ValueError(
+            f"Invalid value for 'agent_timeout_sec': {value!r}. "
+            "Expected the task's wall-clock budget in seconds."
+        ) from None
+
+    if seconds < 1:
+        raise ValueError(
+            f"Invalid value for 'agent_timeout_sec': {value!r}. Must be >= 1. "
+            "Omit it to leave the loop bounded by iterations alone."
+        )
+    return seconds
+
+
 class WoopCode(BaseInstalledAgent):
     """Runs the ``woopcode`` CLI as a Harbor agent.
 
@@ -116,6 +154,17 @@ class WoopCode(BaseInstalledAgent):
             turning this off makes almost every task fail by construction.
         max_iterations: loop budget for a single task (default
             ``_DEFAULT_MAX_ITERATIONS``).
+        agent_timeout_sec: wall-clock budget for a single task, in seconds, as
+            enforced by Harbor. Forwarded verbatim -- the loop subtracts its own
+            reserve. Omitted by default, which leaves the loop bounded by
+            iterations alone, as it was before this existed.
+
+            Harbor does not hand this to the agent: ``AgentContext`` has no such
+            field, and the per-task ``timeout_sec`` is held by ``Trial``
+            (``harbor/trial/trial.py:_compute_agent_timeout_sec``). It has to be
+            supplied by the operator, which is what Harbor's own Cline agent
+            does under this same name -- so pass the number from the task
+            package's ``task.toml``.
     """
 
     # The CLI emits a structured event log that this class converts to ATIF.
@@ -184,12 +233,14 @@ def __init__(
         source_dir: str | None = None,
         auto_approve: bool = True,
         max_iterations: int | None = None,
+        agent_timeout_sec: int | float | str | None = None,
         **kwargs: Any,
     ) -> None:
         super().__init__(*args, **kwargs)
         self._source_dir = source_dir
         self._auto_approve = auto_approve
         self._max_iterations = max_iterations or _DEFAULT_MAX_ITERATIONS
+        self._agent_timeout_sec = _whole_seconds(agent_timeout_sec)
         # Captured in run() so the trajectory can open with the user turn; the
         # event log records the prompt too, but run() has the rendered form
         # after any prompt template has been applied.
@@ -504,6 +555,17 @@ def _build_env(self) -> dict[str, str]:
         # _DEFAULT_MAX_ITERATIONS.
         env["WOOPCODE_MAX_ITERATIONS"] = str(self._max_iterations)
 
+        # The second budget. Harbor kills the process at this number, so the
+        # loop is told it and winds down first -- one trial was otherwise
+        # stopped by its own 200th iteration at 406s of 1800, mid-work.
+        #
+        # Verbatim, and only when the operator gave one: the reserve is
+        # subtracted inside the loop (`runtime/deadline.ts`), and an absent
+        # variable there means unbudgeted, which is the behaviour every run had
+        # before this. Sending `0` instead would arm a deadline already spent.
+        if self._agent_timeout_sec is not None:
+            env["WOOPCODE_MAX_WALL_SEC"] = str(self._agent_timeout_sec)
+
         return env
 
     @override