Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions benchmark/LHTB/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,13 @@ accepts `resume-if-available`. Turn also requires `LOOPX_VALIDATION_COMMAND_JSON
an argv array for the independently protected task validator. No generic
benchmark scoring or hidden-verifier feedback is introduced.

`LOOPX_TASK_ENTRY=seeded-todo` preserves the generic phase Todo default.
`LOOPX_TASK_ENTRY=loopx-planned` invokes the product planning checkpoint before
heartbeat, Turn or LoopX Goal execution. `LOOPX_PLANNING_TIMEOUT_SEC` defaults
to 300 and consumes the existing phase budget. Both entry policies preserve
existing waits when new phases arrive. See the shared runtime for session and
planning-readback semantics.

Model and effort defaults remain unchanged but may be selected explicitly.
`run.sh prepare` performs networking/Harbor preparation. `preflight` now checks
existing preparation without patching Harbor or creating a network. Smoke/full
Expand Down
6 changes: 5 additions & 1 deletion benchmark/LHTB/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ LHTB_MODELONLY_GATEWAY="${LHTB_MODELONLY_GATEWAY:-192.0.2.1}"
LOOPX_SRC_DIR="${LOOPX_SRC_DIR:-$LOOPX_ROOT}"
export PYTHONPATH="$LOOPX_SRC_DIR${PYTHONPATH:+:$PYTHONPATH}"
LOOPX_EXECUTION_MODE="${LOOPX_EXECUTION_MODE:-heartbeat}"
LOOPX_TASK_ENTRY="${LOOPX_TASK_ENTRY:-seeded-todo}"
LOOPX_PLANNING_TIMEOUT_SEC="${LOOPX_PLANNING_TIMEOUT_SEC:-300}"
LOOPX_ITERATION_CONTEXT="${LOOPX_ITERATION_CONTEXT:-fresh}"
LOOPX_VALIDATION_COMMAND_JSON="${LOOPX_VALIDATION_COMMAND_JSON:-[]}"
SHARED_CODEX_AGENT_DIR="$LOOPX_SRC_DIR/benchmark/runtime"
Expand Down Expand Up @@ -115,7 +117,7 @@ if [[ "$MODE" == smoke ]]; then
expected_task_count=1
job_suffix="smoke-${SMOKE_TASK}"
fi
job_name="lhtb-${LOOPX_EXECUTION_MODE}-${LOOPX_ITERATION_CONTEXT}-${job_suffix}-${run_stamp}"
job_name="lhtb-${LOOPX_EXECUTION_MODE}-${LOOPX_TASK_ENTRY}-${LOOPX_ITERATION_CONTEXT}-${job_suffix}-${run_stamp}"
generated_config="$CODE_DIR/.generated/${job_name}.yaml"
jobs_dir="$CODE_DIR/runs"

Expand All @@ -129,6 +131,8 @@ jobs_dir="$CODE_DIR/runs"
--effort "$REASONING_EFFORT" \
--timeout "$AGENT_TIMEOUT_SEC" \
--execution-mode "$LOOPX_EXECUTION_MODE" \
--task-entry "$LOOPX_TASK_ENTRY" \
--planning-timeout "$LOOPX_PLANNING_TIMEOUT_SEC" \
--iteration-context "$LOOPX_ITERATION_CONTEXT" \
--validation-command-json "$LOOPX_VALIDATION_COMMAND_JSON" \
--turn-timeout "$LOOPX_CODEX_TURN_TIMEOUT_SEC" \
Expand Down
1 change: 1 addition & 0 deletions benchmark/LHTB/scripts/preflight.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ def check(label: str, passed: bool, detail: str) -> None:
)
execution = Execution(
mode=kwargs.get("execution_mode", "heartbeat"),
task_entry=kwargs.get("task_entry", "seeded-todo"),
context=kwargs.get("iteration_context", "fresh"),
validation_command=kwargs.get("validation_command", []),
)
Expand Down
7 changes: 6 additions & 1 deletion benchmark/LHTB/scripts/render_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
from pathlib import Path

import yaml
from benchmark.runtime.codex import CONTEXTS, MODES, Execution
from benchmark.runtime.codex import CONTEXTS, MODES, TASK_ENTRIES, Execution


def main() -> int:
Expand All @@ -25,6 +25,8 @@ def main() -> int:
parser.add_argument("--task", action="append", default=[])
parser.add_argument("--execution-mode", choices=MODES, default="heartbeat")
parser.add_argument("--iteration-context", choices=CONTEXTS, default="fresh")
parser.add_argument("--task-entry", choices=TASK_ENTRIES, default="seeded-todo")
parser.add_argument("--planning-timeout", type=float, default=300)
parser.add_argument("--validation-command-json", default="[]")
parser.add_argument("--turn-timeout", type=float, default=4700)
parser.add_argument("--scheduler-timeout", type=int, default=5080)
Expand Down Expand Up @@ -59,13 +61,16 @@ def main() -> int:
context=args.iteration_context,
timeout_seconds=args.turn_timeout,
validation_command=json.loads(args.validation_command_json),
task_entry=args.task_entry,
)
agent["kwargs"].update(
execution_mode=execution.mode,
iteration_context=execution.context,
validation_command=list(execution.validation_command),
turn_timeout_sec=execution.timeout_seconds,
scheduler_timeout_sec=args.scheduler_timeout,
task_entry=execution.task_entry,
planning_timeout_sec=args.planning_timeout,
)
agent["kwargs"]["goals"] = str(execution.native_goal).lower()
agent["kwargs"]["web_search"] = "disabled"
Expand Down
55 changes: 54 additions & 1 deletion benchmark/runtime/RUNTIME.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ agents:
override_timeout_sec: 5400
kwargs:
execution_mode: heartbeat
task_entry: seeded-todo
iteration_context: fresh
reasoning_effort: max
codex_sandbox: danger-full-access
Expand Down Expand Up @@ -46,6 +47,58 @@ The runner supplies no HEAD-moved/clean-worktree/exit-only substitute and never
calls hidden benchmark verification to provide intermediate feedback. Independent
validator protection remains the environment owner's responsibility.

## Task entry and planning ablation

`task_entry` is independent of the execution mode:

- `seeded-todo` (the compatibility default) writes a generic execution Todo.
Follow-up phases update that Todo while it remains live and owned by this
agent; completed or deferred work gets a new Todo. Updates preserve blocked
state. The agent can still plan and replan during execution.
- `loopx-planned` runs the installed `$loopx` skill against the public
`loopx todo plan` checkpoint before execution. The checkpoint shares the
product's planner and continuation-aware Todo delta; it creates no planning
Todo and starts no host loop. Select it only for heartbeat, Turn or LoopX Goal.

The model writes or reuses actual task Todos through the public CLI. The worker
reads the product packet again and checks the input digest, identity, Todo ids
and runnable/blocked state. A fabricated id, changed input, wrong owner, failed
planning process or missing result fails the entry; it never falls back to a
generic Todo. A blocked entry retains the referenced blockers and starts no
execution driver. Readback proves state and ownership, not semantic plan quality.

Planning uses a separate fresh `codex exec` session with native Goals disabled
for that call. Its session is not inserted into core Turn session bindings or
resumed by the subsequent execution. This is a planning-contract ablation, not
an exact reproduction of same-conversation interactive `$loopx` startup.
The default `planning_timeout_sec` is 300; planning and preparation consume the
same `scheduler_timeout_sec` phase budget as execution. Planning sessions are
included in native session/token aggregation. No planning checkpoint is counted
as a completed advancement Todo or settled work Turn.

Each phase keeps an immutable task document. New phases preserve Goal/Agent
identity and expose existing Todos to the planner; they do not clear waiting
state or force the agent active. An unresolved Turn must be recovered before
another phase can replace its task input. These wait/recovery rules apply to
both entry policies; they correct the earlier unconditional phase reset.
Every scheduler wake caps its host timeout against the remaining phase budget
before opening an execution. If only startup and settlement reserve remains,
it records a budget-exhausted no-op without creating a pending Turn.
The deadline uses the task environment's clock, including remote Harbor backends.

To compare entry policies, hold the execution mode, session policy, model,
effort, tools, feedback and total budget fixed, and use separate trials:

```yaml
kwargs:
execution_mode: heartbeat
task_entry: loopx-planned
planning_timeout_sec: 300
iteration_context: fresh
turn_timeout_sec: 4700
scheduler_timeout_sec: 5080
```

## Install and isolate

From the candidate worktree, provide:
Expand All @@ -58,7 +111,7 @@ export PYTHONPATH="$PWD${PYTHONPATH:+:$PYTHONPATH}"

Also set `CODEX_OFFLINE_DIR` (Codex, code-mode sidecar, rg),
`LOOPX_PORTABLE_PYTHON` (Python >=3.11 distribution) and `LOOPX_NODE_DIR`
(Node >=22.18.0 distribution). Staging uses `git archive HEAD`, never local run
(Node >=22.18.0 distribution). Staging archives the verified commit SHA, never local run
artifacts. The host import must come from that checkout, whose tracked files
must match HEAD. Commit the candidate before real validation. Baselines stage only
the runner/native transport, without installing LoopX skills or initializing
Expand Down
6 changes: 6 additions & 0 deletions benchmark/runtime/codex.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@

MODES = ("plain", "native-goal", "heartbeat", "turn", "loopx-goal")
CONTEXTS = ("fresh", "resume-if-available")
TASK_ENTRIES = ("seeded-todo", "loopx-planned")
SANDBOXES = ("read-only", "workspace-write", "danger-full-access")


Expand All @@ -21,10 +22,15 @@ class Execution:
sandbox: str = "danger-full-access"
timeout_seconds: float = 4700
validation_command: tuple[str, ...] = ()
task_entry: str = "seeded-todo"

def __post_init__(self) -> None:
if self.mode not in MODES or self.context not in CONTEXTS:
raise ValueError("unsupported execution mode or iteration context")
if self.task_entry not in TASK_ENTRIES:
raise ValueError("unsupported task entry")
if self.task_entry == "loopx-planned" and not self.uses_loopx:
raise ValueError("loopx-planned requires a LoopX execution mode")
if self.context != "fresh" and self.mode != "turn":
raise ValueError("resume-if-available currently requires mode=turn")
if self.sandbox not in SANDBOXES:
Expand Down
Loading
Loading