fix(agents): run codex + antigravity harnesses on the tempdir/host path#33
Conversation
The pinned 0.1.5 localharness binary requires the GLIBC_ABI_DT_RELR ABI (glibc 2.36+), so it fails to load on the eval agents' Ubuntu 22.04 (glibc 2.35), taking down the antigravity tempdir/host path (the activation suite and container-less skills tasks) while docker-driver tasks stay fine. 0.1.7 links without DT_RELR (max required symbol GLIBC_2.26) and loads on 2.35+. The manylinux_2_17 wheel tag alone does not guarantee this, so the pin comment documents verifying ELF load before any future bump. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codex's read-only / workspace-write sandboxes shell out to an OS-level
helper (Landlock, or a bwrap re-exec) that fails to initialize on managed CI
hosts ("bwrap: execvp .../codex: Permission denied" where unprivileged user
namespaces or exec mounts are restricted); its writes/execs then fail
silently and the task produces no artifacts. The docker path already avoids
this via CODER_EVAL_IN_CONTAINER. Thread a `sandbox_managed` signal through
Agent.start() so the tempdir driver — where coder_eval's ephemeral per-task
dir is the isolation boundary — also falls back to full-access, matching
Claude Code (no in-agent OS sandbox on that path). Hard isolation of
untrusted actions remains the docker driver's job.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The orchestrator now threads `sandbox_managed` into `Agent.start()`, and the codebase's convention is that agent test doubles mirror the ABC signature exactly. Add the keyword-only param (accepted and ignored, like `env_path_prepend`) to every double that declares `start()`, so the orchestrator-routed doubles (MockAgent and the inline DummyAgents) stop raising `unexpected keyword argument 'sandbox_managed'`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…wcw-mr6r pyasn1 0.6.3 (transitive via google-auth → pyasn1-modules) carries two High advisories fixed in 0.6.4. Pre-existing on main; bumping the lock clears both osv-scanner and pip-audit. No suppression entry is added because a fix is available (osv-scanner.toml is only for unfixable advisories). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…4.4) - claude-agent-sdk 0.2.82 -> 0.2.124 - openai-codex 0.1.0b3 -> 0.144.4, and its pinned cli-bin 0.137.0a4 -> 0.144.4 (the SDK version now tracks the codex CLI line; the 0.144.x jump is a re-numbering, not an API rewrite). The public API surface codex_agent.py depends on — Codex/Thread/turn, the notification method strings, item types, and token-usage fields — stayed source-compatible, so no agent code changed. - google-antigravity already at 0.1.7 (latest; pinned by the tempdir fix). Full suite incl. the live codex integration tests is green on all three. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dispatching release.yml from a branch (github.ref != main) now runs a prerelease instead of a real release: stamp a throwaway <next-patch>rcN version into pyproject/__init__, refresh uv.lock, build, publish the wheel to public PyPI (OIDC) and a :<version> GHCR agent image. It does NOT move :latest, tag, commit, or push to main. This lets a branch be dry-run on the ADO nightly infra (pinned via coderEvalVersion) before merge. The main path is unchanged; the bump input is ignored off main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Claude finished @bai-uipath's task in 2m 8s —— View job PR Review in Progress
|
coder_eval always owns a run's isolation boundary — a docker container (docker driver) or an ephemeral per-task tempdir it creates and discards (tempdir driver); those are the only two drivers — so Codex's own in-process OS sandbox (Landlock/seatbelt) is always redundant. Worse, it breaks on the paths we rely on: Landlock is unavailable in the container, the bwrap re-exec is denied on constrained CI agents, and Windows has no OS sandbox at all — in each case a read-only/workspace-write run fails its writes/execs silently and scores 0. This is what left the Windows uipath-rpa tasks unable to run `uip` (exit -1) even after the earlier container/tempdir fallback landed. Run full-access unconditionally (approval_mode stays deny_all), matching Claude Code and Antigravity, which run with no in-agent OS sandbox; hard isolation of untrusted actions is the docker driver's job. Removes the now-obsolete sandbox_managed plumbing and the per-mode sandbox mapping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
uipreliga
left a comment
There was a problem hiding this comment.
Review: coder_eval — pr:33
Scope: pr:33 · branch bai/codex-antigravity-tempdir · 6b43af5 · 2026-07-22T00:34Z · workflow variant
Change class: complex — removes Codex's in-process OS sandbox so it always runs full-access for every permission mode; a security-sensitive control-flow change, plus CI prerelease-mode logic and SDK/CLI-binary version bumps
coder_eval is in strong health (9.5/10) with exemplary type safety, architecture, and harness quality, but a single recent PR — hardcoding the Codex sandbox to full-access (codex_agent.py:1268) — drives every real risk: it strips the tempdir/host driver's only OS-level write confinement (a defense-in-depth regression on the default path) and silently turns the accepted-and-validated permission_mode knob into a no-op with stale security messaging still describing the removed mapping, so the bottom line is that the codebase is production-grade but this concentrated Codex-sandbox change needs a deliberate confinement/opt-in decision plus doc and messaging cleanup before it stops misleading operators.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 9 / 10 | 0 | 1 | 0 | 0 | Unreachable dead branch: workspace-write network-access opener can never fire after sandbox is hardcoded to full-access |
| 2. Type Safety | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 3. Test Health | 9.9 / 10 | 0 | 0 | 0 | 1 | TestSandboxAlwaysFullAccess os_name/in_container dimensions exercise no code path (duplicate cases, mild false-confidence) |
| 4. Security | 9 / 10 | 0 | 1 | 0 | 0 | Hardcoding Codex sandbox to full-access removes the tempdir/host driver's only OS-level write confinement (security + reproducibility impact) |
| 5. Architecture & Design | 10 / 10 | 0 | 0 | 0 | 0 | — |
| 6. Error Handling & Resilience | 9.9 / 10 | 0 | 0 | 0 | 1 | Prerelease can half-publish: PyPI wheel ships even when the sole : GHCR image build fails (continue-on-error), with the workflow still reporting success |
| 7. API Surface & Maintainability | 8.4 / 10 | 0 | 1 | 1 | 1 | Codex permission_mode is now a silently-inert no-op — plan/acceptEdits/default/bypassPermissions all resolve to full-access with no warn/reject |
| 8. Evaluation Harness Quality | 10 / 10 | 0 | 0 | 0 | 0 | — |
Overall Score: 9.5 / 10 · Weakest Axis: API Surface & Maintainability at 8.4 / 10
Totals: 🔴 0 · 🟠 3 · 🟡 1 · 🔵 3 across 8 axes.
Blockers
- [Axis 1] Unreachable dead branch: workspace-write network-access opener can never fire after sandbox is hardcoded to full-access (
src/coder_eval/agents/codex_agent.py:1289) — Line 1268 now hardcodessandbox_mode_str = "full-access", so the guard at line 1289if sandbox_mode_str == "workspace-write":is statically always False and line 1290tool_config["sandbox_workspace_write"] = {"network_access": True}is unreachable dead code (coverage confirms line 1290 is unreached). The whole 5-line comment block at lines 1284-1288 ("Codex's workspace-write sandbox disables network by default... Always open network in workspace-write") describes behavior that can never execute, and reads as load-bearing to a maintainer. This is the residue the PR left behind when it deleted_PERMISSION_MODE_TO_SANDBOX. Fix: delete lines 1284-1290 entirely. Sincesandbox_mode_stris now a single-use constant, also collapse it — assignoptions["sandbox"] = Sandbox.full_accessdirectly (matching the test'sSandbox.full_accessassertion) rather than routing a magic string throughSandbox(sandbox_mode_str)at line 1272. This finding matches the Axis-1 Critical anchor's every clause (dead code on a live path, looks load-bearing, misleads); it is filed High because it has zero runtime effect and the deciding= "full-access"constant sits 21 lines above the guard, making the deadness discoverable at a glance — a reviewer could defensibly rate it Critical. - [Axis 4] Hardcoding Codex sandbox to full-access removes the tempdir/host driver's only OS-level write confinement (security + reproducibility impact) (
src/coder_eval/agents/codex_agent.py:1268) —_build_thread_optionswas changed tosandbox_mode_str = "full-access"unconditionally (line 1268), deleting the_PERMISSION_MODE_TO_SANDBOXtable and theif sandbox_mode_str != "full-access" and os.getenv("CODER_EVAL_IN_CONTAINER")guard. That guard fired ONLY inside a docker container; on the defaultdriver: tempdirhost path (SandboxConfig.driver default="tempdir", models/sandbox.py:309-312)CODER_EVAL_IN_CONTAINERis unset, so pre-PR Codex ran with its Landlock/seatbelt sandbox —plan→read-only,default/acceptEdits(the config default, agent_config.py:139-141)→workspace-write — which confined agent writes/execs to the workspace dir. Post-PR every mode isfull-access: an untrusted agent action (adversarial model, or prompt injection carried in task-repo/prompt content processed unattended by nightly/host runs) can now write, delete, or exec anywhere the eval user can on the host (~/.ssh, ~/.aws, ~/.bashrc, arbitrary rm -rf), where Landlock previously blocked it. The docker path (docker_runner.py:1098 sets CODER_EVAL_IN_CONTAINER=1) was already full-access and is unaffected — so this is a defense-in-depth REGRESSION scoped to the default host driver, not a change to the isolated production path. The code's own comment concedeshard isolation of untrusted actions is the docker driver's job. Recommendation (open question on exact mechanism): gate host/tempdir Codex behind an explicit operator opt-in or fail loud, and note that with full-access now on, the loudly-warned unenforceddisallowed_tools(codex_agent.py:1302-1305) leaves NO backstop guardrail on the host path. Base score 8.0 = High. CVSS:3.1/AV:L/AC:L/PR:N/UI:N/S:U/C:L/I:H/A:H - [Axis 7] Codex permission_mode is now a silently-inert no-op — plan/acceptEdits/default/bypassPermissions all resolve to full-access with no warn/reject (
src/coder_eval/agents/codex_agent.py:1268) —_build_thread_optionsnow hardcodessandbox_mode_str = "full-access"(codex_agent.py:1268) for every permission mode;permission_mode(read at :1253) no longer influences the sandbox. Previouslyplan→read-only andacceptEdits/default→workspace-write, so on the tempdir/host driver a Codex run set topermission_mode: planwas confined; now the identical config yields a fully-writable agent, silently. The field is still accepted and validated (PermissionMode enum, default ACCEPT_EDITS at models/agent_config.py:139) but is inert for Codex, and the SDK gets no signal that the user asked for confinement. docs/TASK_DEFINITION_GUIDE.md:120-124 still documentplanas "Agent proposes changes, waits for approval" with no note that these are inert for Codex. Either emit a one-time warning when a Codex config sets a non-bypassPermissionspermission_mode (that it does not confine — matching the existing loud-on-surprise style elsewhere in this file), or document permission_mode as Claude-only for confinement and update the guide's Permission Modes / Agent Types sections. The design rationale (Codex's in-process Landlock/seatbelt sandbox is redundant given coder_eval's docker/tempdir boundary, and breaks silently on CI/Windows) is sound — the gap is that the user-facing knob stays accepted with zero effect and no warning.
Non-blocking, but please consider before merge
- [Axis 7] Stale messaging referencing the removed permission_mode->sandbox mapping (docstring/debug log at 1279 and security warning at 1355) (
src/coder_eval/agents/codex_agent.py:1355) —_log_config_enforcementstill special-cases only one mode:if self.config.permission_mode.value == "bypassPermissions":(codex_agent.py:1355) emits"[SECURITY] bypassPermissions grants unrestricted sandbox access (full-access). Only use in fully isolated environments...". After this changedefault,acceptEdits(the default), andplanALL resolve to full-access too, but emit no such warning — so the warning now implies the other modes are more confined than bypassPermissions when the sandbox posture is identical across all four. An operator running the defaultacceptEditswith Codex gets full-access with no security notice. Either drop the per-mode conditional (all modes are full-access now, so the notice should fire unconditionally for Codex) or reword it to reflect that Codex always runs full-access regardless of permission_mode. Also note the debug line at :1279 (Permission mode {permission_mode} → sandbox=...) now always prints full-access.
Nits
- [Axis 3] TestSandboxAlwaysFullAccess os_name/in_container dimensions exercise no code path (duplicate cases, mild false-confidence) (
tests/test_codex_agent.py:107) — The matrix at lines 107-109 parametrizesos_name∈{posix,nt} andin_container∈{T,F}, and line 119 doesmonkeypatch.setattr(_os, "name", os_name). But_build_thread_options(the SUT, called at line 121) reads neitheros.namenorCODER_EVAL_IN_CONTAINERafter this PR — it hardcodessandbox_mode_str = "full-access"(codex_agent.py:1268), and the onlyos.nameread (_login_shell_profiles_supported, codex_agent.py:1098) is never on this call path. So the 16 cases reduce to 4 effective ones (one per permission mode); the 8os_name="nt"cases assert identically to theposixcases and give the appearance of Windows coverage that isn't exercised. This is a defensible regression guard against reintroducing an os.name/container branch, so keep it if intentional — but add a one-line comment stating the assertion is invariance-by-construction, or drop the two inert dimensions to keep the matrix honest about what it covers. - [Axis 6] Prerelease can half-publish: PyPI wheel ships even when the sole : GHCR image build fails (continue-on-error), with the workflow still reporting success (
.github/workflows/release.yml:264) — In prerelease mode the:<version>GHCR image is the only image artifact (a real release additionally gets:latestfrom docker-publish.yml, a prerelease does not — see theCompute image tagsstep at lines 231-247), yetBuild and push versioned agent image(lines 263-264) carriescontinue-on-error: true # GHCR image is internal/best-effort; don't block PyPI, and thepublish-pypijob gates only onif: needs.release.outputs.version != ''(line 288). So if the image build fails, the wheel still publishes to public PyPI and the release job reports success, leaving the stamped rc version half-published — a PyPI wheel with no matching:<version>image, which is exactly the artifact the prerelease's stated purpose (ADO nightly dry-run pinned viacoderEvalVersion) depends on. The failure only surfaces downstream in the nightlydocker pull, not in this run. Impact is low (rc is throwaway, the image step still shows a red X in the Actions log, and a re-dispatch mints a fresh rc via the run number), but consider making the image step blocking (or at least gating the prerelease image on success) whenIS_PRERELEASE == 'true', since for a prerelease the image is not merely 'internal/best-effort'. - [Axis 7] release.yml
bumpinput does not disclose it is ignored on non-main (prerelease) dispatch (.github/workflows/release.yml:28) — Thebumpchoice input (release.yml:27-34, description'Version bump to release') is silently ignored when the workflow is dispatched from a non-main branch: the prerelease path always stamps<next-patch>rc<run#>(Determine release mode step) regardless of whether the dispatcher selectedminor/major. The header comment at line 22 documents "Thebumpinput is ignored off main," but that comment is not visible in the Actions dispatch UI — the inputdescriptionis the only text a dispatcher sees. Append the caveat to the input description, e.g.'Version bump to release (ignored on non-main / prerelease dispatch, which always stamps a next-patch rc)', so the branch-ref-keyed behavior is discoverable at the point of use.
What's Missing
Daily/nightly:
- 🟠 Unstated nightly blast radius of forcing Codex to full-access: the PR removes OS-level write confinement on the default
tempdirdriver but never states what the nightly/cron pipeline runs Codex under. If any nightly Codex job uses the defaulttempdirdriver (not docker), it now executes unconfined on the host — a production-path behavior change the PR body should call out and confirm the nightly is docker-only. The change explicitly exists to enable an ADO nightly dry-run (release.yml prerelease comment), so the nightly interaction is squarely in scope yet unaddressed. (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 4: Hardcoding Codex sandbox to full-access removes the tempdir/host driver's OS-level write confinement) - 🟡 Container-rebuild-in-lockstep not stated for the codex-cli-bin 0.144.4 bump + glibc floor: the pyproject note documents that the bundled
localharness/codex ELF must load on Ubuntu 22.04 (glibc 2.35) but the PR never confirms the nightlydocker/Dockerfileimage is rebuilt with the new pin, nor that the actual nightly runner meets the glibc floor. An unrebuilt image or an older-glibc runner fails at codex load time — visible only in the nightly, not in this run. (trigger: pyproject.toml) - 🟡 Prerelease publishes rc wheels to PUBLIC PyPI (not TestPyPI) and
publish-pypigates only onversion != '', so a branch dry-run permanently consumes a public PyPI version while its sole:<version>GHCR image can silently fail to publish (continue-on-error). The cross-consumer contract the prerelease exists to serve (ADO nightly pinningcoderEvalVersion) depends on BOTH artifacts landing; the PR doesn't state that a wheel-without-image is a half-published, nightly-breaking state. (trigger: .github/workflows/release.yml) (restates: Axis 6: Prerelease can half-publish: PyPI wheel ships even when the sole : GHCR image build fails)
Tests:
- 🟡 The new prerelease version-stamping logic ships untested: the rc-version derivation (
{next-patch}rc{run#}), there.subnstamp into pyproject.toml +__init__.py, the release-mode branch, and theCompute image tagsstep that must suppress:lateston a prerelease are all new code paths with no regression test. A bug that moves:lateston a prerelease or mis-stamps the version would only surface on a live dispatch. The embedded Python stamp snippet is extractable and unit-testable even though the surrounding workflow is not. (trigger: .github/workflows/release.yml)
Parallel paths:
- 🟡 Cross-agent permission_mode divergence left undocumented: after this change
permission_modestill confines the Claude agent (and the antigravity localharness policy hook), but is inert for Codex — the three registered agents now diverge on whetherpermission_modeis a confinement knob, with no unifying note in the agent docs or a warning at resolution. A user settingpermission_mode: plangets confinement on Claude and full-access on Codex from identical config. (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 7: Codex permission_mode is now a silently-inert no-op)
Downstream consumers:
- 🟡 Downstream text/logic describing the now-removed permission_mode->sandbox mapping wasn't updated: docs/TASK_DEFINITION_GUIDE.md still documents
planas 'waits for approval' with no Codex caveat, and codex_agent.py's_log_config_enforcementwarns only onbypassPermissionsand the debug line/dead workspace-write branch still narrate per-mode confinement. All are consumers of the mapping the PR deleted and read as load-bearing to a maintainer. (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 7: Codex permission_mode is now a silently-inert no-op)
Harness & Lint Improvements
Static checks (lint / type):
- [pyright] Flip
reportUnnecessaryComparison = "error"(and considerreportUnreachable = "error") in pyproject.toml's[tool.pyright]"Catch real logic bugs with minimal churn" block. After the PR hardcodedsandbox_mode_str = "full-access"(codex_agent.py:1268), the local narrows toLiteral["full-access"], so pyright evaluates the guardif sandbox_mode_str == "workspace-write":(line 1289) as statically always-False and marks its body (line 1290) unreachable — exactly the dead branch this review found. This is the highest-leverage flip because the deciding constant sits 21 lines above the guard, precisely the shape a narrowed-literal comparison check is built for. Standard mode leaves this rule at "none" today. Caveat: turn it on with a one-time baseline sweep ofmake typecheckto clear any pre-existing always-true/false comparisons before wiring it into CI. Prevents: Finding 1 (A1 High) — unreachable dead branch (codex_agent.py:1289-1290) left behind when_PERMISSION_MODE_TO_SANDBOXwas deleted; the always-False guard and its unreachable body would both failmake typecheckin CI before a human reviewed.
Harness improvements (not statically reachable):
- Replace the inert
os_name/in_containermatrix in tests/test_codex_agent.py (TestSandboxAlwaysFullAccess, lines 107-121) with a genuine driver × permission_mode → resolved-sandbox-posture table test: construct_build_thread_optionsfor each (driver, permission_mode) pair and assert the concreteoptions["sandbox"]. Today all 16 cases collapse to 4 identical assertions (the SUT reads neither os.name nor CODER_EVAL_IN_CONTAINER after the hardcode), giving false Windows/container coverage. A posture-table test would have FAILED loudly on this PR's silent switch of plan→read-only and acceptEdits→workspace-write to full-access, turning the reproducibility/security regression into a red test instead of a shipped default. Why not static: The resolved sandbox posture is a runtime product of config resolution (SandboxConfig.driver default, the CODER_EVAL_IN_CONTAINER env probe, and the permission_mode enum) flowing through_build_thread_options; asserting the mapping requires building the agent options object, which a grep/AST lint cannot evaluate. Prevents: Findings 2 (A3 Low, inert test dims), 3 (A4/A5/A8 High, lost OS write confinement), 5 (A7 High, permission_mode silently inert) — a future change to the sandbox mapping again silently altering posture. - Emit a one-time warning from the Codex agent when a config sets a confining permission_mode (plan/acceptEdits/default) that Codex no longer honors — full-access is now unconditional — and add a test asserting the warning fires (log capture) and that
_log_config_enforcement's [SECURITY] full-access notice (codex_agent.py:1355) is no longer gated to onlybypassPermissions. This closes the accepted-but-inert-knob gap: the field is still validated (PermissionMode enum, default ACCEPT_EDITS) yet has zero effect on Codex confinement, with no signal to the operator. Why not static: Whether a validated field is inert is a dataflow/semantic property (permission_mode is read at :1253 but only sinks into a debug log at :1279); detecting the missing warning and the stale per-mode conditional requires exercising the agent and inspecting emitted logs, not a static pattern match. Prevents: Findings 5 (A7 High, silent no-op) and 6 (A7 Medium, stale security warning special-casing only bypassPermissions while default is equally full-access). - Make the prerelease atomic in .github/workflows/release.yml: gate the
publish-pypijob'sif:on the versioned GHCR image build succeeding when IS_PRERELEASE=='true' (or dropcontinue-on-error: trueon the:<version>build step for prereleases, since that image is the prerelease's sole artifact, not 'internal/best-effort'), plus a post-publishdocker pull ghcr.io/...:<version>smoke step. This prevents a PyPI wheel from publishing to public PyPI with the release reporting success while the matching:<version>image is missing — the exact artifact the ADO nightly dry-run pins viacoderEvalVersion. Why not static: The defect is cross-job CI orchestration semantics (a step-levelcontinue-on-errorcombined with a downstream jobif:gate) resolved against the live registry and Actions runtime; it cannot be reached by linting the Python tree, and actionlint validates workflow syntax but not this cross-job artifact dependency. Prevents: Finding 4 (A6 Low) — prerelease half-publish where the wheel ships without its GHCR image and the run still reports green.
Top 5 Priority Actions
- Restore OS-level write confinement for host/tempdir Codex or gate the new full-access behind an explicit operator opt-in / fail-loud (codex_agent.py:1268), since the deleted in-container guard means the default
driver: tempdirhost path now runs untrusted agent actions with no Landlock/seatbelt backstop and the unenforced disallowed_tools warning at codex_agent.py:1302-1305 leaves no guardrail. - Stop silently ignoring permission_mode for Codex: emit a one-time warning (or reject) when a non-bypassPermissions mode is set — plan/acceptEdits/default all now resolve to full-access at codex_agent.py:1268 — and update docs/TASK_DEFINITION_GUIDE.md:120-124 which still document
planas confined. - Fix the now-misleading security messaging: make the [SECURITY] full-access notice in _log_config_enforcement (codex_agent.py:1355) fire for all Codex modes rather than only bypassPermissions, and delete the unreachable workspace-write network-opener dead branch plus stale comment at codex_agent.py:1284-1290, collapsing sandbox_mode_str to
Sandbox.full_accessdirectly. - Make the prerelease GHCR image build blocking (or gate publish-pypi on it) at .github/workflows/release.yml:263-264, so a failed
:<version>image can no longer half-publish an rc wheel to public PyPI while the workflow still reports success. - Tidy the honesty gaps around the change: annotate or drop the inert os_name/in_container matrix dimensions in tests/test_codex_agent.py:107 that assert invariance-by-construction, and disclose in the release.yml:28
bumpinput description that it is ignored on non-main prerelease dispatch.
Stats: 0 🔴 · 3 🟠 · 1 🟡 · 3 🔵 across 8 axes reviewed.
uipreliga
left a comment
There was a problem hiding this comment.
Fix what you agree with and 🚢
Addresses review on PR #33 (always-full-access): - Collapse the sandbox assignment to `Sandbox.full_access` and delete the now-unreachable `workspace-write` network-opener branch (dead since the mode became a constant). - `_log_config_enforcement` now emits the [SECURITY] full-access notice for EVERY permission_mode, not just bypassPermissions — all modes resolve to full-access, so the old per-mode warning misled operators. - Document that permission_mode does not confine Codex and that the docker driver is the OS-level write boundary (code comment + TASK_DEFINITION_GUIDE). - Note the test matrix's os_name/in_container dims are invariance-by-construction. - Disclose in the release.yml `bump` input that it is ignored off-main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

What
Makes the Codex and Antigravity harnesses actually run on coder_eval's tempdir / host execution path — the skill-activation suite and the container-less skills tasks — which today fail before doing any real work. The docker-driver path is unaffected either way.
Codex is now the daily canonical harness and the activation suite runs under every harness, so this path sits on the primary signal — yet it collapses today for independent reasons that share one locus: agents that ship native binaries and run their own in-process sandbox directly on the bare CI host. These are the "known caveat" the harness-refresh PR deliberately deferred as out of scope.
Bug 1 — Codex's in-process OS sandbox can't initialize on the CI host
Symptom. Codex's
read-only/workspace-writesandbox relies on an OS-level helper — Landlock inside the eval container, abwrapre-exec on the managed host — that can't start on the CI agent. Once it fails, the agent's file writes and commands fail silently and the task scores zero with no loud error, so the whole tempdir path (activation + container-less skills) reads as a model failure when it never actually ran.Fix. coder_eval always owns this run's isolation boundary — a docker container (docker driver,
CODER_EVAL_IN_CONTAINER) or an ephemeral per-task tempdir it creates and discards (tempdir driver). Codex's in-process OS sandbox is therefore redundant here, and — as this bug and Bug 3 show — unavailable or unenforceable on every CI host we run on. So_build_thread_optionsnow unconditionally selectssandbox=full-access(approval_mode=deny_all) for every permission mode, OS, and driver; the earlier per-driver conditional and thesandbox_managedflag that was threadedorchestrator → agentare removed. Hard isolation of untrusted actions stays coder_eval's job — the container, or the discarded tempdir — exactly as it already is for Claude Code.tests/test_codex_agent.pypins the invariant with a parametrized matrix (permission mode × in-container × OS → alwaysfull-access).Evidence. The downgrade fires on both slices of the rc11 dry-runs:
Permission mode acceptEdits → sandbox=full_access, approval_mode=deny_allon Linux (container) and Windows (tempdir), and codex tasks run to completion witherrors: 0. The Linux container path selectsfull-accessexactly as the validated codex dry-run 12756034 did (pass rate 90% (19/21) : OK,GATE: PASS); the clean rc11 Linux pass-rate is being reconfirmed on 12757607.Bug 2 — Antigravity's bundled binary can't load against the CI glibc
Symptom.
google-antigravity0.1.5/0.1.6 ship alocalharnessELF that usesDT_RELRrelocations andGLIBC_2.36symbols, but the wheel'smanylinux_2_17tag understates that floor. The binary fails to load on the CI agent's glibc, so the Antigravity agent never starts.Fix. Pin
google-antigravity==0.1.7(pyproject.toml), whoselocalharnessdropsDT_RELRand tops out atGLIBC_2.26.Evidence:
DT_RELR count: 0·max GLIBC symbol: GLIBC_2.26.Antigravity local harness started (model=gemini-3.5-flash)·host-sdk OK, and it passes the Windows tempdir smoke4/4. Every infra check is green (errors: 0 : OK·harness smoke: docker OK · host-sdk OK). The run's overallGATE: FAILis only the pass-rate threshold (86% = 18/21 < 90%) — a single model-quality task on a 21-task smoke, not a load or harness failure.Bug 3 — Codex can't run
uipon Windows at allSymptom. Windows has no OS-level sandbox helper (no Landlock, no seatbelt), so Codex cannot enforce
workspace-writeand denies the child process it spawns — everyuip …invocation exits-1before doing anything. Reported by Mihai on the Windows RPA suite: coded-agent / RPA tasks that shell out touipfailed uniformly on Windows.Root cause & fix. Same locus as Bug 1 — Codex's in-process sandbox is redundant (coder_eval owns isolation) and here outright unusable, because no OS helper exists on Windows to enforce it. The always-
full-accessfix above covers it: with the sandbox disabled, Codex spawnsuipdirectly. Windows is in fact why the earlier "container-or-tempdir" conditional was dropped in favour of an unconditionalfull-access— that conditional keyed on the driver, but the Windows failure is the OS, so anything short of "always" left a gap.Evidence — Windows codex dry-run 12757445 (rc11, always-
full-accessbuild): the tempdir slice logssandbox=full_access;harness smoke [codex]flipped0/1 → 1/1; a real 294s RPA task (template_aware_create) ranuiptoSUCCESS. (That run's Linux slice scored low from an unrelated runner footgun —agentModel=defaultresolved to a Bedrock id that 404s on codex's Azure endpoint — fixed in coder_eval_uipath#61 and reconfirmed on 12757607; not a sandbox regression.)Also in this PR
claude-agent-sdk 0.2.124,openai-codex 0.144.4,google-antigravity 0.1.7.openai-codex 0.144.4hard-pinsopenai-codex-cli-bin==0.144.4at the package level, socoder-eval[codex]pulls the matching CLI bin transitively (the[tool.uv] override-dependenciesentry is now redundant belt-and-suspenders).release.yml): a push on a non-mainbranch stamps<next-patch>rcN, regenerates the lock, and publishes the wheel to PyPI plus a:<version>GHCR image (never:latest), skipping the main commit / tag / push. The ADO pipeline installs a version-pinned wheel + agent image with no branch override, so this is what let the fixes above be validated through the real pipeline before merging.Dry-run set
Windows + Linux, isolated
runs-dryrunblob (no Slack, no prod metrics):harness smoke [codex]1/1 — https://dev.azure.com/uipath/coder_eval/_build/results?buildId=12757445agentModel=default— clean Linux pass-rate + the runner footgun-fix, end to end — https://dev.azure.com/uipath/coder_eval/_build/results?buildId=12757607 (in flight)