Skip to content

fix(agents): run codex + antigravity harnesses on the tempdir/host path#33

Merged
bai-uipath merged 9 commits into
mainfrom
bai/codex-antigravity-tempdir
Jul 22, 2026
Merged

fix(agents): run codex + antigravity harnesses on the tempdir/host path#33
bai-uipath merged 9 commits into
mainfrom
bai/codex-antigravity-tempdir

Conversation

@bai-uipath

@bai-uipath bai-uipath commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

What

Makes the Codex and Antigravity harnesses actually run on coder_eval's tempdir / host execution path — the skill-activation suite and the container-less skills tasks — which today fail before doing any real work. The docker-driver path is unaffected either way.

Codex is now the daily canonical harness and the activation suite runs under every harness, so this path sits on the primary signal — yet it collapses today for independent reasons that share one locus: agents that ship native binaries and run their own in-process sandbox directly on the bare CI host. These are the "known caveat" the harness-refresh PR deliberately deferred as out of scope.

Bug 1 — Codex's in-process OS sandbox can't initialize on the CI host

Symptom. Codex's read-only / workspace-write sandbox relies on an OS-level helper — Landlock inside the eval container, a bwrap re-exec on the managed host — that can't start on the CI agent. Once it fails, the agent's file writes and commands fail silently and the task scores zero with no loud error, so the whole tempdir path (activation + container-less skills) reads as a model failure when it never actually ran.

Fix. coder_eval always owns this run's isolation boundary — a docker container (docker driver, CODER_EVAL_IN_CONTAINER) or an ephemeral per-task tempdir it creates and discards (tempdir driver). Codex's in-process OS sandbox is therefore redundant here, and — as this bug and Bug 3 show — unavailable or unenforceable on every CI host we run on. So _build_thread_options now unconditionally selects sandbox=full-access (approval_mode=deny_all) for every permission mode, OS, and driver; the earlier per-driver conditional and the sandbox_managed flag that was threaded orchestrator → agent are removed. Hard isolation of untrusted actions stays coder_eval's job — the container, or the discarded tempdir — exactly as it already is for Claude Code. tests/test_codex_agent.py pins the invariant with a parametrized matrix (permission mode × in-container × OS → always full-access).

Evidence. The downgrade fires on both slices of the rc11 dry-runs: Permission mode acceptEdits → sandbox=full_access, approval_mode=deny_all on Linux (container) and Windows (tempdir), and codex tasks run to completion with errors: 0. The Linux container path selects full-access exactly as the validated codex dry-run 12756034 did (pass rate 90% (19/21) : OK, GATE: PASS); the clean rc11 Linux pass-rate is being reconfirmed on 12757607.

Bug 2 — Antigravity's bundled binary can't load against the CI glibc

Symptom. google-antigravity 0.1.5/0.1.6 ship a localharness ELF that uses DT_RELR relocations and GLIBC_2.36 symbols, but the wheel's manylinux_2_17 tag understates that floor. The binary fails to load on the CI agent's glibc, so the Antigravity agent never starts.

Fix. Pin google-antigravity==0.1.7 (pyproject.toml), whose localharness drops DT_RELR and tops out at GLIBC_2.26.

Evidence:

  • ELF check on glibc 2.39: DT_RELR count: 0 · max GLIBC symbol: GLIBC_2.26.
  • Antigravity dry-run 12756035: Antigravity local harness started (model=gemini-3.5-flash) · host-sdk OK, and it passes the Windows tempdir smoke 4/4. Every infra check is green (errors: 0 : OK · harness smoke: docker OK · host-sdk OK). The run's overall GATE: FAIL is only the pass-rate threshold (86% = 18/21 < 90%) — a single model-quality task on a 21-task smoke, not a load or harness failure.

Bug 3 — Codex can't run uip on Windows at all

Symptom. Windows has no OS-level sandbox helper (no Landlock, no seatbelt), so Codex cannot enforce workspace-write and denies the child process it spawns — every uip … invocation exits -1 before doing anything. Reported by Mihai on the Windows RPA suite: coded-agent / RPA tasks that shell out to uip failed uniformly on Windows.

Root cause & fix. Same locus as Bug 1 — Codex's in-process sandbox is redundant (coder_eval owns isolation) and here outright unusable, because no OS helper exists on Windows to enforce it. The always-full-access fix above covers it: with the sandbox disabled, Codex spawns uip directly. Windows is in fact why the earlier "container-or-tempdir" conditional was dropped in favour of an unconditional full-access — that conditional keyed on the driver, but the Windows failure is the OS, so anything short of "always" left a gap.

Evidence — Windows codex dry-run 12757445 (rc11, always-full-access build): the tempdir slice logs sandbox=full_access; harness smoke [codex] flipped 0/1 → 1/1; a real 294s RPA task (template_aware_create) ran uip to SUCCESS. (That run's Linux slice scored low from an unrelated runner footgun — agentModel=default resolved to a Bedrock id that 404s on codex's Azure endpoint — fixed in coder_eval_uipath#61 and reconfirmed on 12757607; not a sandbox regression.)

Also in this PR

  • SDKs bumped to current: claude-agent-sdk 0.2.124, openai-codex 0.144.4, google-antigravity 0.1.7. openai-codex 0.144.4 hard-pins openai-codex-cli-bin==0.144.4 at the package level, so coder-eval[codex] pulls the matching CLI bin transitively (the [tool.uv] override-dependencies entry is now redundant belt-and-suspenders).
  • Prerelease-from-branch publishing (release.yml): a push on a non-main branch stamps <next-patch>rcN, regenerates the lock, and publishes the wheel to PyPI plus a :<version> GHCR image (never :latest), skipping the main commit / tag / push. The ADO pipeline installs a version-pinned wheel + agent image with no branch override, so this is what let the fixes above be validated through the real pipeline before merging.

Dry-run set

Windows + Linux, isolated runs-dryrun blob (no Slack, no prod metrics):

bai-uipath and others added 6 commits July 21, 2026 12:32
The pinned 0.1.5 localharness binary requires the GLIBC_ABI_DT_RELR ABI
(glibc 2.36+), so it fails to load on the eval agents' Ubuntu 22.04 (glibc
2.35), taking down the antigravity tempdir/host path (the activation suite
and container-less skills tasks) while docker-driver tasks stay fine. 0.1.7
links without DT_RELR (max required symbol GLIBC_2.26) and loads on 2.35+.
The manylinux_2_17 wheel tag alone does not guarantee this, so the pin
comment documents verifying ELF load before any future bump.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codex's read-only / workspace-write sandboxes shell out to an OS-level
helper (Landlock, or a bwrap re-exec) that fails to initialize on managed CI
hosts ("bwrap: execvp .../codex: Permission denied" where unprivileged user
namespaces or exec mounts are restricted); its writes/execs then fail
silently and the task produces no artifacts. The docker path already avoids
this via CODER_EVAL_IN_CONTAINER. Thread a `sandbox_managed` signal through
Agent.start() so the tempdir driver — where coder_eval's ephemeral per-task
dir is the isolation boundary — also falls back to full-access, matching
Claude Code (no in-agent OS sandbox on that path). Hard isolation of
untrusted actions remains the docker driver's job.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The orchestrator now threads `sandbox_managed` into `Agent.start()`, and
the codebase's convention is that agent test doubles mirror the ABC
signature exactly. Add the keyword-only param (accepted and ignored,
like `env_path_prepend`) to every double that declares `start()`, so the
orchestrator-routed doubles (MockAgent and the inline DummyAgents) stop
raising `unexpected keyword argument 'sandbox_managed'`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…wcw-mr6r

pyasn1 0.6.3 (transitive via google-auth → pyasn1-modules) carries two
High advisories fixed in 0.6.4. Pre-existing on main; bumping the lock
clears both osv-scanner and pip-audit. No suppression entry is added
because a fix is available (osv-scanner.toml is only for unfixable advisories).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…4.4)

- claude-agent-sdk 0.2.82 -> 0.2.124
- openai-codex 0.1.0b3 -> 0.144.4, and its pinned cli-bin 0.137.0a4 -> 0.144.4
  (the SDK version now tracks the codex CLI line; the 0.144.x jump is a
  re-numbering, not an API rewrite). The public API surface codex_agent.py
  depends on — Codex/Thread/turn, the notification method strings, item types,
  and token-usage fields — stayed source-compatible, so no agent code changed.
- google-antigravity already at 0.1.7 (latest; pinned by the tempdir fix).

Full suite incl. the live codex integration tests is green on all three.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dispatching release.yml from a branch (github.ref != main) now runs a
prerelease instead of a real release: stamp a throwaway <next-patch>rcN
version into pyproject/__init__, refresh uv.lock, build, publish the wheel
to public PyPI (OIDC) and a :<version> GHCR agent image. It does NOT move
:latest, tag, commit, or push to main. This lets a branch be dry-run on the
ADO nightly infra (pinned via coderEvalVersion) before merge. The main path
is unchanged; the bump input is ignored off main.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @bai-uipath's task in 2m 8s —— View job


PR Review in Progress

  • Read code review guidelines from .github/code_review.md
  • Examine full PR diff (origin/main...HEAD)
  • Review each changed file with surrounding context
  • Check cross-file consistency
  • Analyze design decisions and identify missing elements
  • Provide comprehensive review feedback

Starting review now...

coder_eval always owns a run's isolation boundary — a docker container
(docker driver) or an ephemeral per-task tempdir it creates and discards
(tempdir driver); those are the only two drivers — so Codex's own
in-process OS sandbox (Landlock/seatbelt) is always redundant. Worse, it
breaks on the paths we rely on: Landlock is unavailable in the container,
the bwrap re-exec is denied on constrained CI agents, and Windows has no
OS sandbox at all — in each case a read-only/workspace-write run fails its
writes/execs silently and scores 0. This is what left the Windows
uipath-rpa tasks unable to run `uip` (exit -1) even after the earlier
container/tempdir fallback landed.

Run full-access unconditionally (approval_mode stays deny_all), matching
Claude Code and Antigravity, which run with no in-agent OS sandbox; hard
isolation of untrusted actions is the docker driver's job. Removes the
now-obsolete sandbox_managed plumbing and the per-mode sandbox mapping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@uipreliga uipreliga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: coder_eval — pr:33

Scope: pr:33 · branch bai/codex-antigravity-tempdir · 6b43af5 · 2026-07-22T00:34Z · workflow variant

Change class: complex — removes Codex's in-process OS sandbox so it always runs full-access for every permission mode; a security-sensitive control-flow change, plus CI prerelease-mode logic and SDK/CLI-binary version bumps

coder_eval is in strong health (9.5/10) with exemplary type safety, architecture, and harness quality, but a single recent PR — hardcoding the Codex sandbox to full-access (codex_agent.py:1268) — drives every real risk: it strips the tempdir/host driver's only OS-level write confinement (a defense-in-depth regression on the default path) and silently turns the accepted-and-validated permission_mode knob into a no-op with stale security messaging still describing the removed mapping, so the bottom line is that the codebase is production-grade but this concentrated Codex-sandbox change needs a deliberate confinement/opt-in decision plus doc and messaging cleanup before it stops misleading operators.

Summary

Axis Score 🔴 🟠 🟡 🔵 Top Issue
1. Code Quality & Style 9 / 10 0 1 0 0 Unreachable dead branch: workspace-write network-access opener can never fire after sandbox is hardcoded to full-access
2. Type Safety 10 / 10 0 0 0 0
3. Test Health 9.9 / 10 0 0 0 1 TestSandboxAlwaysFullAccess os_name/in_container dimensions exercise no code path (duplicate cases, mild false-confidence)
4. Security 9 / 10 0 1 0 0 Hardcoding Codex sandbox to full-access removes the tempdir/host driver's only OS-level write confinement (security + reproducibility impact)
5. Architecture & Design 10 / 10 0 0 0 0
6. Error Handling & Resilience 9.9 / 10 0 0 0 1 Prerelease can half-publish: PyPI wheel ships even when the sole : GHCR image build fails (continue-on-error), with the workflow still reporting success
7. API Surface & Maintainability 8.4 / 10 0 1 1 1 Codex permission_mode is now a silently-inert no-op — plan/acceptEdits/default/bypassPermissions all resolve to full-access with no warn/reject
8. Evaluation Harness Quality 10 / 10 0 0 0 0

Overall Score: 9.5 / 10 · Weakest Axis: API Surface & Maintainability at 8.4 / 10
Totals: 🔴 0 · 🟠 3 · 🟡 1 · 🔵 3 across 8 axes.

Blockers

  1. [Axis 1] Unreachable dead branch: workspace-write network-access opener can never fire after sandbox is hardcoded to full-access (src/coder_eval/agents/codex_agent.py:1289) — Line 1268 now hardcodes sandbox_mode_str = "full-access", so the guard at line 1289 if sandbox_mode_str == "workspace-write": is statically always False and line 1290 tool_config["sandbox_workspace_write"] = {"network_access": True} is unreachable dead code (coverage confirms line 1290 is unreached). The whole 5-line comment block at lines 1284-1288 ("Codex's workspace-write sandbox disables network by default... Always open network in workspace-write") describes behavior that can never execute, and reads as load-bearing to a maintainer. This is the residue the PR left behind when it deleted _PERMISSION_MODE_TO_SANDBOX. Fix: delete lines 1284-1290 entirely. Since sandbox_mode_str is now a single-use constant, also collapse it — assign options["sandbox"] = Sandbox.full_access directly (matching the test's Sandbox.full_access assertion) rather than routing a magic string through Sandbox(sandbox_mode_str) at line 1272. This finding matches the Axis-1 Critical anchor's every clause (dead code on a live path, looks load-bearing, misleads); it is filed High because it has zero runtime effect and the deciding = "full-access" constant sits 21 lines above the guard, making the deadness discoverable at a glance — a reviewer could defensibly rate it Critical.
  2. [Axis 4] Hardcoding Codex sandbox to full-access removes the tempdir/host driver's only OS-level write confinement (security + reproducibility impact) (src/coder_eval/agents/codex_agent.py:1268) — _build_thread_options was changed to sandbox_mode_str = "full-access" unconditionally (line 1268), deleting the _PERMISSION_MODE_TO_SANDBOX table and the if sandbox_mode_str != "full-access" and os.getenv("CODER_EVAL_IN_CONTAINER") guard. That guard fired ONLY inside a docker container; on the default driver: tempdir host path (SandboxConfig.driver default="tempdir", models/sandbox.py:309-312) CODER_EVAL_IN_CONTAINER is unset, so pre-PR Codex ran with its Landlock/seatbelt sandbox — plan→read-only, default/acceptEdits (the config default, agent_config.py:139-141)→workspace-write — which confined agent writes/execs to the workspace dir. Post-PR every mode is full-access: an untrusted agent action (adversarial model, or prompt injection carried in task-repo/prompt content processed unattended by nightly/host runs) can now write, delete, or exec anywhere the eval user can on the host (~/.ssh, ~/.aws, ~/.bashrc, arbitrary rm -rf), where Landlock previously blocked it. The docker path (docker_runner.py:1098 sets CODER_EVAL_IN_CONTAINER=1) was already full-access and is unaffected — so this is a defense-in-depth REGRESSION scoped to the default host driver, not a change to the isolated production path. The code's own comment concedes hard isolation of untrusted actions is the docker driver's job. Recommendation (open question on exact mechanism): gate host/tempdir Codex behind an explicit operator opt-in or fail loud, and note that with full-access now on, the loudly-warned unenforced disallowed_tools (codex_agent.py:1302-1305) leaves NO backstop guardrail on the host path. Base score 8.0 = High. CVSS:3.1/AV:L/AC:L/PR:N/UI:N/S:U/C:L/I:H/A:H
  3. [Axis 7] Codex permission_mode is now a silently-inert no-op — plan/acceptEdits/default/bypassPermissions all resolve to full-access with no warn/reject (src/coder_eval/agents/codex_agent.py:1268) — _build_thread_options now hardcodes sandbox_mode_str = "full-access" (codex_agent.py:1268) for every permission mode; permission_mode (read at :1253) no longer influences the sandbox. Previously plan→read-only and acceptEdits/default→workspace-write, so on the tempdir/host driver a Codex run set to permission_mode: plan was confined; now the identical config yields a fully-writable agent, silently. The field is still accepted and validated (PermissionMode enum, default ACCEPT_EDITS at models/agent_config.py:139) but is inert for Codex, and the SDK gets no signal that the user asked for confinement. docs/TASK_DEFINITION_GUIDE.md:120-124 still document plan as "Agent proposes changes, waits for approval" with no note that these are inert for Codex. Either emit a one-time warning when a Codex config sets a non-bypassPermissions permission_mode (that it does not confine — matching the existing loud-on-surprise style elsewhere in this file), or document permission_mode as Claude-only for confinement and update the guide's Permission Modes / Agent Types sections. The design rationale (Codex's in-process Landlock/seatbelt sandbox is redundant given coder_eval's docker/tempdir boundary, and breaks silently on CI/Windows) is sound — the gap is that the user-facing knob stays accepted with zero effect and no warning.

Non-blocking, but please consider before merge

  1. [Axis 7] Stale messaging referencing the removed permission_mode->sandbox mapping (docstring/debug log at 1279 and security warning at 1355) (src/coder_eval/agents/codex_agent.py:1355) — _log_config_enforcement still special-cases only one mode: if self.config.permission_mode.value == "bypassPermissions": (codex_agent.py:1355) emits "[SECURITY] bypassPermissions grants unrestricted sandbox access (full-access). Only use in fully isolated environments...". After this change default, acceptEdits (the default), and plan ALL resolve to full-access too, but emit no such warning — so the warning now implies the other modes are more confined than bypassPermissions when the sandbox posture is identical across all four. An operator running the default acceptEdits with Codex gets full-access with no security notice. Either drop the per-mode conditional (all modes are full-access now, so the notice should fire unconditionally for Codex) or reword it to reflect that Codex always runs full-access regardless of permission_mode. Also note the debug line at :1279 (Permission mode {permission_mode} → sandbox=...) now always prints full-access.

Nits

  1. [Axis 3] TestSandboxAlwaysFullAccess os_name/in_container dimensions exercise no code path (duplicate cases, mild false-confidence) (tests/test_codex_agent.py:107) — The matrix at lines 107-109 parametrizes os_name∈{posix,nt} and in_container∈{T,F}, and line 119 does monkeypatch.setattr(_os, "name", os_name). But _build_thread_options (the SUT, called at line 121) reads neither os.name nor CODER_EVAL_IN_CONTAINER after this PR — it hardcodes sandbox_mode_str = "full-access" (codex_agent.py:1268), and the only os.name read (_login_shell_profiles_supported, codex_agent.py:1098) is never on this call path. So the 16 cases reduce to 4 effective ones (one per permission mode); the 8 os_name="nt" cases assert identically to the posix cases and give the appearance of Windows coverage that isn't exercised. This is a defensible regression guard against reintroducing an os.name/container branch, so keep it if intentional — but add a one-line comment stating the assertion is invariance-by-construction, or drop the two inert dimensions to keep the matrix honest about what it covers.
  2. [Axis 6] Prerelease can half-publish: PyPI wheel ships even when the sole : GHCR image build fails (continue-on-error), with the workflow still reporting success (.github/workflows/release.yml:264) — In prerelease mode the :<version> GHCR image is the only image artifact (a real release additionally gets :latest from docker-publish.yml, a prerelease does not — see the Compute image tags step at lines 231-247), yet Build and push versioned agent image (lines 263-264) carries continue-on-error: true # GHCR image is internal/best-effort; don't block PyPI, and the publish-pypi job gates only on if: needs.release.outputs.version != '' (line 288). So if the image build fails, the wheel still publishes to public PyPI and the release job reports success, leaving the stamped rc version half-published — a PyPI wheel with no matching :<version> image, which is exactly the artifact the prerelease's stated purpose (ADO nightly dry-run pinned via coderEvalVersion) depends on. The failure only surfaces downstream in the nightly docker pull, not in this run. Impact is low (rc is throwaway, the image step still shows a red X in the Actions log, and a re-dispatch mints a fresh rc via the run number), but consider making the image step blocking (or at least gating the prerelease image on success) when IS_PRERELEASE == 'true', since for a prerelease the image is not merely 'internal/best-effort'.
  3. [Axis 7] release.yml bump input does not disclose it is ignored on non-main (prerelease) dispatch (.github/workflows/release.yml:28) — The bump choice input (release.yml:27-34, description 'Version bump to release') is silently ignored when the workflow is dispatched from a non-main branch: the prerelease path always stamps <next-patch>rc<run#> (Determine release mode step) regardless of whether the dispatcher selected minor/major. The header comment at line 22 documents "The bump input is ignored off main," but that comment is not visible in the Actions dispatch UI — the input description is the only text a dispatcher sees. Append the caveat to the input description, e.g. 'Version bump to release (ignored on non-main / prerelease dispatch, which always stamps a next-patch rc)', so the branch-ref-keyed behavior is discoverable at the point of use.

What's Missing

Daily/nightly:

  • 🟠 Unstated nightly blast radius of forcing Codex to full-access: the PR removes OS-level write confinement on the default tempdir driver but never states what the nightly/cron pipeline runs Codex under. If any nightly Codex job uses the default tempdir driver (not docker), it now executes unconfined on the host — a production-path behavior change the PR body should call out and confirm the nightly is docker-only. The change explicitly exists to enable an ADO nightly dry-run (release.yml prerelease comment), so the nightly interaction is squarely in scope yet unaddressed. (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 4: Hardcoding Codex sandbox to full-access removes the tempdir/host driver's OS-level write confinement)
  • 🟡 Container-rebuild-in-lockstep not stated for the codex-cli-bin 0.144.4 bump + glibc floor: the pyproject note documents that the bundled localharness/codex ELF must load on Ubuntu 22.04 (glibc 2.35) but the PR never confirms the nightly docker/Dockerfile image is rebuilt with the new pin, nor that the actual nightly runner meets the glibc floor. An unrebuilt image or an older-glibc runner fails at codex load time — visible only in the nightly, not in this run. (trigger: pyproject.toml)
  • 🟡 Prerelease publishes rc wheels to PUBLIC PyPI (not TestPyPI) and publish-pypi gates only on version != '', so a branch dry-run permanently consumes a public PyPI version while its sole :<version> GHCR image can silently fail to publish (continue-on-error). The cross-consumer contract the prerelease exists to serve (ADO nightly pinning coderEvalVersion) depends on BOTH artifacts landing; the PR doesn't state that a wheel-without-image is a half-published, nightly-breaking state. (trigger: .github/workflows/release.yml) (restates: Axis 6: Prerelease can half-publish: PyPI wheel ships even when the sole : GHCR image build fails)

Tests:

  • 🟡 The new prerelease version-stamping logic ships untested: the rc-version derivation ({next-patch}rc{run#}), the re.subn stamp into pyproject.toml + __init__.py, the release-mode branch, and the Compute image tags step that must suppress :latest on a prerelease are all new code paths with no regression test. A bug that moves :latest on a prerelease or mis-stamps the version would only surface on a live dispatch. The embedded Python stamp snippet is extractable and unit-testable even though the surrounding workflow is not. (trigger: .github/workflows/release.yml)

Parallel paths:

  • 🟡 Cross-agent permission_mode divergence left undocumented: after this change permission_mode still confines the Claude agent (and the antigravity localharness policy hook), but is inert for Codex — the three registered agents now diverge on whether permission_mode is a confinement knob, with no unifying note in the agent docs or a warning at resolution. A user setting permission_mode: plan gets confinement on Claude and full-access on Codex from identical config. (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 7: Codex permission_mode is now a silently-inert no-op)

Downstream consumers:

  • 🟡 Downstream text/logic describing the now-removed permission_mode->sandbox mapping wasn't updated: docs/TASK_DEFINITION_GUIDE.md still documents plan as 'waits for approval' with no Codex caveat, and codex_agent.py's _log_config_enforcement warns only on bypassPermissions and the debug line/dead workspace-write branch still narrate per-mode confinement. All are consumers of the mapping the PR deleted and read as load-bearing to a maintainer. (trigger: src/coder_eval/agents/codex_agent.py) (restates: Axis 7: Codex permission_mode is now a silently-inert no-op)

Harness & Lint Improvements

Static checks (lint / type):

  • [pyright] Flip reportUnnecessaryComparison = "error" (and consider reportUnreachable = "error") in pyproject.toml's [tool.pyright] "Catch real logic bugs with minimal churn" block. After the PR hardcoded sandbox_mode_str = "full-access" (codex_agent.py:1268), the local narrows to Literal["full-access"], so pyright evaluates the guard if sandbox_mode_str == "workspace-write": (line 1289) as statically always-False and marks its body (line 1290) unreachable — exactly the dead branch this review found. This is the highest-leverage flip because the deciding constant sits 21 lines above the guard, precisely the shape a narrowed-literal comparison check is built for. Standard mode leaves this rule at "none" today. Caveat: turn it on with a one-time baseline sweep of make typecheck to clear any pre-existing always-true/false comparisons before wiring it into CI. Prevents: Finding 1 (A1 High) — unreachable dead branch (codex_agent.py:1289-1290) left behind when _PERMISSION_MODE_TO_SANDBOX was deleted; the always-False guard and its unreachable body would both fail make typecheck in CI before a human reviewed.

Harness improvements (not statically reachable):

  • Replace the inert os_name/in_container matrix in tests/test_codex_agent.py (TestSandboxAlwaysFullAccess, lines 107-121) with a genuine driver × permission_mode → resolved-sandbox-posture table test: construct _build_thread_options for each (driver, permission_mode) pair and assert the concrete options["sandbox"]. Today all 16 cases collapse to 4 identical assertions (the SUT reads neither os.name nor CODER_EVAL_IN_CONTAINER after the hardcode), giving false Windows/container coverage. A posture-table test would have FAILED loudly on this PR's silent switch of plan→read-only and acceptEdits→workspace-write to full-access, turning the reproducibility/security regression into a red test instead of a shipped default. Why not static: The resolved sandbox posture is a runtime product of config resolution (SandboxConfig.driver default, the CODER_EVAL_IN_CONTAINER env probe, and the permission_mode enum) flowing through _build_thread_options; asserting the mapping requires building the agent options object, which a grep/AST lint cannot evaluate. Prevents: Findings 2 (A3 Low, inert test dims), 3 (A4/A5/A8 High, lost OS write confinement), 5 (A7 High, permission_mode silently inert) — a future change to the sandbox mapping again silently altering posture.
  • Emit a one-time warning from the Codex agent when a config sets a confining permission_mode (plan/acceptEdits/default) that Codex no longer honors — full-access is now unconditional — and add a test asserting the warning fires (log capture) and that _log_config_enforcement's [SECURITY] full-access notice (codex_agent.py:1355) is no longer gated to only bypassPermissions. This closes the accepted-but-inert-knob gap: the field is still validated (PermissionMode enum, default ACCEPT_EDITS) yet has zero effect on Codex confinement, with no signal to the operator. Why not static: Whether a validated field is inert is a dataflow/semantic property (permission_mode is read at :1253 but only sinks into a debug log at :1279); detecting the missing warning and the stale per-mode conditional requires exercising the agent and inspecting emitted logs, not a static pattern match. Prevents: Findings 5 (A7 High, silent no-op) and 6 (A7 Medium, stale security warning special-casing only bypassPermissions while default is equally full-access).
  • Make the prerelease atomic in .github/workflows/release.yml: gate the publish-pypi job's if: on the versioned GHCR image build succeeding when IS_PRERELEASE=='true' (or drop continue-on-error: true on the :<version> build step for prereleases, since that image is the prerelease's sole artifact, not 'internal/best-effort'), plus a post-publish docker pull ghcr.io/...:<version> smoke step. This prevents a PyPI wheel from publishing to public PyPI with the release reporting success while the matching :<version> image is missing — the exact artifact the ADO nightly dry-run pins via coderEvalVersion. Why not static: The defect is cross-job CI orchestration semantics (a step-level continue-on-error combined with a downstream job if: gate) resolved against the live registry and Actions runtime; it cannot be reached by linting the Python tree, and actionlint validates workflow syntax but not this cross-job artifact dependency. Prevents: Finding 4 (A6 Low) — prerelease half-publish where the wheel ships without its GHCR image and the run still reports green.

Top 5 Priority Actions

  1. Restore OS-level write confinement for host/tempdir Codex or gate the new full-access behind an explicit operator opt-in / fail-loud (codex_agent.py:1268), since the deleted in-container guard means the default driver: tempdir host path now runs untrusted agent actions with no Landlock/seatbelt backstop and the unenforced disallowed_tools warning at codex_agent.py:1302-1305 leaves no guardrail.
  2. Stop silently ignoring permission_mode for Codex: emit a one-time warning (or reject) when a non-bypassPermissions mode is set — plan/acceptEdits/default all now resolve to full-access at codex_agent.py:1268 — and update docs/TASK_DEFINITION_GUIDE.md:120-124 which still document plan as confined.
  3. Fix the now-misleading security messaging: make the [SECURITY] full-access notice in _log_config_enforcement (codex_agent.py:1355) fire for all Codex modes rather than only bypassPermissions, and delete the unreachable workspace-write network-opener dead branch plus stale comment at codex_agent.py:1284-1290, collapsing sandbox_mode_str to Sandbox.full_access directly.
  4. Make the prerelease GHCR image build blocking (or gate publish-pypi on it) at .github/workflows/release.yml:263-264, so a failed :<version> image can no longer half-publish an rc wheel to public PyPI while the workflow still reports success.
  5. Tidy the honesty gaps around the change: annotate or drop the inert os_name/in_container matrix dimensions in tests/test_codex_agent.py:107 that assert invariance-by-construction, and disclose in the release.yml:28 bump input description that it is ignored on non-main prerelease dispatch.

Stats: 0 🔴 · 3 🟠 · 1 🟡 · 3 🔵 across 8 axes reviewed.

@uipreliga
uipreliga self-requested a review July 22, 2026 00:52

@uipreliga uipreliga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix what you agree with and 🚢

bai-uipath and others added 2 commits July 21, 2026 18:11
Addresses review on PR #33 (always-full-access):
- Collapse the sandbox assignment to `Sandbox.full_access` and delete the
  now-unreachable `workspace-write` network-opener branch (dead since the
  mode became a constant).
- `_log_config_enforcement` now emits the [SECURITY] full-access notice for
  EVERY permission_mode, not just bypassPermissions — all modes resolve to
  full-access, so the old per-mode warning misled operators.
- Document that permission_mode does not confine Codex and that the docker
  driver is the OS-level write boundary (code comment + TASK_DEFINITION_GUIDE).
- Note the test matrix's os_name/in_container dims are invariance-by-construction.
- Disclose in the release.yml `bump` input that it is ignored off-main.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@bai-uipath
bai-uipath merged commit 04498a9 into main Jul 22, 2026
12 checks passed
@bai-uipath
bai-uipath deleted the bai/codex-antigravity-tempdir branch July 22, 2026 01:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants