Run Harbor Compose tasks on Prime and Modal - #2549
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| while ( | ||
| services.get(owner, {}) | ||
| .get("network_mode", "") | ||
| .startswith("service:") | ||
| ): | ||
| owner = services[owner]["network_mode"].split(":", 1)[1] |
There was a problem hiding this comment.
🟠 High harbor/cloud.py:306
A cyclic network_mode: service: chain makes this loop spin forever before docker compose config runs, blocking the event loop and preventing the sandbox from being torn down. Track visited services and raise a SandboxError when a service repeats so invalid topology is rejected without hanging.
- while (
- services.get(owner, {})
- .get("network_mode", "")
- .startswith("service:")
- ):
- owner = services[owner]["network_mode"].split(":", 1)[1]
+ seen: set[str] = set()
+ while (
+ services.get(owner, {})
+ .get("network_mode", "")
+ .startswith("service:")
+ ):
+ if owner in seen:
+ raise SandboxError("Cyclic Compose network_mode service chain")
+ seen.add(owner)
+ owner = services[owner]["network_mode"].split(":", 1)[1]🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/tasksets/harbor/cloud.py around lines 306-311:
A cyclic `network_mode: service:` chain makes this loop spin forever before `docker compose config` runs, blocking the event loop and preventing the sandbox from being torn down. Track visited services and raise a `SandboxError` when a service repeats so invalid topology is rejected without hanging.
| self.owner.ops.env, | ||
| ) | ||
| try: | ||
| async with asyncio.timeout(30): |
There was a problem hiding this comment.
🟡 Medium harbor/cloud.py:130
When docker compose exec exits before the wrapper writes pidfile, open_process ignores that failure and keeps polling cat for up to 30 seconds instead of surfacing the process's exit status promptly. Monitor the outer RuntimeProcess while waiting for the PID and raise its failure as soon as it exits.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/tasksets/harbor/cloud.py around line 130:
When `docker compose exec` exits before the wrapper writes `pidfile`, `open_process` ignores that failure and keeps polling `cat` for up to 30 seconds instead of surfacing the process's exit status promptly. Monitor the outer `RuntimeProcess` while waiting for the PID and raise its failure as soon as it exits.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1d815d794
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| source=entry.source, | ||
| service=effective_artifact_service(entry), |
There was a problem hiding this comment.
When a local Docker Compose task declares a relative artifact for a non-main service whose working_dir is not /, accepting the service here collects the wrong absolute path. HarborComposeRuntime.service() hardcodes every sidecar's workdir to /, and DockerRuntime.run() passes that value via docker exec --workdir, so relative artifacts such as output.json are looked up as /output.json instead of, for example, /app/output.json (and sidecar collect hooks similarly run from /). Inspect each sidecar's configured/container working directory as the remote adapters do before enabling these declarations.
Useful? React with 👍 / 👎.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This PR adds Prime and Modal VM-backed Harbor Compose execution and substantially changes shared runtime, rollout lifecycle, networking, teardown, and multi-service artifact paths. The new infrastructure scope and unresolved concerns around remote bootstrap, sandbox cleanup, and sidecar artifact handling warrant human review. Not approved because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
d1d815d to
5d98ce7
Compare
| if self._owner is not None: | ||
| raise SandboxError("Only the main Compose service publishes a runtime port") | ||
| url = ( | ||
| await self._host.expose(port) |
There was a problem hiding this comment.
🟠 High harbor/runtime.py:223
Prime-hosted Compose runs fail whenever runtime.expose() is needed: this delegates to PrimeComposeVM.expose(), which unconditionally raises SandboxError, so SERVICE_PORT can never be published for non-colocated tools. Route Prime Compose exposure through the Compose service's published port instead of the host's unsupported expose() implementation.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/tasksets/harbor/runtime.py around line 223:
Prime-hosted Compose runs fail whenever `runtime.expose()` is needed: this delegates to `PrimeComposeVM.expose()`, which unconditionally raises `SandboxError`, so `SERVICE_PORT` can never be published for non-colocated tools. Route Prime Compose exposure through the Compose service's published port instead of the host's unsupported `expose()` implementation.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5d98ce71ff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "-c", | ||
| ( | ||
| "export DEBIAN_FRONTEND=noninteractive; apt-get update -qq && " | ||
| "apt-get install -y -qq --no-install-recommends docker.io docker-cli docker-compose iptables " |
There was a problem hiding this comment.
Every Prime Compose run boots the hard-coded python:3.11-slim Debian-family image and executes this install, but its standard APT repositories do not provide a package named docker-cli (the CLI is supplied by docker.io, while Compose v2 needs the appropriate plugin package). Because APT aborts the whole transaction when any requested package is unavailable, PrimeComposeVM.start() reports Docker bootstrap failed before starting dockerd, making the newly advertised Prime Compose runtime unusable.
Useful? React with 👍 / 👎.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 similar comment
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
| logger.warning( | ||
| "runtime teardown failed (rollout %s)", trace.id, exc_info=True | ||
| ) | ||
| trace.is_completed = True |
There was a problem hiding this comment.
🟡 Medium v1/rollout.py:553
While task.cleanup() runs, the rollout remains marked live even though every timing span is closed, so _stage() falls back to "boot" and the dashboard shows cleanup/sidecar collection as booting. Mark trace.is_completed = True before cleanup begins so live status reports completion instead.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 553:
While `task.cleanup()` runs, the rollout remains marked live even though every timing span is closed, so `_stage()` falls back to `"boot"` and the dashboard shows cleanup/sidecar collection as booting. Mark `trace.is_completed = True` before cleanup begins so live status reports completion instead.
| if sandbox is None: | ||
| return | ||
| # Preserve the handle for the cleanup backstop until termination is confirmed. | ||
| async with asyncio.timeout(60): |
There was a problem hiding this comment.
🟠 High harbor/modal.py:34
When sandbox.terminate.aio() or sandbox.wait.aio(...) exceeds the 60-second timeout, teardown raises while cleanup still depends on this runtime instance. Once Compose unwinds, the runtime can disappear from the WeakSet, leaving the paid Modal VM running until its 24-hour provider timeout. Retain the sandbox in a strong cleanup mechanism or complete termination independently of the runtime object's lifetime before propagating the exception.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/tasksets/harbor/modal.py around line 34:
When `sandbox.terminate.aio()` or `sandbox.wait.aio(...)` exceeds the 60-second timeout, `teardown` raises while cleanup still depends on this runtime instance. Once Compose unwinds, the runtime can disappear from the `WeakSet`, leaving the paid Modal VM running until its 24-hour provider timeout. Retain the sandbox in a strong cleanup mechanism or complete termination independently of the runtime object's lifetime before propagating the exception.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 4 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 044e2b0. Configure here.
044e2b0 to
c1ce953
Compare
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
1 similar comment
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
| collected = await collect( | ||
| runtime, | ||
| self.data.artifacts, | ||
| max_bytes=self.data.artifact_max_bytes - used, |
There was a problem hiding this comment.
🟠 High harbor/taskset.py:262
Retrying cleanup can fail after a sidecar artifact was collected successfully: finalize counts the existing archive in used, then recollects it with the reduced artifact_max_bytes budget, so an archive larger than half the budget exceeds the retry limit. Make artifact collection idempotent by skipping already-recorded artifacts (or otherwise excluding them from the retry budget).
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/tasksets/harbor/taskset.py around line 262:
Retrying `cleanup` can fail after a sidecar artifact was collected successfully: `finalize` counts the existing archive in `used`, then recollects it with the reduced `artifact_max_bytes` budget, so an archive larger than half the budget exceeds the retry limit. Make artifact collection idempotent by skipping already-recorded artifacts (or otherwise excluding them from the retry budget).
1 participant
Extends Harbor Compose tasks to Prime and Modal VM sandboxes, using one project owner and the existing Docker execution machinery across local and cloud runs.
Depends on #2547, which builds on #2528.
The shared container runtime can execute its Docker CLI through an owned provider runtime. Process streams, signalling, detached jobs and bounded file reads use the same container implementation; binary transfers stage through the provider filesystem. Harbor supplies Compose templates, startup environment overrides and directory-transfer helpers. Prime and Modal adapters handle VM provisioning, Docker startup and confirmed termination.
Compose retains authored service networking. Prime requires
vm=True; Modal uses its VM runtime and publishes main's service port through its encrypted tunnel, including shared network namespaces. GPU tasks are unsupported, and Modal Compose requires public networking.Artifacts and collect hooks can target individual services. For separate grading, main is stopped after harness cleanup before collecting sidecar evidence. Collection enforces one configurable byte budget across services and rejects overlapping restore paths. The solver project or VM is removed before the fresh verifier starts.
Note
Run Harbor Compose tasks on Prime and Modal VMs
HarborComposeRuntimesupport for hosting a Docker Compose project inside a Prime or Modal VM. The VM runtime becomes the host;ContainerRuntimedelegates command execution, file I/O, and process management to that host, and binary data is staged through files on the provider filesystem.PrimeComposeVMinstalls Docker and launchesdockerd;ModalComposeVMuses a VM-backed sandbox with Docker-in-Docker on a private Unix socket.RuntimeProcesswith a non-blockingpoll, addsRuntime.serviceandRuntime.stop_servicedefaults, and adds aTask.cleanuphook invoked byRollout.closebefore runtime teardown.CollectHook.service,Artifact.service), collects main evidence in finalize and sidecar evidence in cleanup, and enforces a sharedartifact_max_bytesbudget across services.PrimeComposeVM.exposenow raisesSandboxError(Prime SDK does not support port exposure); GPU Compose is rejected universally; restricted networking is rejected for local Docker only;collectrejects duplicate or cross-service overlapping destinations; VM teardown now blocks up to 60 s waiting for confirmed sandbox termination;DockerRuntime.run_backgroundno longer dereferences an absent proxy under restricted networking.Macroscope summarized c1ce953.
Note
Medium Risk
Changes rollout teardown order, remote VM Docker bootstrap, and grading artifact collection; Prime Compose explicitly cannot expose ports, which may break tool servers that rely on
exposeon that path.Overview
Harbor Docker Compose tasks can run on local Docker, Prime VMs (
vm=true), or Modal’s experimental VM runtime (network_access=true), not only unrestricted local Docker. One Compose project still drives all services; remote runs stage the task environment into the VM, bootstrap Docker there, and routedocker compose/execthrough a shared host-backedContainerRuntime.The container stack gains
Runtime.service/stop_service, per-service views inHarborComposeRuntime, andpoll()on live processes. Host delegation fixes remote CLI I/O (staged binary read/write, longer process startup deadlines).Sidecar artifacts and collect hooks are supported: each entry’s
serviceselects the source; separate grading collects main infinalize, thentask.cleanup(new rollout hook) stops main and collects sidecars.artifact_max_bytescaps total collected bytes; overlapping paths across services are rejected.Docs drop the “sidecar unsupported” gap and describe Prime/Modal Compose constraints (no GPU Compose, Prime can’t publish VM ports externally, Modal tunnels main’s service port).
Reviewed by Cursor Bugbot for commit c1ce953. Bugbot is set up for automated code reviews on this repo. Configure here.