Skip to content

refactor(benchmark): share Docker checkpoint capture - #8838

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-shared-checkpoint-worker
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-shared-checkpoint-worker

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Behavior

  • Move the Docker writer barrier, prepared-source verification, and sealed checkpoint export into the shared benchmark runtime.
  • EdgeBench supplies its existing SForge capture operation and phase clock through the shared path. Other Docker-backed controllers can supply their own native capture operation and explicitly opt into an open solver phase.
  • Capture uses the original handoff and sealing deadlines. The worker is released before the sealed point is published.

Validation

  • Ran 137 focused Linux checks across the checkpoint controller, real Docker transport, storage and selection boundaries, failure diagnostics, and phase handling. Re-ran the diagnostic checks after updating their fault injection to use the new transport and the existing EdgeBench provider.
  • Exercised the same transport with a FrontierCS 2 bridge, a deterministic test evaluator, and two Docker solver containers. The receiver recovered the workspace, selected candidate, original evaluation ID, released feedback, and consumed evaluation time. An unresolved original evaluation left the source running.
  • Python lint and patch whitespace checks passed.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent / gpt-6.1-sol / OpenAI / runtime_reported / reasoning effort max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机

运行 Docker 求解器并准备把研究工作和原生评估进度交给接收端的 benchmark controller 操作者会遇到这个问题。主干只有 EdgeBench 的 controller 能冻结求解器并保存其文件和评估状态。此 PR 抽出共用入口,让其他 Docker controller 交入自己的评估回调,省去维护另一套取档和解冻流程。EdgeBench 仍能用原 phase clock 生成快照。显式开放 phase 的回调也能生成快照。调用者在捕获期间刷新同一个请求对象时,共用入口可以锁住原 worker 却保存另一个 worker。本次评审覆盖快照的选择、冻结、读取、解冻与封存。真实评测成绩、完整跨 benchmark 恢复和线上采用仍待独立核验。

改动思路

保留共用 Docker transport 及 typed artifact owner。EdgeBench 通过薄适配器提供 SForge 和原 phase clock,普通 controller 通过明确的回调提供原生胶囊。锁、容器选择和封存必须消费同一次输入。当前 PR 交付一条有实际 EdgeBench 调用的共用 transport。此 PR 的修复应绑定共享入口的请求快照,并用同一个公开入口的双容器测试验证。主干的契约继续要求在原时间窗内冻结所选 writer,并保存对应文件与原生机会。主干的 checkpoint 契约 · 共用入口说明

具体改动

  • edgebench-writer-window 已实现。隔离 Linux Docker 实跑保留了 EdgeBench 的原 phase、WAL 已提交字节和 writer 恢复,也读回了显式 open phase 下的原生胶囊。原生 callback 和 phase 仍由 EdgeBench 提供。EdgeBench 适配器
  • open-phase-opt-in 已实现。phase_deadline_path 为 null 时默认拒绝。显式开启后,回调在暂停期间执行,原生胶囊封存后仍能读回。真实 Docker 夹具
  • exact-worker 未满足。加锁后重新从调用者的可变请求读取容器 ID,见下方 P2。锁和容器选择

规范 benchmark/README.md @ f7e3f427197bab6c66ab535e26613a7f9c2fdbe0。

关键代码讲解

capture_worker 从私有文件解析请求,提供 SForge callback。validate_worker_request 校验全长 ID、组件和 phase,但把原 dict 交回。_worker_lock 用 daemon 和容器 ID 定位锁。capture_docker_worker 拿锁后重新读取 dict,再暂停 worker、保存文件、调用评估者并解冻。最终 manifest 从实际读取的容器记录 ID,并将暂存文件交给 typed artifact owner。输入校验 · 回执与封存

语义与 CI 对齐

核对了精确 base/head 的 16 个差异文件。合格 Node 22.22.3 与 SQLite 3.51.3、同一 Docker 镜像和 SForge 来源下,扩展套件基线 134 通过,现头 136 通过。两边共同的 scheduler 用例因精简 checkout 缺 examples 未通过,补齐各自精确提交的源文件后各自重跑通过。基线的 read_race 注入未接收 deadline 关键字,现头修正了测试注入。另有一次现头续跑时钟选回原 phase 的观察,后续完整套件及三次定向复跑通过,初次差异尚未归因。改动文件 Ruff 和 diff 检查通过。两轮双容器反例在实际 Docker 和 typed 文件封存上都复现。只在夹具中复制原请求,结果改为 A 的 ID 和字节,两个容器最终都处于 running。原 EdgeBench writer 夹具

对主干的风险

[P2] 把 writer 锁和封存绑定到入口选定的容器。 在隔离 Linux Docker 双容器复现。请求进入时指向 worker A。选择 A 的锁键后,调用方线程将同一个 dict 的 container_id 改为 B。代码持有 A 的锁,却暂停、复制并封存 B。密封 manifest 报 B,文件内容来自 B,integrity_verified=true,两个容器随后均为 running。入口先构造并验证本次调用独占的请求快照,包含容器 ID、组件和 handoff。用其中的同一 ID 做加锁、lookup 和校验,保留显式的 barrier 内组件回调。用双容器反例断言锁键、产物身份和字节全都属于 A。

我的整体评价

精确提交 2257b06430a4725cf7489decc678df901b3c2f02 需要修改。共用 transport 与现有 EdgeBench 适配器的切分合理。共享入口需保证原请求与 writer lock 一致,接收方才能依据此次封存判定来源。本次评审覆盖快照的选择、冻结、读取、解冻与封存。真实评测成绩、完整跨 benchmark 恢复和线上采用仍待独立核验。

English verdict: REQUEST_CHANGES | Exact head 2257b06430a4725cf7489decc678df901b3c2f02. The shared capture can hold worker A's lock and seal worker B's files with a successful integrity receipt. Bind the lock, lookup and seal to a single validated request snapshot.

try:
with exclusive_file_lock(_worker_lock(client, request["container_id"]), timeout_seconds=_remaining(deadline),
operation="checkpoint_worker_capture"):
container = client.containers.get(request["container_id"])

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 把 writer 锁和封存绑定到入口选定的容器。 在隔离 Linux Docker 双容器复现。请求进入时指向 worker A。选择 A 的锁键后,调用方线程将同一个 dict 的 container_id 改为 B。代码持有 A 的锁,却暂停、复制并封存 B。密封 manifest 报 B,文件内容来自 B,integrity_verified=true,两个容器随后均为 running。入口先构造并验证本次调用独占的请求快照,包含容器 ID、组件和 handoff。用其中的同一 ID 做加锁、lookup 和校验,保留显式的 barrier 内组件回调。用双容器反例断言锁键、产物身份和字节全都属于 A。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant