From e725e278532151ae3606c6af70bf1c60e6a0628b Mon Sep 17 00:00:00 2001 From: LoopX Agent <337587101+loopx-agent@users.noreply.github.com> Date: Mon, 16 Nov 2026 00:12:42 +0800 Subject: [PATCH] docs(rfc): update benchmark portfolio priority ranking and Bitter-Lesson boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add §2.2 summarizing the Bitter-Lesson raw-artifact reset boundary and five failure regimes requiring governed control planes, require a Raw-Artifact Reset Baseline under §5.2 Native baseline for episodic scalar/RSI tracks, add §3.8 with the 2026-11 benchmark portfolio priority matrix (P0/P1/P2/Control-Only) and platform constraints, update §17/§18 in EN/ZH mirrors without touching §6.4, and link the updated ranking from docs/research/README.md. Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com> --- ...n-harness-benchmark-research-program-v0.md | 248 +++++++++++++++++- ...ess-benchmark-research-program-v0.zh-CN.md | 201 +++++++++++++- docs/research/README.md | 6 + 3 files changed, 453 insertions(+), 2 deletions(-) diff --git a/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md b/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md index 61d7d40611..7102ce96e2 100644 --- a/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md +++ b/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md @@ -104,6 +104,109 @@ proved. Private trajectories and audit artifacts remain local evidence. Only their generalized typed contracts and public-safe conclusions may be promoted to the repository. +### 2.2 Bitter-Lesson boundary: raw-artifact resets versus governed control + +As frontier coding models improve at autonomous filesystem navigation and +self-authored inspection scripts, heavy hand-written context compression and +multi-layered outer prompts risk incurring net protocol tax ($\Delta J(B \mid +s_0) \le 0$). On **episodic, resettable, well-structured tasks with dense or +continuous isolated scalar verification**, a minimal outer loop that retains raw +prior artifacts and scalar scores across clean per-generation context resets—letting +the agent author its own Python/filesystem retrieval probes—can match or +outperform lossy hand-written summaries: + +- [Meta-Harness](https://arxiv.org/abs/2603.28052) shows an outer proposer reading + raw source code, scalar scores, and raw execution traces across clean resets + (median ~82 files read per iteration at ~10M tokens/iteration) outperforms + compressed-text optimizers (`+7.7` classification points with `4x` fewer + context tokens; `76.4%` on Terminal-Bench 2.0 with Opus 4.6). +- [Vercel `d0`](https://vercel.com/blog/we-removed-80-percent-of-our-agents-tools) + shows removing ~80% of custom tools (`17 -> 2`: `bash` + SQL) over a clean, + well-documented semantic filesystem improved success `80% -> 100%` while + cutting tokens `37%` and latency `3.5x`. +- [AlphaEvolve](https://arxiv.org/abs/2506.13131) succeeds on algorithmic code + discovery precisely by keeping a fixed immutable outer loop over deterministic + programmatic verifiers without letting the solver alter its own harness or + grading contract. +- Published LoopX context studies ([context delivery](https://loopx-project.github.io/loopx/blog/context-delivery/) + and [EdgeBench feedback and memory](https://loopx-project.github.io/loopx/blog/edgebench-feedback-and-memory/)) + likewise show that verbose serialization and post-peak replan loops can consume + substantial uncached context when raw-code inspection and local probes would + suffice. + +However, empirical evidence establishes **five failure regimes** where raw +artifacts, scalar rewards, and clean resets cannot replace a governed control +plane: + +1. **Goodharting and verifier tampering:** Whenever the solver shares process, + filesystem, network, or secret boundaries with the grader, unguarded + optimization exploits shortcut channels—observed in `LHTB 2.0` (`14/17` early + perfect scores came from grader leaks prior to process freezing and verifier + isolation), the [Reward Hacking Benchmark](https://arxiv.org/abs/2605.02964) + (`0%` to `13.9%` exploit rates across 13 frontier models, `72%` with explicit + chain-of-thought rationalizations, and alignment breakdown past intractability + thresholds), [SWE-Marathon](https://arxiv.org/abs/2606.07682) (`13.8%` + rollout reward hacking), [Darwin Gödel Machine](https://arxiv.org/abs/2505.22954) + (Node 114 falsifying verification logs), and + [METR reward-hacking evaluations](https://metr.org/blog/2025-06-05-recent-reward-hacking/). + In-agent Python probes cannot enforce OS/container verifier isolation or + `benchmark_integrity_audit_v0` (§6.5, §7.1). +2. **Out-of-distribution (OOD) overfitting and harness bloat:** Unregularized + evolution over raw artifacts memorizes training traces and degrades OOD. + [RRSI](https://arxiv.org/abs/2609.24972) finds unregularized Meta-Harness + gains only `+0.9` OOD (`40.6` vs `39.7`) and `TTHE` drops `-1.7` (`38.0` with + an `8,500`-word bloated prompt), whereas critic-pruned, budget-annealed + regularization lifts OOD to `43.6` (`+3.9`) with `30%` fewer policy tokens; + [Harness Updating Is Not Harness Benefit](https://arxiv.org/abs/2605.30621) + similarly shows harness edits mainly lift mid-tier models while saturating or + regressing frontier models without OOD gating. Sequential validity, held-out + splits, and decision-preserving rate-distortion memory are governed alongside + this program in the [Frontier Science Research Program](./frontier-science-research-program-v0.md#2-problem-and-motivation) + (§2 and §5 `T01`–`T07`). +3. **Premature termination and sparse or binary terminal verifiers:** Fresh + retrieval over local files cannot stop an agent from declaring completion as + soon as local smoke fixtures pass, nor extract dense guidance from `0/1` + terminal rewards. In LoopX's published `SWE-Marathon` study + ([`benchmark/swe-marathon/README.md`](../../../benchmark/swe-marathon/README.md)), + on `zstd-decoder`, `plain` Codex stopped after visible fixtures (`0.721` + partial, reward `0`) whereas governed `heartbeat` continued to protocol + self-tests and memory-safety verification (`1.000`, reward `1`); on `LHTB` + `covshift` ([EdgeBench long-horizon repairs](https://loopx-project.github.io/loopx/blog/edgebench-long-horizon-repairs/)), + default completion exited at minute 38 (`0.000`) while an owner-locked + acceptance gate forced completion to `0.82` (five-arm `heartbeat` `0.743`). + External completion rejection, `stopped_recovery.py`, and hidden-coverage + checks remain necessary. +4. **Proxy misalignment and tail TLE cliffs:** Uncalibrated local yardsticks and + aggregate averages invert native rankings or trigger timeout cliffs. LoopX's + [`benchmark/proxy_order.py`](../../../benchmark/proxy_order.py) analysis + ([behavior findings](../../reference/benchmark-behavior-findings.md#compare-a-local-proxy-with-released-scores)) + shows local proxies reversing native acceptance ordering; + [DuoDiscover](https://arxiv.org/abs/2610.17184) shows adding uncalibrated + yardsticks *degraded* performance (`12.85x / 69.12%` to `10.33x / 66.91%`, + with `41%`–`45%` of yardsticks invalidating across code edits) until explicit + fidelity gating lifted outcomes to `14.93x / 80.80%`; and + [LangChain's Terminal-Bench 2.0 study](https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering) + showed uniform `xhigh` reasoning dropped accuracy (`63.6% -> 53.9%`) via + timeout cliffs without phase-aware effort control. Governed yardstick + calibration and tail-latency pressure gating remain necessary safeguards around + native optimization. +5. **Stateful dependency-DAG, coverage, and workspace/settlement invariants:** + Multi-unit dependency frontiers, non-episodic environments, linked-worktree + patch normalization, and idempotent settlement cannot be repaired by solver + scripts inside the container. [LoopsBench](https://arxiv.org/abs/2608.00267) + (`112` tasks, `>5,300` dependency-DAG units) holds even top loop-enabled SOTA + at `25.00%`; on `EdgeBench` `New Foundations Consistency`, an explicit + declaration-coverage ledger broke an early `91.9412` plateau up to `92.3419` + ([EdgeBench feedback and memory](https://loopx-project.github.io/loopx/blog/edgebench-feedback-and-memory/)); + and real benchmark infrastructure requires host-owned invariants such as + [`workspace_delivery.py`](../../../benchmark/deepswe-gptxhigh-v1/workspace_delivery.py) + (normalizing linked-worktree diffs back to `/app`), + [`frontiercs2/README.md`](../../../benchmark/frontiercs2/README.md) (`v2.patch` + tmpfs `docker exec` streaming and durable admission recovery), and + [`widesearch/pier/README.md`](../../../benchmark/widesearch/pier/README.md) + (`$HOME` bind mounts, numeric `reward.json` validation, and + `provider_credential_shell_excluded`). + ## 3. Benchmark Primer and Fit Analysis Benchmark counts and leaderboard values in this section are an informative @@ -364,6 +467,86 @@ The six must still be reported in their native metric spaces; StartupBench rubrics, LoopsBench verifier tiers, and WideSearch Item F1 must not be averaged into one number or into the ALE/LHTB/DeepSWE results. +### 3.8 External Benchmark Survey and Priority Ranking (2026-11) + +This 2026-11-15 audit updates the August 2026 six-benchmark primer (§3.1–§3.7) +and guides integration sequencing (§9, §11) across EdgeBench and 20 external +benchmarks. Rankings weigh four empirical criteria: (1) remaining frontier +headroom against pure base-model scaling, (2) verifier determinism, resistance to +Goodharting, and availability of continuous or multi-unit progress signals, (3) +direct stress on the five governed failure regimes (§2.2), and (4) repository +readiness (`benchmark/` holds `52,031` `.py` LOC across `177` files: `27,338` +tests, `9,082` `runtime/`, `6,899` `edgebench/`, `4,433` `deepswe*`, `1,430` +`swe-marathon/`, `1,285` `frontiercs2*`, `596` `LHTB/`, `592` `widesearch/`, +`376` root helpers) versus Linux x86\_64 Docker, GPU, and cloud requirements. + +**Shared prerequisite before scaling multi-arm studies:** Scaling new Harbor, +cloud, or GPU benchmark runs is gated on converging adapters onto the unified +`benchmark/runtime/` Turn/Harbor entrypoint with matching native Stop-hook +baseline parity and clean E0–E2 lifecycle, integrity, and countability gates +(§11.2), rather than cloning bespoke per-benchmark runners. + +#### Priority Matrix + +| Tier | Benchmark | Tasks & Horizon | Reward / Verifier Shape | Frontier Headroom & Regime Signal | Engineering Readiness & Platform Constraints | +|---|---|---|---|---|---| +| **P0** | **EdgeBench** | Continuous algorithmic optimization (`12h–18h` runs) | Deterministic online cooldown judge; scalar task metrics (higher/lower by task) | Strong Iso-Budget Pareto separation (`New Foundations 92.3419`, `ANN 7,159 QPS`, while strong Stop-hook `Official` still leads on `Portfolio 53.87`, `Grid 1,791`) | **Ready & evaluated** (`edgebench/` `6,899` LOC); requires **Linux x86\_64 Docker + SForge `iptables`** (macOS Docker socket lacks network boundary) | +| **P0** | **FrontierCS-2** | `48` public / `150` total (`13` domains); `5h`, `3–100` capped train calls | Deterministic hidden-workload Driver/Judge; binary `PASS` (must beat reference on *every* metric/workload) + `FCI`/`FDI` scalar | Top of 12 evaluated agents passes **`58%`**, **`16/48` at `0%`**; directly tests proxy calibration and multi-metric resistance | **Pilot bridge ready** (`frontiercs2*` `1,285` LOC, `v2.patch` transport repair; frozen v1 `batch_local_counting` archive is `score_countable=false`, `integrity_qualified=false`; `0` countable LoopX outcomes yet); **Linux x86\_64 Docker required** (macOS ARM Rosetta/QEMU breaks seccomp); EKS+Modal eval `~$1,400`/48 tasks | +| **P0** | **LHTB / LHTB 2.0** | `v1: 46` (`90m`); `v2.0: 77` (`46` orig + `31` reconstruction, `3h` w/ `30m` ckpts) | Continuous scalar `[0,1]` + 6-stage ladder (`R>=0.95`/`1.0`) + frozen-process Harbor binary rejection loop | `v1` LoopX `0.4948` vs Goal `0.4475` / Plain `0.4218` (unequal budgets); `v2.0` top `0.692` (`37/77`); `14/17` early perfect leaks proved verifier-isolation need | **`v1` Harbor adapter ready** (`LHTB/` `596` LOC); **`LHTB 2.0` has `0` LOC** in repo; non-GPU Docker / Daytona | +| **P0** | **LoopsBench** | `112` dependency-DAG tasks (`>5,300` units, `8` langs) | Deterministic unit + regression tests along ready DAG frontier (`incomplete` / `partial` / `complete`) | Top loop-enabled SOTA **`25.00%`** (`Opus-4.7` + Claude Code + outer loop); built specifically for outer-loop DAG & regression invariants | **`0` active code** in `benchmark/` (planned in §3.5/§9.5); non-GPU Linux x86\_64 Docker preferred | +| **P1** | **SWE-Marathon** | `20` ultra-long tasks (`15`-task v1.1 subset in-repo); `avg 27.2M tok`/run | Multi-layer adversarial verifier; binary solve + continuous `partial_score` | `v1` `<30%` frontier solve, **`13.8%` rollout reward hacking**; in-repo `30%` timeout `0.333 (5/15)` HB vs `0.267 (4/15)` Plain/Goal | **Ready & evaluated** (`swe-marathon/` `1,430` LOC); `amd64` multi-lang Docker; high token & wall-time cost (`1,592m–2,375m` per 15-task arm) | +| **P1** | **PaperBenchX** *(preferred over PaperBench)* | `93` papers (`12` areas / `6` non-ML sciences, `>3,000` criteria); `4–24h` (med `7h`) | Clean-from-scratch replay + deterministic checks & shared LLM-Judge (`>=90%` full pass) | Mean rubric `27.53–59.08%`, **`14%` full reproduction** (`>=90%`) — sharper validation gap & broader stacks than `PaperBench` (`20` ICML'24 papers, `24–36h`, `26.0%` SOTA) | **`0` code** in repo (`~2–4 wk` lift); heavy Linux Docker + domain stacks (`HFSS`, `Meep`, `GPU4PySCF`, `Habitat`) + Judge API secret isolation | +| **P1** | **ALE (Agents' Last Exam)** *(distinct from `ALE-Bench`)* | `1,500+` taxonomy (`v1.1: 122`: `119` CPU / `3` GPU, `78` Docker); multi-hour VMs | Post-run hidden reference `evaluate()`, scalar `[0,1]` | `<1%` full pass on hardest tier; tests cross-surface CLI+GUI professional transfer | **`0` active code** in repo (planned §3.1/§9.1; `~2–4 wk` lift); hard **GCP/GCE VM + CUA MCP** dependency (not locally runnable on macOS) | +| **P1** | **WideSearch** | `200` (`100 EN + 100 ZH`, `18` industries); `44+` pages / `~2.3h` human time | Automated atomic table grader (`Item F1` / `Row F1` + binary `SR`); non-deterministic live web drift | Frontier `SR <= 5.1%` (`o3` multi-agent `5.1%`, single-agent `4.5%`, `Gemini 2.5 Pro 4.3%`); high fit for completeness/fan-out, low–med for iterative RSI | **Adapter + `pier` templates ready** (`widesearch/` `592` LOC, `0` executed runs); requires Linux/cloud `$HOME` bind mounts (macOS `/tmp` fails Colima) & `permitted_solving` network | +| **P1** | **KernelBench / AlgoTune** | `KernelBench`: `250` PyTorch/CUDA ops (`L1–L3`+`L4`); `AlgoTune`: `154` SciPy/sklearn/CVXPY funcs | Deterministic correctness + GPU (`fast_p`) or CPU geom-mean speedup | `KernelBench` `<20% fast_1`; `AlgoTune` `AlgoTuner 1.72x` (`~2.05x` frontier); pure objective speedup for evolutionary/RLVR loops | **`0` code** in repo; `KernelBench` requires **NVIDIA CUDA/Triton GPU Linux/Modal** (impossible on Mac); `AlgoTune` requires **pinned multi-core Linux x86\_64** timing host | +| **P2** | **ALE-Bench (SakanaAI)** *(distinct from `ALE`)* | `40` AtCoder Heuristic Contest tasks (`C++/Rust/Py`, `300+` cases); `150`-step loops | Deterministic relative heuristic score | Strong 1-shot vs iterative separation; trails human consistency | **`0` active code** in repo; heavy multi-core Linux/AWS Docker; upstream rules bar bundling private seeds/inputs | +| **P2** | **StartupBench** | `97` deliverable tasks (`DOCX/XLSX/PPTX/PDF`, `6` domains, `25.3` rubrics/task) | Evidence views + weighted Agent-as-Judge `0–100` (`pass >= 90`, `92%` human agr.) | Top avg `Kimi-k3 73.67`, top `pass@1` `GPT-5.6-sol 31.27%`; low–med RSI fit (open-ended deliverable rubric) | **`0` active code** in repo (planned §3.4/§9.4); multi-turn office-artifact API + Judge boundary; non-GPU | +| **P2** | **MLE-bench** | `75` Kaggle tasks (`22` Lite `158 GB`, `3.3 TB` total); `24h` std (LB paused `2026-04`) | Deterministic Kaggle metric + medal thresholds | Main SOTA `64.44±1.18%`; contamination risk on older Kaggle comps | **`0` active code** in repo; prohibitive infra (`36 vCPU`, `440 GB RAM`, `24 GB A10 GPU`, `3.3 TB` disk, Kaggle auth/LFS) | +| **P2** | **tb-science** *(episodic transfer samples only; **excluded from RSI**)* | `96` merged tasks + `366` open PRs; Harbor format | **100% binary `1/0` `reward.txt`** (`public` agent vs `no-network` verifier) | High frontier difficulty on selected PRs, but **unsuitable for RSI** (zero per-gen gradient, no train/held-out split, CI-locked single horizon) | **`0` code** in repo (`~1–2 wk` Harbor lift); `ci_checks/check-task-timeout.sh` hard-locks **`28800s` (`8h`)**; Linux x86\_64 Docker preferred | +| **Control-Only / Deprioritized** | **DeepSWE** | `113` original tasks (`91` repos, `5` langs) | Hand-written behavioral verifiers (`0/1` binary, `1.4%` judge disagreement) | `v1.1` **approaching saturation around a `74%` three-way frontier tie**; retain only as sanity/regression control | **Ready** (`4,433` LOC w/ `workspace_delivery.py`); Linux Docker host worker required; historical v1 arms unequal-budget | +| **Control-Only / Deprioritized** | **SWE-bench Pro** | `v1: 1,865` (`731` pub); `v2: 642` + `HARD-51` | Patch + unit tests (`0/1` binary; `32.4%` judge disagreement) | **`HARD-51` up to `98.0%`** — saturating under base-model scaling | **`0` active code** in repo; `linux/amd64` Docker images (`ghcr.io`), non-native on Mac ARM | +| **Control-Only / Deprioritized** | **CORE-Bench** | `270` (`90` papers, `3` diffs); via HAL | Output-question binary match | Rose `21% -> 95%` (Opus 4.5 after Anthropic fixed tolerances/scaffold bugs) — saturated | **`0` code** in repo; Docker/Azure (`Python`/`R`) | +| **Control-Only / Deprioritized** | **ScienceAgentBench** | `102` (`44` papers); `30m` Docker runs | Valid Execution + rubric/code check | `32.4%–42.2%` (`o1-preview`); single-script scope limits long-horizon control separation | **`0` code** in repo; `30m` / 8-thread Docker | +| **Control-Only / Deprioritized** | **BixBench** | `60` capsules / `205` Qs (`v1.5`); `24–48h` 5-rep | Binary exact-match MCQ / LLM-Judge open answer | `17%` open-answer (older baselines, lacks 2026 SOTA); high 5-replica binary cost | **`0` code** in repo; high multi-replica API cost | +| **Control-Only / Deprioritized** | **FrontierScience** | `160` QA items (`100` Olympiad + `60` Research) | Short-answer + 10-pt rubric | `GPT-5.2 77%` Olympiad / `25%` Research; no interactive repo/sandbox loop | **`0` code** in repo; API reasoning only | +| **Control-Only / Deprioritized** | **RE-Bench** *(deprecated)* | `7` ML R&D tasks (`2h–32h`) | Deterministic ML eval (clock/memory exploit history) | **Deprecated upstream** (`METR/RE-Bench`); superseded by `FrontierCS-2`, `PaperBenchX`, and `KernelBench` | **`0` code** in repo; Linux x86\_64 + NVIDIA CUDA GPU required | + +#### Key Operational Disambiguations and Exclusions + +1. **`ALE` versus `SakanaAI/ALE-Bench` (and non-existent `"ALE Science"`):** + No public benchmark or track named `"ALE Science"` exists. `ALE` + ([`rdi-berkeley/agents-last-exam`](https://github.com/rdi-berkeley/agents-last-exam)) + is a broad professional CLI+GUI suite containing science subdomains, whereas + `ALE-Bench` ([`SakanaAI/ALE-Bench`](https://github.com/SakanaAI/ALE-Bench)) + covers AtCoder Heuristic Contest algorithmic optimization. They are ranked + separately above (`P1` vs `P2`). +2. **Why `tb-science` is excluded from RSI and retained only for episodic transfer:** + Following the `laude-institute` to `harbor-framework` migration, + [`harbor-framework/terminal-bench-science`](https://github.com/harbor-framework/terminal-bench-science) + is a scientific suite separate from `terminal-bench-2` (`89` general tasks at + `900s` default timeout). Despite `96` merged tasks and `366` open PRs, + `tb-science` is **not suited for iterative RSI loops**: CI + (`ci_checks/check-task-timeout.sh`) hard-locks `[agent] timeout_sec == 28800.0` + (`8h`), `reward.txt` is **100% binary `1/0`** (even when internal scorers compute + continuous metrics), there is no train/held-out split, `[agent] network_mode` is + `public`, and `rubrics/task-implementation.toml` rejects open-ended + hypothesis tasks. Retain only hard episodic transfer samples (e.g., Lean + formalization `#2530`, `#2197`, `#2653`, `#2629`, biology `#1518`, physics + `#2804`, or hosted GPU control `#2531`). +3. **`PaperBenchX` over legacy `PaperBench`, and `DeepSWE` terminology:** + `PaperBenchX` (`93` cross-discipline non-ML science papers with clean replay) + replaces legacy `PaperBench` (`20` ICML'24 ML papers) as the primary `P1` + reproduction target. `DeepSWE` here designates Datacurve's `113`-task + benchmark ([`datacurve-ai/deep-swe`](https://deepswe.datacurve.ai/)), + distinct from Agentica's `DeepSWE-Preview` model; its saturation near `74%` + shifts new `C2`/`C3` causal investment toward `P0`/`P1` benchmarks. +4. **Platform execution constraints:** Do not run score-countable benchmarks for Linux-seccomp, + `iptables`, CUDA, or multi-core timing suites (`EdgeBench`, `FrontierCS-2`, + `KernelBench`, `AlgoTune`, `SWE-bench Pro`, `tb-science`) on macOS ARM64 + Rosetta/QEMU emulation; run them on pinned Linux x86\_64 (and dedicated CUDA + or cloud hosts where required), and avoid macOS `/tmp` bind mounts on Colima + (`widesearch/pier/README.md`). + ## 4. Claim Ladder Every public result must name the highest claim level it supports. @@ -421,7 +604,14 @@ family” are not identity. The program distinguishes four arm types: 1. **Native baseline**: the benchmark's supported reference scaffold or a - declared CLI harness with no LoopX control. + declared CLI harness with no LoopX control. On **episodic RSI and scalar + optimization tracks**, Native baseline evaluations must additionally account + for or include a zero-protocol-tax **Raw-Artifact Reset Baseline** (`all prior + artifacts + scalar scores` across clean per-generation context resets, using + only the model's native filesystem/Python retrieval without LoopX packets, + ledgers, or outer control overhead), so any **Governed LoopX** claim proves + causal uplift beyond self-authored retrieval rather than winning against + strawman lossy context compression (§2.2). 2. **Passive LoopX**: identical worker decisions and benchmark contract, with read-only LoopX observation and compact settlement. 3. **Governed LoopX**: a declared LoopX profile may affect continuation, @@ -1599,6 +1789,7 @@ This is a living research RFC, not a frozen benchmark snapshot. | 2026-08-23 | Add an engineering readiness ladder and bounded delivery-slice rules so repository construction is tracked separately from C0--C4 research claims. | | 2026-08-25 | Add StartupBench, LoopsBench, and WideSearch to the benchmark primer and integration plan as a second, complementary trio; require native metrics per benchmark and keep benchmark material out of reusable LoopX state. | | 2026-10-16 | Archive [feedback-first observations](https://github.com/loopx-project/loopx/blob/1d60e7edb2a05fe048767444f872934c3adef992/deprecate/benchmark-legacy/edgebench/feedback-first-observations-20261016.md) separately from active practice. Runner source-subset and configuration-projection corrections remain E0/E1 work; mixed-profile, interrupted and running observations do not close E2/E3 or isolate a planning effect. General guard re-entry relaxation is deferred pending affected-caller validation. | +| 2026-11-15 | Codify the Bitter-Lesson boundary and five governed control-plane failure regimes (§2.2), require a zero-protocol-tax Raw-Artifact Reset Baseline on episodic RSI / scalar optimization tracks (§5.2), and adopt the 2026-11 benchmark portfolio ranking (§3.8): **P0** (`EdgeBench`, `FrontierCS-2`, `LHTB / LHTB 2.0`, `LoopsBench`), **P1** (`SWE-Marathon`, `PaperBenchX` over `PaperBench`, `ALE` distinct from `ALE-Bench`, `WideSearch`, `KernelBench` / `AlgoTune`), **P2** (`ALE-Bench`, `StartupBench`, `MLE-bench`, `tb-science` episodic transfer samples only—excluded from RSI due to CI-locked `28800s` budget and 100% binary `reward.txt`), and **Control-Only / Deprioritized** (`DeepSWE`, `SWE-bench Pro`, `CORE-Bench`, `ScienceAgentBench`, `BixBench`, `FrontierScience`, deprecated `RE-Bench`), gating multi-arm expansion on unified `benchmark/runtime/` + Stop-hook E0–E2 readiness and Linux x86\_64 / GPU hosts. | ## 18. References @@ -1607,11 +1798,13 @@ This is a living research RFC, not a frozen benchmark snapshot. - [Agents' Last Exam evaluation framework](https://github.com/rdi-berkeley/agents-last-exam) - [Long-Horizon Terminal Benchmark repository](https://github.com/zli12321/LHTB) - [LHTB benchmark report](https://zli12321.github.io/LHTB/index.html) +- [LHTB paper](https://arxiv.org/abs/2607.08964) - [DeepSWE project and leaderboard](https://deepswe.datacurve.ai/) - [DeepSWE paper](https://arxiv.org/abs/2607.07946) - [DeepSWE run guide](https://deepswe.datacurve.ai/run) - [StartupBench project](https://startupbench.github.io/) - [StartupBench paper](https://startupbench.github.io/assets/StartupBench.pdf) +- [StartupBench arXiv](https://arxiv.org/abs/2608.17800) - [LoopsBench project](https://loopsbench.ai/) - [LoopsBench repository](https://github.com/microsoft/Loopsbench) - [LoopsBench paper](https://arxiv.org/abs/2608.00267) @@ -1620,6 +1813,59 @@ This is a living research RFC, not a frozen benchmark snapshot. - [WideSearch repository](https://github.com/ByteDance-Seed/WideSearch) - [WideSearch paper](https://arxiv.org/abs/2508.07999) - [WideSearch dataset](https://huggingface.co/datasets/ByteDance-Seed/WideSearch) +- [EdgeBench repository](https://github.com/ByteDance-Seed/EdgeBench) +- [FrontierCS-2 project](https://frontier-cs.org/) +- [FrontierCS-2 repository](https://github.com/FrontierCS/FrontierCS-2) +- [FrontierCS-2 paper](https://arxiv.org/abs/2610.29932) +- [SWE-Marathon repository](https://github.com/abundant-ai/swe-marathon) +- [SWE-Marathon paper](https://arxiv.org/abs/2606.07682) +- [PaperBenchX repository](https://github.com/UniPat-AI/PaperBenchX) +- [PaperBenchX paper](https://arxiv.org/abs/2610.39744) +- [PaperBench reference implementation](https://github.com/openai/frontier-evals/tree/main/project/paperbench) +- [PaperBench paper](https://arxiv.org/abs/2504.01848) +- [KernelBench repository](https://github.com/ScalingIntelligence/KernelBench) +- [KernelBench paper](https://arxiv.org/abs/2502.10517) +- [AlgoTune repository](https://github.com/oripress/AlgoTune) +- [AlgoTune paper](https://arxiv.org/abs/2507.15887) +- [ALE-Bench (SakanaAI) repository](https://github.com/SakanaAI/ALE-Bench) +- [ALE-Bench paper](https://arxiv.org/abs/2506.09050) +- [MLE-bench repository](https://github.com/openai/mle-bench) +- [MLE-bench paper](https://arxiv.org/abs/2410.07095) +- [Terminal-Bench Science project](https://www.terminal-bench-science.ai/) +- [Terminal-Bench Science repository](https://github.com/harbor-framework/terminal-bench-science) +- [SWE-bench Pro repository](https://github.com/scaleapi/SWE-bench_Pro-os) +- [SWE-bench Pro paper](https://arxiv.org/abs/2509.16941) +- [CORE-Bench repository](https://github.com/siegelz/core-bench) +- [CORE-Bench paper](https://arxiv.org/abs/2409.11363) +- [ScienceAgentBench repository](https://github.com/OSU-NLP-Group/ScienceAgentBench) +- [ScienceAgentBench paper](https://arxiv.org/abs/2410.05080) +- [BixBench repository](https://github.com/Future-House/BixBench) +- [BixBench paper](https://arxiv.org/abs/2503.00096) +- [RE-Bench (deprecated) repository](https://github.com/METR/RE-Bench) +- [RE-Bench paper](https://arxiv.org/abs/2411.15114) +- [Meta-Harness paper](https://arxiv.org/abs/2603.28052) +- [Meta-Harness repository](https://github.com/stanford-iris-lab/meta-harness) +- [RRSI paper](https://arxiv.org/abs/2609.24972) +- [RRSI repository](https://github.com/google-research/rrsi) +- [Harness Updating Is Not Harness Benefit](https://arxiv.org/abs/2605.30621) +- [Agentic Harness Engineering (AHE)](https://arxiv.org/abs/2604.25850) +- [Self-Harness](https://arxiv.org/abs/2606.09498) +- [Reward Hacking Benchmark (RHB)](https://arxiv.org/abs/2605.02964) +- [DuoDiscover paper](https://arxiv.org/abs/2610.17184) +- [AlphaEvolve paper](https://arxiv.org/abs/2506.13131) +- [Darwin Gödel Machine (DGM) paper](https://arxiv.org/abs/2505.22954) +- [Darwin Gödel Machine repository](https://github.com/jennyzzt/dgm) +- [LangChain: Improving Deep Agents with harness engineering](https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering) +- [Vercel `d0`: We removed 80% of our agent's tools](https://vercel.com/blog/we-removed-80-percent-of-our-agents-tools) +- [METR: Recent frontier models are reward hacking](https://metr.org/blog/2025-06-05-recent-reward-hacking/) +- [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) +- [Anthropic: Effective harnesses for long-running agents](https://www.anthropic.com/research/effective-harnesses-for-long-running-agents) +- [Anthropic: Harness design for long-running application development](https://www.anthropic.com/research/harness-design-long-running-apps) +- [LoopX: Context delivery study](https://loopx-project.github.io/loopx/blog/context-delivery/) +- [LoopX: EdgeBench feedback and memory study](https://loopx-project.github.io/loopx/blog/edgebench-feedback-and-memory/) +- [LoopX: EdgeBench long-horizon repairs](https://loopx-project.github.io/loopx/blog/edgebench-long-horizon-repairs/) +- [Frontier Science Research Program v0](./frontier-science-research-program-v0.md) +- [Exploratory benchmark behavior findings](../../reference/benchmark-behavior-findings.md) - [Benchmark research workspace](https://github.com/huangruiteng/loopx/blob/main/benchmark/README.md) - [DeepSWE research practice](https://github.com/huangruiteng/loopx/blob/main/benchmark/deepswe/README.md) - [Legacy benchmark history](https://github.com/loopx-project/loopx/tree/54d42da44283bb414b24f71d16642c25cbcaf5dd/deprecate/benchmark-legacy) diff --git a/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md b/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md index 6510e90498..d5b572f121 100644 --- a/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md +++ b/docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md @@ -84,6 +84,74 @@ validation 后才能 accountable settlement、精简 run reduction,以及 trea 无法证明时显式判无效。私有 trajectory 与 audit artifact 只保留为本地证据;只有泛化后的 typed contract 与 public-safe 结论可以进入仓库。 +### 2.2 Bitter-Lesson 边界:原始产物重置与受治理控制面 + +随着前沿编码模型在自主文件系统检索与自编探针脚本上持续增强,厚重的手工上下文压缩与多层 +外置提示词容易带来净协议税($\Delta J(B \mid s_0) \le 0$)。在**可重置的 episode 型、结构清晰、 +且具备稠密或连续隔离标量验证的任务**上,仅保留原始历史产物与标量分数、并在每轮间执行干净 +上下文重置(让 Agent 自行编写 Python / 文件系统检索脚本)的极简外循环,可以打平甚至超越 +有损手工摘要: + +- [Meta-Harness](https://arxiv.org/abs/2603.28052) 表明,在干净上下文重置下直接读取原始 + 源码、标量分数与原始执行轨迹的外层 proposer(每轮中位数读取约 82 个文件、消耗约 10M + tokens)优于压缩文本优化器(分类准确率 `+7.7` 分且上下文 token 少 `4x`;在 Terminal-Bench 2.0 + 上使用 Opus 4.6 达到 `76.4%`)。 +- [Vercel `d0`](https://vercel.com/blog/we-removed-80-percent-of-our-agents-tools) 表明, + 在结构清晰的语义文件系统上删去约 80% 自定义工具(`17 -> 2`:`bash` 与 SQL),使成功率从 + `80%` 升至 `100%`,同时减少 `37%` token 并将延迟降低 `3.5x`。 +- [AlphaEvolve](https://arxiv.org/abs/2506.13131) 在算法代码发现上的成功,恰恰依赖固定不变的 + 外层循环与确定性程序 verifier,而不允许求解器改写自身 harness 或评分契约。 +- LoopX 已发布的上下文研究([上下文交付研究](https://loopx-project.github.io/loopx/blog/context-delivery/) + 与 [EdgeBench 反馈与记忆研究](https://loopx-project.github.io/loopx/blog/edgebench-feedback-and-memory/)) + 同样表明,冗余序列化与见顶后的重复 replan 循环会在原始代码检查与本地探针即可解决的场景下消耗大量 + uncached 上下文。 + +然而,实证研究确立了**五类失败机制**,证明原始产物、标量奖励与干净重置无法替代受治理控制面: + +1. **Goodhart 与 verifier 篡改:** 当求解器与评分器共享进程、文件系统、网络或密钥边界时, + 缺乏防护的优化会利用捷径通道——实证见于 `LHTB 2.0`(在冻结 agent 进程并隔离 verifier 之前, + 早期 `14/17` 个满分属于评分器泄漏)、[Reward Hacking Benchmark](https://arxiv.org/abs/2605.02964) + (13 个前沿模型利用漏洞比例为 `0%` 至 `13.9%`,其中 `72%` 带有显式思维链合理化,且在任务复杂度 + 超过诚实求解阈值后生产对齐保护失效)、[SWE-Marathon](https://arxiv.org/abs/2606.07682) + (观测到 `13.8%` rollout 存在 reward hacking)、[Darwin Gödel Machine](https://arxiv.org/abs/2505.22954) + (Node 114 伪造验证日志),以及 [METR reward-hacking 评估](https://metr.org/blog/2025-06-05-recent-reward-hacking/)。 + Agent 容器内的 Python 探针无法提供 OS/容器级 verifier 隔离与 `benchmark_integrity_audit_v0`(§6.5、§7.1)。 +2. **分布外(OOD)过拟合与 harness 膨胀:** 在原始产物上做无正则进化会死记训练轨迹并导致 OOD 退化。 + [RRSI](https://arxiv.org/abs/2609.24972) 发现未正则的 Meta-Harness 在 OOD 仅提升 `+0.9` + (`40.6` vs `39.7`),`TTHE` 甚至下降 `-1.7`(`38.0` 并膨胀出 `8,500` 词提示词),而加入 critic + 剪枝与退火预算正则化后 OOD 升至 `43.6`(`+3.9`)且策略 token 减少 `30%`; + [Harness Updating Is Not Harness Benefit](https://arxiv.org/abs/2605.30621) 同样显示 harness 修改 + 主要提升中等模型,在缺少 OOD 门控时前沿模型会饱和或回退。连续有效统计推断、留出集隔离及保留 + 决策差异的率失真记忆由本计划协同 [前沿科学研究计划](./frontier-science-research-program-v0.zh-CN.md#2-问题与动机) + (第 2 节与第 5 节 `T01`–`T07`)共同治理。 +3. **过早终止与稀疏 / 二值终局 verifier:** 本地文件重读无法阻止模型在可见冒烟测试通过后立即宣布完成, + 也无法从 `0/1` 终局奖励中提取中间梯度。在 LoopX 已发布的 `SWE-Marathon` 研究 + ([`benchmark/swe-marathon/README.md`](../../../benchmark/swe-marathon/README.md))中, + `zstd-decoder` 任务上 `plain` Codex 在可见测试通过后早停(partial `0.721`,reward `0`),而受控 + `heartbeat` 持续推进协议自测与内存安全验证(partial `1.000`,reward `1`);在 `LHTB` `covshift` + ([EdgeBench 长程修复研究](https://loopx-project.github.io/loopx/blog/edgebench-long-horizon-repairs/))中, + 默认完成判断在第 38 分钟退出(`0.000`),而由 owner 锁定的验收门控将其推至 `0.82`(五臂 `heartbeat` 为 + `0.743`)。外部完成拒绝、`stopped_recovery.py` 与隐藏覆盖压力仍然不可或缺。 +4. **代理指标偏移(Proxy Misalignment)与尾部 TLE 悬崖:** 未校准的本地 yardstick 与均值指标会倒挂 + 原生排名或触碰超时悬崖。LoopX 的 [`benchmark/proxy_order.py`](../../../benchmark/proxy_order.py) 分析 + ([行为发现说明](../../reference/benchmark-behavior-findings.md#compare-a-local-proxy-with-released-scores)) + 展示了本地代理排序与原生接受顺序的反转;[DuoDiscover](https://arxiv.org/abs/2610.17184) 表明直接加入 + 未校准代理指标反而使成绩*下降*(`12.85x / 69.12%` 降至 `10.33x / 66.91%`,代码演进后 `41%`–`45%` + 的 yardstick 失效),直至加入保真度门控才提升至 `14.93x / 80.80%`;而 + [LangChain 的 Terminal-Bench 2.0 研究](https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering) + 显示,在缺乏分阶段推理深度控制时,统一使用 `xhigh` 推理深度因超时悬崖从 `63.6%` 跌至 `53.9%`。 + 受治理的 yardstick 校准与尾延迟压力门控是原生优化的必要保护。 +5. **有状态依赖 DAG、覆盖台账与 workspace / 结算不变量:** 多单元依赖前沿、非 episode 持续世界、 + linked-worktree 补丁归一化与幂等结算无法靠容器内的 Agent 脚本自行修复。[LoopsBench](https://arxiv.org/abs/2608.00267) + (`112` 个任务、`>5,300` 个依赖 DAG 单元)使当前最强外循环 SOTA 仍停留在 `25.00%`;在 `EdgeBench` + `New Foundations Consistency` 上,显式声明覆盖台账打破了 `91.9412` 的平台期并升至 `92.3419` + ([EdgeBench 反馈与记忆研究](https://loopx-project.github.io/loopx/blog/edgebench-feedback-and-memory/)); + 实际评测基建同样依赖宿主侧不变量,例如 [`workspace_delivery.py`](../../../benchmark/deepswe-gptxhigh-v1/workspace_delivery.py) + (将 linked worktree 改动归一化回 `/app`)、[`frontiercs2/README.md`](../../../benchmark/frontiercs2/README.md) + (`v2.patch` 的 tmpfs `docker exec` 流式上传与持久准入恢复),以及 + [`widesearch/pier/README.md`](../../../benchmark/widesearch/pier/README.md)(`$HOME` 绑定挂载、纯数值 + `reward.json` 校验与 `provider_credential_shell_excluded`)。 + ## 3. Benchmark 介绍与适配分析 本节的 benchmark 数量和 leaderboard 值是 2026-08-16 审阅时的信息快照,不是规范。 @@ -291,6 +359,76 @@ DeepSWE pass rate 平均成一个 LoopX 数字。 六个 benchmark 仍须在各自主语空间报告;StartupBench rubric、LoopsBench verifier 分层与 WideSearch Item F1 不得互相平均,也不得并入 ALE/LHTB/DeepSWE 的结果。 +### 3.8 外部 Benchmark 调研与优先级排序(2026-11 更新) + +本次 2026-11-15 调研更新 2026-08 的六基准组合入门(§3.1–§3.7),并为后续集成排序 +(§9、§11)覆盖 EdgeBench 及 20 个外部 benchmark。排序依据四项实证标准:(1)相对纯基模 +scaling 仍保留的前沿区分度与上升空间;(2)verifier 确定性、抗 Goodhart 能力以及是否具备 +连续或多单元进度信号;(3)对第 2.2 节五类受治理失败机制的直接压力测试;(4)仓库就绪度 +(`benchmark/` 目录现有 `177` 个 `.py` 文件、共 `52,031` 行:`tests` `27,338` 行、`runtime/` +`9,082` 行、`edgebench/` `6,899` 行、`deepswe*` `4,433` 行、`swe-marathon/` `1,430` 行、 +`frontiercs2*` `1,285` 行、`LHTB/` `596` 行、`widesearch/` `592` 行、根目录辅助 `376` 行)与 +Linux x86\_64 Docker、GPU 及云资源约束之间的权衡。 + +**扩大多臂对照前的共同前置条件:** 在大规模扩展新的 Harbor、云端或 GPU benchmark 多臂 +对比前,应先将各适配器收敛至统一的 `benchmark/runtime/` Turn/Harbor 入口,对齐原生 +Stop-hook 基线对等性,并完成 E0–E2 生命周期、完整性与可计分门控(§11.2),避免继续复制 +烟囱式独立 runner。 + +#### 优先级矩阵 + +| 优先级 | Benchmark | 任务规模与时长 | 奖励与 Verifier 形态 | 前沿余量与机制信号 | 工程就绪度与平台约束 | +|---|---|---|---|---|---| +| **P0** | **EdgeBench** | 连续算法优化(`12h–18h` 长程运行) | 确定性在线冷却裁判;任务原生标量指标(按任务最大化或最小化) | 具备强等预算帕累托区分度(`New Foundations 92.3419`、`ANN 7,159 QPS`;同时强 Stop-hook `Official` 仍在 `Portfolio 53.87`、`Grid 1,791` 领先) | **已就绪并已评测**(`edgebench/` `6,899` 行);需 **Linux x86\_64 Docker + SForge `iptables`**(macOS Docker socket 缺少网络边界) | +| **P0** | **FrontierCS-2** | `48` 公开 / `150` 总计(`13` 个领域);`5h`,限 `3–100` 次 train 调用 | 确定性隐藏工作负载 Driver/Judge;二值 `PASS`(须在*全部*指标与负载上胜过参考解)+ `FCI`/`FDI` 标量 | 12 个参评 agent 中最高通过率 **`58%`**,**`16/48` 零通过**;直接检验代理指标校准与抗刷分能力 | **单机桥接已就绪**(`frontiercs2*` `1,285` 行、`v2.patch` 传输修复;冻结 v1 `batch_local_counting` 归档为 `score_countable=false`、`integrity_qualified=false`;尚无有效 LoopX 计分结果);**强制要求 Linux x86\_64 Docker**(macOS ARM Rosetta/QEMU 下 seccomp 失败);官方 EKS+Modal 全测约 `$1,400`/48 题 | +| **P0** | **LHTB / LHTB 2.0** | `v1: 46`(`90m`);`v2.0: 77`(`46` 原题 + `31` 重构题,`3h` 含 `30m` 检查点) | 连续标量 `[0,1]` + 6 级阶梯(`R>=0.95`/`1.0`)+ 冻结进程 Harbor 二值拒止循环 | `v1` LoopX `0.4948` vs Goal `0.4475` / Plain `0.4218`(非等预算);`v2.0` 最高 `0.692`(`37/77`);早期 `14/17` 满分泄漏实证了 verifier 隔离必要性 | **`v1` Harbor 适配器已就绪**(`LHTB/` `596` 行);**`LHTB 2.0` 尚无代码**;非 GPU Docker / Daytona | +| **P0** | **LoopsBench** | `112` 个依赖 DAG 任务(`>5,300` 单元,`8` 种语言) | 沿就绪 DAG 前沿释放的确定性单元测试 + 回归测试(`incomplete` / `partial` / `complete`) | 最强带外循环 SOTA 仅 **`25.00%`**(`Opus-4.7` + Claude Code + 外循环);专为外循环 DAG 与回归不变量设计 | **仓库暂无活跃代码**(§3.5/§9.5 规划中);非 GPU,建议 Linux x86\_64 Docker | +| **P1** | **SWE-Marathon** | `20` 超长程任务(仓内已评 `15` 题 v1.1 子集);平均 `27.2M tok`/次 | 多层对抗式 verifier;二值求解 + 连续 `partial_score` | `v1` 前沿通过率 `<30%`,**`13.8%` rollout 出现 reward hacking**;仓内 `30%` 预算实验 `heartbeat 0.333 (5/15)` vs `Plain/Goal 0.267 (4/15)` | **已就绪并已评测**(`swe-marathon/` `1,430` 行);`amd64` 多语言 Docker;token 与墙钟成本高(每组 15 题 `1,592m–2,375m`) | +| **P1** | **PaperBenchX** *(优先于 PaperBench)* | `93` 篇论文(`12` 方向 / `6` 个非 ML 学科,`>3,000` 条细则);`4–24h`(中位 `7h`) | 从零干净重放 + 确定性检查与共享 LLM-Judge(`>=90%` 为完整复现) | 细则均分 `27.53–59.08%`,**完整复现仅 `14%`**(`>=90%`)——比旧版 `PaperBench`(`20` 篇 ICML'24 论文、`24–36h`、SOTA `26.0%`)学科更广且验证缺口更锐利 | **仓库 `0` 代码**(约 `2–4` 周适配);需重型 Linux Docker + 专业科学软件栈(`HFSS`、`Meep`、`GPU4PySCF`、`Habitat`)+ Judge API 密钥隔离 | +| **P1** | **ALE (Agents' Last Exam)** *(区别于 `ALE-Bench`)* | `1,500+` 分类体系(`v1.1: 122` 题:`119` CPU / `3` GPU、`78` Docker);多小时 VM | 运行后注入隐藏参考并执行 `evaluate()`,标量 `[0,1]` | 最难层级完整通过率 `<1%`;检验跨 CLI+GUI 专业工作迁移 | **仓库暂无活跃代码**(§3.1/§9.1 规划中,约 `2–4` 周适配);强依赖 **GCP/GCE VM 与 CUA MCP**(无法在 macOS 本地直接运行) | +| **P1** | **WideSearch** | `200`(`100 EN + 100 ZH`,`18` 个行业);`44+` 网页 / 人类均值约 `2.3h` | 自动化原子表格评分(`Item F1` / `Row F1` + 二值 `SR`);存在实时网页漂移非确定性 | 前沿 `SR <= 5.1%`(`o3` 多智能体 `5.1%`、单智能体 `4.5%`、`Gemini 2.5 Pro 4.3%`);高适配扇出/完备性 harness,迭代 RSI 适配度低至中 | **适配器与 `pier` 模板已就绪**(`widesearch/` `592` 行,尚无实跑记录);需 Linux/云端 `$HOME` 挂载(macOS `/tmp` 在 Colima 下失效)及 `permitted_solving` 网络 | +| **P1** | **KernelBench / AlgoTune** | `KernelBench`: `250` PyTorch/CUDA 算子(`L1–L3`+`L4`);`AlgoTune`: `154` 个 SciPy/sklearn/CVXPY 函数 | 确定性正确性校验 + GPU(`fast_p`)或 CPU 几何平均加速比 | `KernelBench` `<20% fast_1`;`AlgoTune` `AlgoTuner 1.72x`(前沿约 `2.05x`);纯客观加速比适合进化/RLVR 循环 | **仓库 `0` 代码**;`KernelBench` 强制依赖 **NVIDIA CUDA/Triton GPU Linux/Modal**(Mac 无法运行);`AlgoTune` 需**固定绑核的独占多核 Linux x86\_64** 计时宿主机 | +| **P2** | **ALE-Bench (SakanaAI)** *(区别于 `ALE`)* | `40` 道 AtCoder 启发式竞赛题(`C++/Rust/Py`,`300+` 用例);`150` 步迭代 | 确定性相对启发式得分 | 单发相对迭代反馈差距显著;跨题稳定性仍落后人类高手 | **仓库暂无活跃代码**;重型多核 Linux/AWS Docker;上游赛题协议禁止打包私有测试种子与输入 | +| **P2** | **StartupBench** | `97` 交付物任务(`DOCX/XLSX/PPTX/PDF`,`6` 域,均 `25.3` 细则) | Evidence view + 加权 Agent-as-Judge `0–100`(`>=90` 通过,`92%` 人类一致性) | 最高均分 `Kimi-k3 73.67`,最高 `pass@1` `GPT-5.6-sol 31.27%`;RSI 适配度低至中(开放交付细则) | **仓库暂无活跃代码**(§3.4/§9.4 规划中);多轮办公产物 API + Judge 边界;非 GPU | +| **P2** | **MLE-bench** | `75` 场 Kaggle 竞赛(`22` Lite `158 GB`,全量 `3.3 TB`);`24h` 标准(榜单于 `2026-04` 暂停) | 确定性 Kaggle 指标 + 奖牌阈值 | 主榜 SOTA `64.44±1.18%`;旧竞赛存在训练污染风险 | **仓库暂无活跃代码**;基础设施门槛极高(`36 vCPU`、`440 GB RAM`、`24 GB A10 GPU`、`3.3 TB` 磁盘、Kaggle 鉴权与 Git LFS) | +| **P2** | **tb-science** *(仅保留单次迁移样本;**排除出 RSI**)* | `96` 已合并任务 + `366` 开放 PR;Harbor 格式 | **100% 二值 `1/0` `reward.txt`**(agent `public` vs verifier `no-network`) | 部分 PR 前沿难度极高,但**不适合 RSI**(无每代连续梯度、无 train/held-out 划分、CI 锁死单轮预算) | **仓库 `0` 代码**(约 `1–2` 周 Harbor 接入);`ci_checks/check-task-timeout.sh` 硬锁 **`28800s`(`8h`)**;建议 Linux x86\_64 Docker | +| **仅作对照 / 降级** | **DeepSWE** | `113` 原创任务(`91` 仓库,`5` 语言) | 手写行为 verifier(`0/1` 二值,`1.4%` judge 分歧率) | `v1.1` 在前沿三模型间呈现约 **`74%` 并列趋近饱和**;仅保留作回归与基线对照 | **已就绪**(`4,433` 行含 `workspace_delivery.py`);需 Linux Docker 宿主机 worker;历史 v1 存在非等预算限制 | +| **仅作对照 / 降级** | **SWE-bench Pro** | `v1: 1,865`(`731` 公开);`v2: 642` + `HARD-51` | 补丁 + 单元测试(`0/1` 二值;`32.4%` judge 分歧率) | **`HARD-51` 已达 `98.0%`**——受基模 scaling 快速饱和 | **仓库暂无活跃代码**;`linux/amd64` Docker 镜像(`ghcr.io`),Mac ARM 非原生支持 | +| **仅作对照 / 降级** | **CORE-Bench** | `270`(`90` 论文,`3` 难度);经 HAL 运行 | 输出问答二值匹配 | 修复容差与脚手架缺陷后,Claude Opus 4.5 从 `21%` 升至 **`95%`**——已饱和 | **仓库 `0` 代码**;Docker/Azure(`Python`/`R`) | +| **仅作对照 / 降级** | **ScienceAgentBench** | `102`(`44` 论文);`30m` Docker 运行 | 有效执行 + 细则/代码检查 | `32.4%–42.2%`(`o1-preview`);单脚本范围限制长程控制面区分度 | **仓库 `0` 代码**;`30m` / 8 线程 Docker | +| **仅作对照 / 降级** | **BixBench** | `60` 胶囊 / `205` 问题(`v1.5`);`24–48h` 5 次重复 | 二值精确匹配 MCQ / LLM-Judge 开放问答 | 开放问答 `17%`(早期基线,缺 2026 前沿 SOTA);5 副本二值评估成本高 | **仓库 `0` 代码**;高昂多副本 API 成本 | +| **仅作对照 / 降级** | **FrontierScience** | `160` 问答项(`100` 奥赛 + `60` 研究) | 简答 + 10 分研究细则 | `GPT-5.2` 奥赛 `77%` / 研究 `25%`;无交互仓库与沙箱闭环 | **仓库 `0` 代码**;纯 API 推理 | +| **仅作对照 / 降级** | **RE-Bench** *(已废弃)* | `7` ML 研发任务(`2h–32h`) | 确定性 ML 评测(含历史时钟/内存作弊案例) | **上游已标废弃**(`METR/RE-Bench`);由 `FrontierCS-2`、`PaperBenchX` 与 `KernelBench` 替代 | **仓库 `0` 代码**;需 Linux x86\_64 + NVIDIA CUDA GPU | + +#### 关键口径辨析与排除说明 + +1. **`ALE` 与 `SakanaAI/ALE-Bench` 区分(及不存在公开的 `"ALE Science"`):** + 公开文献与仓库中不存在名为 `"ALE Science"` 的独立 benchmark 或赛道。`ALE` + ([`rdi-berkeley/agents-last-exam`](https://github.com/rdi-berkeley/agents-last-exam)) + 是包含科学子域的广谱 CLI+GUI 专业工作集,而 `ALE-Bench` + ([`SakanaAI/ALE-Bench`](https://github.com/SakanaAI/ALE-Bench))面向 AtCoder + 启发式算法竞赛优化。两者在上表分别归入 `P1` 与 `P2`。 +2. **`tb-science` 排除出 RSI、仅留单次迁移样本的原因:** + 随着 `laude-institute` 迁移至 `harbor-framework`, + [`harbor-framework/terminal-bench-science`](https://github.com/harbor-framework/terminal-bench-science) + 成为独立于 `terminal-bench-2`(`89` 个通用任务,默认 `900s` 超时)的科学领域套件。尽管已有 `96` 个 + 已合并任务和 `366` 个开放 PR,**`tb-science` 不适合迭代式 RSI 循环**:CI + (`ci_checks/check-task-timeout.sh`)硬性锁定 `[agent] timeout_sec == 28800.0`(`8h`), + 终局 `reward.txt` 为 **100% 二值 `1/0`**(即使任务内部计算连续分),缺少 train/held-out 划分, + `[agent] network_mode` 为 `public`,且 `rubrics/task-implementation.toml` 明确拒绝开放式假说任务。 + 本计划仅保留高难度单次迁移样本(如 Lean 形式化 `#2530`、`#2197`、`#2653`、`#2629`,生物 + `#1518`,物理 `#2804` 或托管 GPU 控制 `#2531`)。 +3. **`PaperBenchX` 优先于旧版 `PaperBench`,及 `DeepSWE` 术语边界:** + `PaperBenchX`(`93` 篇跨学科非 ML 科学论文、从零干净重放)替代旧版 `PaperBench`(`20` 篇 + ICML'24 论文)成为 `P1` 主复现目标。本文中的 `DeepSWE` 指 Datacurve 的 `113` 题 benchmark + ([`datacurve-ai/deep-swe`](https://deepswe.datacurve.ai/)),不同于 Agentica 的 + `DeepSWE-Preview` 模型;其在 `74%` 附近的饱和趋势使新的 `C2`/`C3` 因果投入转向 `P0`/`P1`。 +4. **平台执行约束:** 勿在 macOS ARM64 Rosetta/QEMU 模拟下对依赖 Linux seccomp、`iptables`、 + CUDA 或多核稳定计时的套件(`EdgeBench`、`FrontierCS-2`、`KernelBench`、`AlgoTune`、 + `SWE-bench Pro`、`tb-science`)运行可计分对比;统一在固定配置的 Linux x86\_64(及所需专用 + CUDA/云主机)上执行,并避免在 Colima 上使用 macOS `/tmp` 绑定挂载(见 + `widesearch/pier/README.md`)。 + ## 4. 主张阶梯 每个公开结果必须声明其支持的最高主张等级。 @@ -343,7 +481,12 @@ overhead 与 authority budget,并通过非 benchmark product canary。只有 计划区分四种 arm: 1. **Native baseline**:benchmark 支持的 reference scaffold,或声明过的、没有 LoopX - 控制的 CLI harness。 + 控制的 CLI harness。在 **episode 型 RSI 与标量优化赛道**上,Native baseline 评估还必须 + 同时包含或计入一个零协议税的 **Raw-Artifact Reset Baseline(原始产物重置基线)** + (在每代之间做干净上下文重置,向模型暴露全部历史原始产物与标量分数,仅使用模型原生 + 文件系统/Python 检索而不注入 LoopX packet、ledger 或外置控制开销),使任何 + **Governed LoopX** 主张都必须证明超越模型自编检索的真实因果增益,而非依靠胜过有损手工 + 上下文压缩(§2.2)。 2. **Passive LoopX**:worker decision 和 benchmark 合同完全相同,只增加只读 LoopX observation 与精简 settlement。 3. **Governed LoopX**:声明过的 LoopX profile 可以影响 continuation、checkpoint、 @@ -1304,6 +1447,7 @@ promotion 或修改 LoopX default。 | 2026-08-23 | 增加工程 readiness 阶梯与有界 delivery-slice 规则,将仓库建设与 C0--C4 研究主张分开跟踪。 | | 2026-08-25 | 把 StartupBench、LoopsBench 与 WideSearch 加入 benchmark primer 与集成计划,作为第二组互补 trio;每个 benchmark 必须在原生 metric space 报告,且 benchmark 材料不得进入可复用的 LoopX state。 | | 2026-10-16 | 将[反馈优先观察](https://github.com/loopx-project/loopx/blob/1d60e7edb2a05fe048767444f872934c3adef992/deprecate/benchmark-legacy/edgebench/feedback-first-observations-20261016.md)归入历史报告;运行器源码子集与配置投影修复仍属 E0/E1,混合处理、中断和进行中的观察不关闭 E2/E3,也不分离单一规划效应。通用 guard 重入放宽待受影响调用方独立验证。 | +| 2026-11-15 | 确立 Bitter-Lesson 边界与五类受治理控制面失败机制(§2.2),要求在 episode 型 RSI / 标量优化赛道纳入零协议税的 Raw-Artifact Reset Baseline(§5.2),并采纳 2026-11 Benchmark 组合优先级排序(§3.8):**P0**(`EdgeBench`、`FrontierCS-2`、`LHTB / LHTB 2.0`、`LoopsBench`)、**P1**(`SWE-Marathon`、`PaperBenchX` 优先于 `PaperBench`、`ALE` 区别于 `ALE-Bench`、`WideSearch`、`KernelBench` / `AlgoTune`)、**P2**(`ALE-Bench`、`StartupBench`、`MLE-bench`、`tb-science` 仅保留单次迁移样本——因 CI 锁定 `28800s` 预算与 100% 二值 `reward.txt` 排除出 RSI),以及 **仅作对照 / 降级**(`DeepSWE`、`SWE-bench Pro`、`CORE-Bench`、`ScienceAgentBench`、`BixBench`、`FrontierScience`、已废弃 `RE-Bench`);多臂扩展以统一 `benchmark/runtime/` + Stop-hook E0–E2 就绪及 Linux x86\_64 / GPU 宿主机为前置门槛。 | ## 18. 参考资料 @@ -1312,11 +1456,13 @@ promotion 或修改 LoopX default。 - [Agents' Last Exam 评估框架](https://github.com/rdi-berkeley/agents-last-exam) - [Long-Horizon Terminal Benchmark 仓库](https://github.com/zli12321/LHTB) - [LHTB benchmark report](https://zli12321.github.io/LHTB/index.html) +- [LHTB 论文](https://arxiv.org/abs/2607.08964) - [DeepSWE 项目与 leaderboard](https://deepswe.datacurve.ai/) - [DeepSWE 论文](https://arxiv.org/abs/2607.07946) - [DeepSWE 运行指南](https://deepswe.datacurve.ai/run) - [StartupBench 项目](https://startupbench.github.io/) - [StartupBench 论文](https://startupbench.github.io/assets/StartupBench.pdf) +- [StartupBench arXiv](https://arxiv.org/abs/2608.17800) - [LoopsBench 项目](https://loopsbench.ai/) - [LoopsBench 仓库](https://github.com/microsoft/Loopsbench) - [LoopsBench 论文](https://arxiv.org/abs/2608.00267) @@ -1325,6 +1471,59 @@ promotion 或修改 LoopX default。 - [WideSearch 仓库](https://github.com/ByteDance-Seed/WideSearch) - [WideSearch 论文](https://arxiv.org/abs/2508.07999) - [WideSearch 数据集](https://huggingface.co/datasets/ByteDance-Seed/WideSearch) +- [EdgeBench 仓库](https://github.com/ByteDance-Seed/EdgeBench) +- [FrontierCS-2 项目](https://frontier-cs.org/) +- [FrontierCS-2 仓库](https://github.com/FrontierCS/FrontierCS-2) +- [FrontierCS-2 论文](https://arxiv.org/abs/2610.29932) +- [SWE-Marathon 仓库](https://github.com/abundant-ai/swe-marathon) +- [SWE-Marathon 论文](https://arxiv.org/abs/2606.07682) +- [PaperBenchX 仓库](https://github.com/UniPat-AI/PaperBenchX) +- [PaperBenchX 论文](https://arxiv.org/abs/2610.39744) +- [PaperBench 参考实现](https://github.com/openai/frontier-evals/tree/main/project/paperbench) +- [PaperBench 论文](https://arxiv.org/abs/2504.01848) +- [KernelBench 仓库](https://github.com/ScalingIntelligence/KernelBench) +- [KernelBench 论文](https://arxiv.org/abs/2502.10517) +- [AlgoTune 仓库](https://github.com/oripress/AlgoTune) +- [AlgoTune 论文](https://arxiv.org/abs/2507.15887) +- [ALE-Bench (SakanaAI) 仓库](https://github.com/SakanaAI/ALE-Bench) +- [ALE-Bench 论文](https://arxiv.org/abs/2506.09050) +- [MLE-bench 仓库](https://github.com/openai/mle-bench) +- [MLE-bench 论文](https://arxiv.org/abs/2410.07095) +- [Terminal-Bench Science 项目](https://www.terminal-bench-science.ai/) +- [Terminal-Bench Science 仓库](https://github.com/harbor-framework/terminal-bench-science) +- [SWE-bench Pro 仓库](https://github.com/scaleapi/SWE-bench_Pro-os) +- [SWE-bench Pro 论文](https://arxiv.org/abs/2509.16941) +- [CORE-Bench 仓库](https://github.com/siegelz/core-bench) +- [CORE-Bench 论文](https://arxiv.org/abs/2409.11363) +- [ScienceAgentBench 仓库](https://github.com/OSU-NLP-Group/ScienceAgentBench) +- [ScienceAgentBench 论文](https://arxiv.org/abs/2410.05080) +- [BixBench 仓库](https://github.com/Future-House/BixBench) +- [BixBench 论文](https://arxiv.org/abs/2503.00096) +- [RE-Bench(已废弃)仓库](https://github.com/METR/RE-Bench) +- [RE-Bench 论文](https://arxiv.org/abs/2411.15114) +- [Meta-Harness 论文](https://arxiv.org/abs/2603.28052) +- [Meta-Harness 仓库](https://github.com/stanford-iris-lab/meta-harness) +- [RRSI 论文](https://arxiv.org/abs/2609.24972) +- [RRSI 仓库](https://github.com/google-research/rrsi) +- [Harness Updating Is Not Harness Benefit](https://arxiv.org/abs/2605.30621) +- [Agentic Harness Engineering (AHE)](https://arxiv.org/abs/2604.25850) +- [Self-Harness](https://arxiv.org/abs/2606.09498) +- [Reward Hacking Benchmark (RHB)](https://arxiv.org/abs/2605.02964) +- [DuoDiscover 论文](https://arxiv.org/abs/2610.17184) +- [AlphaEvolve 论文](https://arxiv.org/abs/2506.13131) +- [Darwin Gödel Machine (DGM) 论文](https://arxiv.org/abs/2505.22954) +- [Darwin Gödel Machine 仓库](https://github.com/jennyzzt/dgm) +- [LangChain: Improving Deep Agents with harness engineering](https://www.langchain.com/blog/improving-deep-agents-with-harness-engineering) +- [Vercel `d0`: We removed 80% of our agent's tools](https://vercel.com/blog/we-removed-80-percent-of-our-agents-tools) +- [METR: Recent frontier models are reward hacking](https://metr.org/blog/2025-06-05-recent-reward-hacking/) +- [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) +- [Anthropic: Effective harnesses for long-running agents](https://www.anthropic.com/research/effective-harnesses-for-long-running-agents) +- [Anthropic: Harness design for long-running application development](https://www.anthropic.com/research/harness-design-long-running-apps) +- [LoopX:上下文交付研究](https://loopx-project.github.io/loopx/blog/context-delivery/) +- [LoopX:EdgeBench 反馈与记忆研究](https://loopx-project.github.io/loopx/blog/edgebench-feedback-and-memory/) +- [LoopX:EdgeBench 长程修复研究](https://loopx-project.github.io/loopx/blog/edgebench-long-horizon-repairs/) +- [前沿科学研究计划 v0](./frontier-science-research-program-v0.zh-CN.md) +- [探索性 benchmark 行为发现](../../reference/benchmark-behavior-findings.md) - [Benchmark 研究工作区](https://github.com/huangruiteng/loopx/blob/main/benchmark/README.md) - [DeepSWE 研究实践](https://github.com/huangruiteng/loopx/blob/main/benchmark/deepswe/README.md) - [旧 Benchmark 历史](https://github.com/loopx-project/loopx/tree/54d42da44283bb414b24f71d16642c25cbcaf5dd/deprecate/benchmark-legacy) diff --git a/docs/research/README.md b/docs/research/README.md index 8d4d12eb8e..d8e63aa8bb 100644 --- a/docs/research/README.md +++ b/docs/research/README.md @@ -27,6 +27,12 @@ establish an equal-budget efficiency gain or a general performance guarantee. ## Research Index +- [Benchmark portfolio priority ranking and Bitter-Lesson boundary (2026-11)](../architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md#38-external-benchmark-survey-and-priority-ranking-2026-11) + ([中文](../architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.zh-CN.md#38-外部-benchmark-调研与优先级排序2026-11-更新)): + updated P0/P1/P2/Control-Only ranking across EdgeBench and 20 external + benchmarks, engineering readiness and platform constraints, and the + Raw-Artifact Reset Baseline contract separating self-authored retrieval from + the five governed control-plane failure regimes. - [Budgeted research planning: goals, theory and design rationale](budgeted-research-planning.md) ([中文](budgeted-research-planning.zh-CN.md)): why conditional plans, costly measurements and causal comparisons serve better long-budget outcomes; links