Skip to content

Archive real SWE-bench validation for Issue #33 - #36

Merged
Ev3rGan merged 2 commits into
mainfrom
codex/issue-33-real-swebench-acceptance
Sep 8, 2026
Merged

Archive real SWE-bench validation for Issue #33#36
Ev3rGan merged 2 commits into
mainfrom
codex/issue-33-real-swebench-acceptance

Conversation

@Ev3rGan

@Ev3rGan Ev3rGan commented Sep 8, 2026

Copy link
Copy Markdown
Owner

关联 #33

目的

本 PR 现在包含两部分相互绑定的交付:

  1. 将 Kernel、SWEbenchRunConfig 与公开 swebench run CLI 共用的默认 Context 字符窗口从 100,000 调整为 20,000;
  2. 归档 Issue [Feature]: 实现 Context Compaction v2 语义 checkpoint 与权威上下文重投影 #33 在真实 DeepSeek + SWE-bench Verified 场景中的验收证据。

单一权威常量为 DEFAULT_CONTEXT_MAX_CHARACTERS = 20_000--context-max-characters 显式覆盖能力保持不变,CLI help 会显示 default: 20000

验收边界

真实 SWE-bench 成功运行先于默认值 follow-up commit 完成,并显式使用同一个 20,000 值。因此:

  • 真实运行证明 20k 窗口会实际触发 semantic compaction,并通过官方 Harness;
  • 仓库测试证明省略参数后,产品默认、config.jsonmanifest.json 会选择并记录 20,000;
  • pytest 不被表述为新的外部 SWE-bench 验收。

被验收的候选实现是 PR #35 merge commit 815b166f45627a17f5c5f7641e508497648ddba3

真实验收结果

  • pallets__flask-5014,20,000 字符:Agent exit 0;3 个 version 2 compaction checkpoints;官方 Harness F2P 1/1、P2P 59/59,resolved。
  • scikit-learn__scikit-learn-14141,20,000 字符:Agent exit 0;1 个 version 2 compaction checkpoint;官方 Harness F2P 1/1、P2P 2/2,resolved。
  • 保留 Flask 5,000 字符失败:8 个合法 v2 checkpoints 持久化后,第 9 次摘要超过 2,500 字符限制,Agent exit 4,未进入 Harness。20k 是本次验收使用并归档的 runtime configuration hotfix;成功复跑仍实际触发 3 次 compaction。

产品默认值 follow-up

Commit 9fdf2125e79254fa2c35a6214003d73a05e01fbf

  • 9 files,+107/-11;stable patch-id 1673c5bcbb11744d2cd1a74e421b0510c13dd0d4
  • 定向 Context/SWE-bench:61 passed,1 skipped;
  • 全量 pytest:365 passed,1 skipped;
  • strict mypy:34 source files PASS;
  • Ruff lint / format、git diff-check:PASS;
  • wheel 构建、隔离安装、API/CLI help 默认值检查:PASS;
  • wheel SHA-256:2e4bd3f8f2a5207a29ccf5cc429fb70200662729c74a6d880c9bd3bd0bfd9b77

证据归档

归档包含配置、provenance、Session/ToolResult、生命周期事件、checkpoint coverage/lineage、prediction、workspace patch、官方 Harness 输出、source manifests 与逐文件 SHA-256。它不归档完整工作区或 token-level 流事件。

更新后的根 archive.sha256fa2c9a26a44b9dad3e0581a6d8fe456b3be84fa4ee830f1661ea3a00f38630d0,154/154 项匹配;87 个 JSON、31 个 JSONL、12 个 Markdown 本地链接均通过,credential/private-path 扫描为 0。

Issue #33 在本 PR 完成合并前保持 OPEN。

English appendix

Relates to #33.

This PR combines the real DeepSeek/SWE-bench acceptance archive with a follow-up product change that sets the shared Kernel, SWEbenchRunConfig, and public CLI default character window to 20,000 through DEFAULT_CONTEXT_MAX_CHARACTERS. Explicit CLI overrides remain supported.

The real Flask and scikit-learn runs explicitly used the same 20k value before the default was changed, so they validate actual compaction and official Harness outcomes at that runtime configuration. Repository tests validate only the omitted-argument default selection and persisted configuration; they are not presented as a new external acceptance run. The retained 5k failure and successful 20k rerun remain archived for auditability.

@Ev3rGan
Ev3rGan merged commit 02b6382 into main Sep 8, 2026
1 check passed
@Ev3rGan
Ev3rGan deleted the codex/issue-33-real-swebench-acceptance branch September 8, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant