Archive real SWE-bench validation for Issue #33 - #36
Merged
Conversation
16 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
关联 #33
目的
本 PR 现在包含两部分相互绑定的交付:
SWEbenchRunConfig与公开swebench runCLI 共用的默认 Context 字符窗口从 100,000 调整为 20,000;单一权威常量为
DEFAULT_CONTEXT_MAX_CHARACTERS = 20_000。--context-max-characters显式覆盖能力保持不变,CLI help 会显示default: 20000。验收边界
真实 SWE-bench 成功运行先于默认值 follow-up commit 完成,并显式使用同一个 20,000 值。因此:
config.json与manifest.json会选择并记录 20,000;被验收的候选实现是 PR #35 merge commit
815b166f45627a17f5c5f7641e508497648ddba3。真实验收结果
pallets__flask-5014,20,000 字符:Agent exit 0;3 个 version 2 compaction checkpoints;官方 Harness F2P 1/1、P2P 59/59,resolved。scikit-learn__scikit-learn-14141,20,000 字符:Agent exit 0;1 个 version 2 compaction checkpoint;官方 Harness F2P 1/1、P2P 2/2,resolved。产品默认值 follow-up
Commit
9fdf2125e79254fa2c35a6214003d73a05e01fbf:1673c5bcbb11744d2cd1a74e421b0510c13dd0d4;2e4bd3f8f2a5207a29ccf5cc429fb70200662729c74a6d880c9bd3bd0bfd9b77。证据归档
归档包含配置、provenance、Session/ToolResult、生命周期事件、checkpoint coverage/lineage、prediction、workspace patch、官方 Harness 输出、source manifests 与逐文件 SHA-256。它不归档完整工作区或 token-level 流事件。
更新后的根
archive.sha256为fa2c9a26a44b9dad3e0581a6d8fe456b3be84fa4ee830f1661ea3a00f38630d0,154/154 项匹配;87 个 JSON、31 个 JSONL、12 个 Markdown 本地链接均通过,credential/private-path 扫描为 0。Issue #33 在本 PR 完成合并前保持 OPEN。
English appendix
Relates to #33.
This PR combines the real DeepSeek/SWE-bench acceptance archive with a follow-up product change that sets the shared Kernel,
SWEbenchRunConfig, and public CLI default character window to 20,000 throughDEFAULT_CONTEXT_MAX_CHARACTERS. Explicit CLI overrides remain supported.The real Flask and scikit-learn runs explicitly used the same 20k value before the default was changed, so they validate actual compaction and official Harness outcomes at that runtime configuration. Repository tests validate only the omitted-argument default selection and persisted configuration; they are not presented as a new external acceptance run. The retained 5k failure and successful 20k rerun remain archived for auditability.