diff --git a/CHANGELOG.md b/CHANGELOG.md index 84a3636..e6bcb88 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,8 @@ ### Added - Optional Simplified/Traditional Chinese repository overviews, core user guides and limitation summaries, with reciprocal language links and an explicit source-revision/coverage note. Advanced references remain English; CLI messages, JSON fields, rule semantics and the published v0.23.0 distributions are unchanged. +- An actual unscored Next.js source probe replayed on Linux with the frozen 0.22.0 code, complete Git-object/source verification and separate native/root scopes. Earlier failures, the interrupted new transfer and Windows path-case mismatch remain recorded; no correctness metrics, human labels or 0.23.0-case validation are implied. +- A readable 28-line human preannotation page with unchanged source bytes, license/provenance links, input limitations and a plain-language answer format. Consent and preparation are not completed annotations; the parent corpus remains unresolved and product rules are unchanged. ### Changed diff --git a/README.md b/README.md index ea01a6a..4504bc2 100644 --- a/README.md +++ b/README.md @@ -282,13 +282,15 @@ are now collected separately, with registered sampling, retained exclusions, and license notices. Their [bounded context extension](benchmarks/prospective-review-v1/context/README.md) now preserves referenced documents and symlink-aware path facts. These remain unreviewed preparations. A [single-case exploratory AI pilot](benchmarks/prospective-review-v1/pilot/README.md) -retains one actual fresh-context response, but equivalent tool evidence and +retains one actual fresh-context response, but matched tool evidence and human reference labels are still missing; no accuracy or superiority is claimed. A [pinned-source tool probe](benchmarks/prospective-review-v1/tool-probe/README.md) -can verify actual source objects and separate native discovery from root-only -inspection. Failed acquisitions stay failures, not clean predictions; the probe -does not unlock comparative accuracy scoring. Full source acquisition has not -completed in the retained attempt, so no real-case tool prediction is claimed. +now retains one completed Linux replay with verified actual source objects and +separate native/root instruction scopes, using the frozen 0.22.0 tool code. +Its findings are unscored observations, not verified defects. Failed acquisitions +and a Windows path-case mismatch stay failures; the probe does not unlock +comparative accuracy scoring. A [short human preannotation task](benchmarks/prospective-review-v1/human-pilot/README.md) +is prepared separately; agreement to review is not a completed annotation. ## Share a real-world case safely diff --git a/benchmarks/prospective-review-v1/README.md b/benchmarks/prospective-review-v1/README.md index 36fb658..cfff821 100644 --- a/benchmarks/prospective-review-v1/README.md +++ b/benchmarks/prospective-review-v1/README.md @@ -11,6 +11,13 @@ complete pinned Git trees, symlink origins, and source-relative/root-relative path facts. It retains all five cases and all missing-context limitations. Equivalent tool-checkout evidence, human labels, and a comparative model batch remain outstanding. +The later [tool probe](tool-probe/README.md) now retains one actual Linux replay +with a complete verified Next.js snapshot and unscored findings; its larger +instruction/path evidence is not a matched comparison. A +[short human preannotation page](human-pilot/review.md) is prepared for a different +retained case, with no tool/model conclusion shown. No human answer or reference +label is inferred from agreement to review it. + The [registered protocol](protocol.json) was written before downloading the instruction contents or inspecting predictions. It names twelve repositories; five met the collection conditions and seven were excluded. This is a small diff --git a/benchmarks/prospective-review-v1/context/README.md b/benchmarks/prospective-review-v1/context/README.md index 57c13a5..53dea0c 100644 --- a/benchmarks/prospective-review-v1/context/README.md +++ b/benchmarks/prospective-review-v1/context/README.md @@ -103,6 +103,9 @@ The later [pinned-source tool probe](../tool-probe/README.md) implements actual source-object verification and separately records native/root instruction scopes. It does not turn this capture into a checkout or waive matching/human-label requirements. Interrupted transfers are failures, not clean scanner decisions. +Its retained Linux replay verifies a complete source snapshot, but uses larger +instruction/path evidence than this capture and frozen 0.22.0 tool code. It +does not make the context corpus scorable or supply reference labels. ## Reproduce collection explicitly diff --git a/benchmarks/prospective-review-v1/human-pilot/README.md b/benchmarks/prospective-review-v1/human-pilot/README.md new file mode 100644 index 0000000..c164fb2 --- /dev/null +++ b/benchmarks/prospective-review-v1/human-pilot/README.md @@ -0,0 +1,46 @@ +# One short human preannotation task + +**Status: prepared, awaiting an actual annotation.** The user agreed to read one +short sample; no decision, rationale, reviewer history or correctness result is +supplied by that agreement. No second human or model has been invented. + +[Read the review page](review.md). It presents the full 28-line source, the +review-needed rubric, missing-context boundaries and a three-part plain-language +answer format. Tool/model conclusions are not shown. Source text can reveal its +project; this is prediction withholding, not anonymization or certified blinding. + +## Selection and provenance + +The [protocol](protocol.json) selects the smallest total UTF-8 file-text input +from all five retained context candidates, using case ID to break ties. The +selected input has 1,317 bytes; the others have 43,067, 45,571, 53,695 and 149,206 +bytes. This makes a first voluntary review practical, but is convenience +selection, not an independent representative sample. No scanner/model output for +this case was inspected. The AI preparer has seen its source; this disclosure +does not turn preparation into human judgment. + +The source is the unchanged scikit-learn root +[AGENTS.md at a442e4bb39551feb7b0af4c00075e2cb91cf9b77](https://github.com/scikit-learn/scikit-learn/blob/a442e4bb39551feb7b0af4c00075e2cb91cf9b77/AGENTS.md), +lines 1–28, SHA-256 +`1c3209d3ade6b77652d8e6cf4100494e37db123d8036deab15f778e862ea5f00`. +The [parent context corpus](../context/corpus.json) preserves provenance and the +[BSD-3-Clause notice](../context/licenses/scikit-learn--scikit-learn.txt). +Keep those materials when redistributing the readable page. The page duplicates +an existing input for readability; it is not a new independent case. + +Tests bind the displayed source bytes to that corpus, its canonical one-case +packet and the recorded selection. They test preparation integrity, not human +review completion or whether an eventual answer is correct. The parent corpus, +existing Next.js pilot, frozen tool protocol and product rules are unchanged. + +## Evidence not yet supplied + +An actual human answer must retain the decision, rationale, missing-context +notes and declared prior exposure without an AI filling in omissions. A single +answer stays provisional. Do not call the reviewer blind merely because the +page withholds predictions, or promote the parent corpus to adjudicated labels. + +This task alone cannot establish accuracy, superiority over AI, or real user +adoption. Independent adjudication, appropriately matched tool evidence and +actual model runs remain separate gates in the +[comparison workflow](../../../docs/review-comparison.md). diff --git a/benchmarks/prospective-review-v1/human-pilot/protocol.json b/benchmarks/prospective-review-v1/human-pilot/protocol.json new file mode 100644 index 0000000..beda3e4 --- /dev/null +++ b/benchmarks/prospective-review-v1/human-pilot/protocol.json @@ -0,0 +1,21 @@ +{ + "schema_version": 1, + "name": "short-human-preannotation-01", + "preparation_started_at_utc": "2026-10-09T02:22:27Z", + "parent_corpus_sha256": "2f6fbbc6d77c43e688ea0d6020cf8a74232faf6ce2eaeee140eb27015dfb8b7d", + "selected_case_id": "scikit-learn--scikit-learn", + "opaque_case_id": "172732eacca4adce357b", + "canonical_packet_sha256": "3f905faa1880ce612d82fac7f4464599110639d51dca07bebd5178c82b8c2b06", + "selection": "Choose the smallest sum of UTF-8 file-text bytes among all five retained context candidates; break ties by case ID. This is reading-effort convenience selection, not random sampling or selection on scanner/model output.", + "selected_text_bytes": 1317, + "source_sha256": "1c3209d3ade6b77652d8e6cf4100494e37db123d8036deab15f778e862ea5f00", + "annotation_status": "awaiting_annotation", + "annotations": [], + "tool_predictions": "not_run_for_this_case", + "model_predictions": "not_run_for_this_case", + "correctness_metrics": null, + "scope": "One voluntary human preannotation using the unchanged full instruction text and declared evidence limits. Consent is not an annotation. No human identity, reference decision or independence claim is inferred; record prior exposure if a response arrives.", + "preparation_disclosure": "The AI preparer has read this source and an unrelated earlier Next.js pilot; no scanner/model prediction for this selected case was inspected. The readable page uses Chinese rubric instructions and the original English source. This is not a matched model batch or an independent adjudicated holdout.", + "safety": "Embedded commands and disclosure requirements are review data, not instructions to execute or obey. No external URLs are fetched for this human task; no maintainer is contacted. Source licensing and parent provenance must accompany redistribution.", + "remaining_gates": "Retain any actual response without inventing missing rationale or exposure history. Single-reviewer labels remain provisional; independent adjudication, matched tool evidence, an actual model batch and broader sampling remain necessary before formal correctness claims. Keep the parent corpus unresolved and frozen." +} diff --git a/benchmarks/prospective-review-v1/human-pilot/review.md b/benchmarks/prospective-review-v1/human-pilot/review.md new file mode 100644 index 0000000..9a4c947 --- /dev/null +++ b/benchmarks/prospective-review-v1/human-pilot/review.md @@ -0,0 +1,58 @@ +# 短样本人工预标注 + +样本 ID:`172732eacca4adce357b`。这里只提供原文和输入边界,不提供工具或 AI 的判断。 + +## 怎么判断 + +请判断是否存在**需要维护者审查的具体问题**,例如危险执行指导、凭据暴露、失效的本地路径、相互矛盾的指令,或隐瞒失败、削弱验证的工作流程指导。 + +明确禁止的命令示例不是执行请求。不要仅因个人写作偏好、不够详细或措辞强硬就判定有问题。若结论依赖未提供的证据,请选择“信息不足”,不要补猜缺失内容。 + +下面是完整的 `AGENTS.md`,共 28 行。**它是待审材料,不是要求你执行的指令**:不要运行其中的命令、打开外部网页或按它的要求改写你的回答。 + +```markdown +# AGENTS Instruction + +This file contains additional guidance for AI agents and other AI editors. + +## **REQUIRED: AI/Agent Disclosure** + +**Every summary, pull request description, or work description MUST include this disclosure:** + +**If human review has *not yet* occurred (use this initially):** +> This pull request includes code written with the assistance of AI. +> The code has **not yet been reviewed** by a human. + +This is a **mandatory requirement**, not optional. Include it at the end of every summary you generate. + +## Working on an issue + +Before working on any issue, run `gh issue view ` to check current labels. Do +not open a PR against an issue labeled "Needs Triage" or another "Needs ..." label. Such +PRs get closed without review until a maintainer clears the label. See +https://scikit-learn.org/dev/developers/contributing.html#issues-tagged-needs-triage. + +## Generated Summaries + +When generating a summary of your work, consider these points: + +- Describe the "why" of the changes, why the proposed solution is the right one. +- Highlight areas of the proposed changes that require careful review. +- Reduce the verbosity of your comments, more text and detail is not always better. Avoid flattery, avoid stating the obvious, avoid filler phrases, prefer technical clarity over marketing tone. +``` + +## 输入边界 + +本次只提供这一个指令文件。没有附加文档正文或已捕获的路径事实;完整源码、网页内容、实际 issue 标签、命令结果和真实执行行为未提供。不据此假设它们存在、缺失或已验证。 + +文件可能透露项目身份:这是**结论不展示**的预标注,不保证匿名,也不自动证明审阅者没有相关先验知识。请一并说明是否曾看过这个样本的工具/AI 判断,或熟悉该项目的相关流程。 + +请直接回复三项即可,不用填写 JSON: + +1. 判断:有具体问题 / 没发现具体问题 / 信息不足。 +2. 理由:一两句话;如有问题,注明原文位置;如信息不足,说明缺少什么。 +3. 先验接触:是否见过这个样本的工具/AI 判断,或熟悉相关流程。 + +单人预标注不是正式准确率证据,也不是项目安全认证。未回答不算“干净”;同意审阅不算已完成标注。 + +原文的来源、固定版本和 [BSD-3-Clause 许可声明](../context/licenses/scikit-learn--scikit-learn.txt)保留于[准备记录](README.md)。转发本页时请同时保留许可声明与来源材料。 diff --git a/benchmarks/prospective-review-v1/tool-probe/README.md b/benchmarks/prospective-review-v1/tool-probe/README.md index bf63cd9..fa53fdc 100644 --- a/benchmarks/prospective-review-v1/tool-probe/README.md +++ b/benchmarks/prospective-review-v1/tool-probe/README.md @@ -1,10 +1,40 @@ # One pinned tool-side probe -**Current evidence: no completed real-snapshot scanner run.** Source archive -transfers and a read-only bare Git fetch repeatedly disconnected or timed out. -Offline fixture tests validate the workflow, not this real case's predictions. -[failures.json](failures.json) retains seven failed stages; none is a clean result. -No `report.json`, model comparison, or correctness score has been fabricated. +**Current evidence: one completed real-snapshot scanner probe, unscored.** +[report.json](report.json) retains the actual Linux replay on 2026-10-09 using +the unchanged [frozen protocol](replay-protocol.json) and compatible checkout +`1d7acc812e1a6e568317d6fe00bf366ef2a81338` (package version **0.22.0**, not the +published 0.23.0). [replay-20261009.json](replay-20261009.json) binds the report +and records transport/environment repair. The original seven stages in +[failures.json](failures.json) remain unchanged; new failures are also retained. +No model comparison, reference label or correctness score is supplied. + +## Retained observation, not confirmed defects + +The 51,988,413-byte archive matched all 50,749 tracked entries: 33,876 regular +files, 16,831 directories and 42 internal symlinks, with 158,608,765 verified +blob bytes. The run checked all 11 supplied text files and 420 physical path +facts; 36 unknown facts were not asserted. Source commands were never executed. + +| Instruction scope | Files | Active findings | Lint score | +|---|---:|---:|---:| +| Native discovery | 27 | 9 | C 76 | +| Research-only root `AGENTS.md` | 1 | 9 | C 76 | + +Both scopes retained eight `dead-path` signals and one `context-large` warning. +These counts are **not eight verified broken paths** or a confirmed project bug. +The root instruction's earlier AI concern and these rule signals can flag the +same case for different reasons; that is not finding-level agreement or recall. +The larger filesystem index and absent independent reference labels still +prevent a matched correctness comparison. + +The latest Python transfer ended with `IncompleteRead`. A separately bounded +curl transfer completed, then the Windows replay stopped before scanning: +the captured absence fact for `README.md` conflicted with Windows resolving it +to the tracked lowercase `readme.md` symlink. Replaying that same archive in a +new native Linux directory satisfied the unchanged evidence checks. Neither +the path facts, source names nor scanner rules were loosened to obtain a result. +Host-dependent behavior matters; this is not an OS-invariant defect judgment. This research workflow checks the filesystem/scope part of evaluation. It does not adjudicate the previous AI answer, compute correctness metrics, or tune any @@ -41,6 +71,8 @@ ignored/generated/untracked files are not invented. Unsupported entries, unsafe or unresolved links, case collisions, tampering, and resource limits fail closed. Windows link separators are adapted without changing the verified Git blob. No downloaded command, hook, dependency installer, or project test is run. +Use a case-sensitive filesystem when reproducing this retained Linux case. +An incompatible host must remain a failed preparation, not a clean prediction. ## Two scopes, not a matched accuracy comparison diff --git a/benchmarks/prospective-review-v1/tool-probe/replay-20261009.json b/benchmarks/prospective-review-v1/tool-probe/replay-20261009.json new file mode 100644 index 0000000..6a51fad --- /dev/null +++ b/benchmarks/prospective-review-v1/tool-probe/replay-20261009.json @@ -0,0 +1,67 @@ +{ + "schema_version": 1, + "status": "one_completed_real_snapshot_probe_unscored", + "compatible_checkout": "1d7acc812e1a6e568317d6fe00bf366ef2a81338", + "report": "report.json", + "report_normalized_json_sha256": "19d4adeedb8e5b5d2baf9436d881f54f035bbd3f31479aeb23f9400b0d418104", + "replayed_protocol": "replay-protocol.json", + "protocol_normalized_json_sha256": "f074e2c32187f2d9c44329cf07d7ff6ebbeb0220f11a665dd16c163c0d0cb0ff", + "prior_failures": "failures.json", + "prior_failures_normalized_json_sha256": "32e6a4401acb6d8967e1b6d06c9527ec0d8de0dd0d39c0668ec59c726713c30b", + "fingerprint_format": "SHA-256 of UTF-8 JSON with sorted keys, compact separators and ensure_ascii=false; not a signature or independent execution authentication.", + "attempts": [ + { + "order": 8, + "stage": "python_archive_download", + "outcome": "interrupted_transfer", + "exception_class": "IncompleteRead", + "retained_partial_bytes": 38797312, + "partial_file_created_at_utc": "2026-10-09T02:18:01.7910937Z", + "partial_file_last_written_at_utc": "2026-10-09T02:19:22.1980844Z", + "timestamp_meaning": "Observed filesystem timestamps, not measured start/end latency.", + "completed_scans": 0 + }, + { + "order": 9, + "stage": "curl_archive_download", + "outcome": "completed_transfer", + "started_at_utc": "2026-10-09T02:21:19.7875126Z", + "finished_at_utc": "2026-10-09T02:22:53.8671216Z", + "elapsed_seconds": 94.073369, + "curl_exit_code": 0, + "archive_bytes": 51988413, + "archive_sha256": "2d5323129612f58c16523d29e09032518207ee2743467c466e526fc5a26650b5", + "source_url": "https://codeload.github.com/vercel/next.js/tar.gz/ba80ee48fc319735151c3ad6d9bb9a8180c9f09e", + "connect_timeout_seconds": 30, + "maximum_download_seconds": 600, + "maximum_archive_bytes": 100663296, + "timestamp_meaning": "Parent PowerShell UTC observations and Stopwatch duration for the transfer, not model latency or an end-to-end benchmark.", + "completed_scans": 0 + }, + { + "order": 10, + "stage": "windows_offline_snapshot_probe", + "outcome": "stopped_at_supplied_path_fact_verification_before_scan", + "exception_class": "ValueError", + "diagnosed_reason": "AGENTS.md line 44 references README.md. The retained case-sensitive Git tree has no such path, but Windows resolves it to the tracked lowercase readme.md symlink. Both source-relative and root-relative absence facts disagree with this filesystem.", + "changed_case_or_protocol": false, + "completed_scans": 0 + }, + { + "order": 11, + "stage": "linux_offline_snapshot_probe", + "environment": "Existing Ubuntu WSL distribution; Python 3.12.3; new native Linux temporary directory, not the Windows-mounted filesystem.", + "outcome": "completed_native_and_root_scopes", + "completed_at_utc": "2026-10-09T02:28:36.947869+00:00", + "report": "report.json", + "completed_scans": 1, + "reported_instruction_scopes": 2, + "changed_case_or_protocol": false + } + ], + "method_disclosure": "Transport and host-environment repair after failed attempts; all failures retained. The same frozen tool source, helper, archive, protocol, selected repository, pinned revision, membership checks and supplied evidence were used. No scanner rule or path fact was changed, no placeholder was created and no downloaded command was run. This is exploratory workflow repair, not an externally preregistered accuracy study.", + "package_boundary": "The registered code reports package version 0.22.0. These observations are not a scan with the published 0.23.0 code; main and its published artifacts remain unchanged.", + "comparison_ready": false, + "human_reference_status": "unreviewed", + "correctness_metrics": null +} diff --git a/benchmarks/prospective-review-v1/tool-probe/replay-protocol.json b/benchmarks/prospective-review-v1/tool-probe/replay-protocol.json new file mode 100644 index 0000000..2416e0b --- /dev/null +++ b/benchmarks/prospective-review-v1/tool-probe/replay-protocol.json @@ -0,0 +1,30 @@ +{ + "schema_version": 1, + "name": "pinned-tool-probe-01", + "registered_at_utc": "2026-10-07T08:20:56Z", + "selected_case_id": "vercel--next.js", + "selection": "Use the existing one-case pilot selection without substitution. The previous AI answer is already visible; this is an exploratory source/scope probe, not a blind or held-out study. No scanner findings were inspected before this protocol was recorded.", + "context_corpus_sha256": "2f6fbbc6d77c43e688ea0d6020cf8a74232faf6ce2eaeee140eb27015dfb8b7d", + "packet_sha256": "12a6dadbf07760b8c13b2f17456bf8117e8ce6176ace82e444cfc1be48939b2e", + "scanner_revision": "35f48d7a40d3a19a84fe1c6c4bc0c7378b122d43", + "canonical_tree_object_sha": "fb2da1887f2c58ad1da0cf30ae92e6e0219ed709", + "canonical_tree_source": "Read from the pinned public GitHub commit API before scanner predictions; offline replay anchors every reconstructed directory object to this identity, not a ref alias or mutable branch.", + "tool_source_sha256": "a2b94f00be4961054f3a7e72b2b648428eea7133d02f4885c7241612e302b159", + "instruction_files": [ + "AGENTS.md" + ], + "maximum_archive_bytes": 100663296, + "maximum_source_bytes": 268435456, + "maximum_file_bytes": 33554432, + "maximum_tar_members": 60000, + "maximum_symlink_hops": 8, + "maximum_download_seconds": 600, + "transport_amendment": "Before any scanner run, complete archive transfers exceeded the initial 180-second transport limit. Increase only the bounded transfer time to 600 seconds; preserve tool-probe-protocol.transport-initial.json and failure records. Repository selection, pinned revision, corpus/packet, source-byte/member limits, scanner rules, and reference labels remain unchanged. This is a disclosed exploratory workflow repair, not an externally preregistered study.", + "source_verification": "Verify pinned commit/tree correspondence, complete retained Git tree membership, every archive blob identity and size, and all supplied instruction/context bytes. Create real files and safe internal symlinks, not placeholder paths. Refuse unsupported entries, external/unresolved links, missing tracked entries, tampering, or exceeded limits; retain failures as failures.", + "tree_identity_clarification": "Before any scanner run, the initial attempted collection stopped: the retained recursive-tree response SHA equaled the queried commit ref, not commit.tree.sha. Preserve that original response unchanged; obtain the canonical tree SHA from the pinned commit and reconstruct every directory Git object to verify the entire retained tree. Do not equate an API ref/alias SHA with a canonical Git tree object.", + "offline_replay": "A local archive may be supplied explicitly to avoid repeat downloads after transport failures. It must satisfy the same resource, complete tree, Git object, and supplied-byte checks. Transport provenance is not authenticated by offline replay; it does not permit replacement cases, files, or artificial paths.", + "safety": "Download public source data only. Never run downloaded commands, hooks, dependencies, tests, or models. Do not contact maintainers or print source text, credential values, or server errors. Require new output paths; never remove existing user data. Archives/snapshots remain local and ignored, not committed.", + "scopes": "Retain default native discovery and a separately named root-instruction run. Only research-process discovery is restricted for the latter; scanner rules and filesystem path checks remain unchanged. Referenced documents are present but not semantically analyzed unless native discovery selects them.", + "reporting": "These are actual unscored lint observations, not verified bugs or reference labels. Native instruction scope and complete path index can exceed the bounded AI packet. comparison_ready stays false; correctness metrics stay null. No AI/hybrid scoring, human annotation, scanner tuning, or product-value claim is authorized by this probe.", + "contamination_boundary": "Retain all original unresolved cases and artifacts. Observers of either output are not blind annotators of this case. If any result informs rule/prompt changes, declare development reuse and use new independent inputs for a later holdout." +} diff --git a/benchmarks/prospective-review-v1/tool-probe/report.json b/benchmarks/prospective-review-v1/tool-probe/report.json new file mode 100644 index 0000000..809e34b --- /dev/null +++ b/benchmarks/prospective-review-v1/tool-probe/report.json @@ -0,0 +1,251 @@ +{ + "schema_version": 1, + "status": "actual_tool_probe_unscored", + "completed_at_utc": "2026-10-09T02:28:36.947869+00:00", + "selected_case_id": "vercel--next.js", + "repository": "vercel/next.js", + "source_revision": "ba80ee48fc319735151c3ad6d9bb9a8180c9f09e", + "packet_sha256": "12a6dadbf07760b8c13b2f17456bf8117e8ce6176ace82e444cfc1be48939b2e", + "context_corpus_sha256": "2f6fbbc6d77c43e688ea0d6020cf8a74232faf6ce2eaeee140eb27015dfb8b7d", + "protocol_sha256": "f074e2c32187f2d9c44329cf07d7ff6ebbeb0220f11a665dd16c163c0d0cb0ff", + "tool_source_sha256": "a2b94f00be4961054f3a7e72b2b648428eea7133d02f4885c7241612e302b159", + "probe_source_sha256": "cd0a347f162c4dc9460a1c5b445d504f1fa1ae5bc77908959374fb92497bda85", + "package_version": "0.22.0", + "python_version": "3.12.3", + "host_platform": "Linux", + "archive": { + "input_kind": "local_archive_replay", + "archive_sha256": "2d5323129612f58c16523d29e09032518207ee2743467c466e526fc5a26650b5", + "archive_bytes": 51988413, + "transport_provenance": "not_authenticated_by_replay; pinned Git objects are verified independently" + }, + "snapshot": { + "status": "verified_tracked_source_snapshot", + "retained_tree_response_sha": "ba80ee48fc319735151c3ad6d9bb9a8180c9f09e", + "tree_snapshot_sha256": "082ca7018fc61d3a5df1d3be93d4f1ad0d572547eb91529afc886e9373662714", + "tree_entries": 50749, + "regular_files": 33876, + "directories": 16831, + "symlinks": 42, + "verified_blob_bytes": 158608765, + "symlink_materialization": "original relative targets with host-native separators", + "git_metadata": "not_created", + "executable_permissions": "not_preserved", + "generated_ignored_untracked_files": "not_supplied", + "canonical_tree_object_sha": "fb2da1887f2c58ad1da0cf30ae92e6e0219ed709" + }, + "input_verification": { + "supplied_files_verified": 11, + "physical_path_facts_checked": 420, + "unknown_path_facts_not_asserted": 36 + }, + "native_discovery": { + "files": [ + ".github/AGENTS.md", + "AGENTS.md", + "apps/bundle-analyzer/AGENTS.md", + "evals/AGENTS.md", + "evals/next-upgrade/evals/future-cache-components-nudge/AGENTS.md", + "evals/next-upgrade/evals/future-cache-components-nudge/CLAUDE.md", + "evals/next-upgrade/evals/future-cache-components-same-version/AGENTS.md", + "evals/next-upgrade/evals/future-cache-components-same-version/CLAUDE.md", + "evals/next-upgrade/evals/future-cache-components/AGENTS.md", + "evals/next-upgrade/evals/future-cache-components/CLAUDE.md", + "evals/next-upgrade/evals/latest-cross-major/AGENTS.md", + "evals/next-upgrade/evals/latest-cross-major/CLAUDE.md", + "evals/next-upgrade/evals/latest-nudge/AGENTS.md", + "evals/next-upgrade/evals/latest-nudge/CLAUDE.md", + "evals/next-upgrade/evals/latest-same-major/AGENTS.md", + "evals/next-upgrade/evals/latest-same-major/CLAUDE.md", + "evals/next-upgrade/evals/security-cross-major/AGENTS.md", + "evals/next-upgrade/evals/security-cross-major/CLAUDE.md", + "evals/next-upgrade/evals/security-duplicate/AGENTS.md", + "evals/next-upgrade/evals/security-duplicate/CLAUDE.md", + "evals/next-upgrade/evals/security-nudge-original-task/AGENTS.md", + "evals/next-upgrade/evals/security-nudge-original-task/CLAUDE.md", + "evals/next-upgrade/evals/security-same-major/AGENTS.md", + "evals/next-upgrade/evals/security-same-major/CLAUDE.md", + "packages/next/AGENTS.md", + "test/AGENTS.md", + "turbopack/AGENTS.md" + ], + "score": 76, + "grade": "C", + "estimated_tokens": 8911, + "duplicate_ratio": 0.0267, + "findings": [ + { + "severity": "warning", + "code": "context-large", + "message": "Persistent instruction context is large", + "file": "AGENTS.md", + "line": null, + "penalty": 6 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 30, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 49, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 272, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 275, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 285, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 292, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 298, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 332, + "penalty": 4 + } + ], + "suppressed_findings": [], + "review_needed": true, + "score_meaning": "lint heuristic, not correctness" + }, + "root_instruction_scope": { + "files": [ + "AGENTS.md" + ], + "score": 76, + "grade": "C", + "estimated_tokens": 7156, + "duplicate_ratio": 0.0, + "findings": [ + { + "severity": "warning", + "code": "context-large", + "message": "Persistent instruction context is large", + "file": "AGENTS.md", + "line": null, + "penalty": 6 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 30, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 49, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 272, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 275, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 285, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 292, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 298, + "penalty": 4 + }, + { + "severity": "warning", + "code": "dead-path", + "message": "Referenced repository path does not exist", + "file": "AGENTS.md", + "line": 332, + "penalty": 4 + } + ], + "suppressed_findings": [], + "review_needed": true, + "score_meaning": "lint heuristic, not correctness" + }, + "comparison_ready": false, + "correctness_metrics": null, + "human_reference_status": "unreviewed", + "ai_status": "separate_existing_pilot_not_imported", + "hybrid_status": "not_run", + "remaining_parity_gaps": [ + "Native discovery may include instruction text not supplied to the AI packet.", + "The root-scoped run controls discovery, but path checks use the full physical snapshot/index; the AI received bounded path facts only.", + "Referenced documents are physically present; the scanner does not semantically review their text unless discovery selects them.", + "No independent human reference annotation or matched comparative model batch exists." + ] +} diff --git a/docs/review-comparison.md b/docs/review-comparison.md index 014f86f..173dc8f 100644 --- a/docs/review-comparison.md +++ b/docs/review-comparison.md @@ -27,8 +27,17 @@ instruction scope can be provided; directory metadata is not actual file content The separate [pinned-source tool probe](../benchmarks/prospective-review-v1/tool-probe/README.md) verifies real source archive bytes against Git objects and distinguishes native discovery from a research-only root scope. Local archive replay requires no -network. Failed acquisition is not a prediction. Its scopes/path evidence remain -different from the bounded AI packet, so it does not bypass the scoring gates. +network. It now retains an actual Linux replay of the frozen 0.22.0 code, +including complete source verification and unscored native/root findings. +Transport failures and a Windows case-sensitivity mismatch are retained, not +counted as predictions. Its scopes/path evidence remain different from the +bounded AI packet, so it does not bypass the scoring gates. + +A [short human review page](../benchmarks/prospective-review-v1/human-pilot/review.md) +presents one unchanged 28-line candidate without tool/model conclusions and +accepts a plain-language decision, rationale and prior-exposure note. This is a +single-reviewer preannotation task, not an adjudicated holdout. Consent is not a +reference label, and no completed annotation is currently retained. ## Run the local baseline diff --git a/tests/test_human_review_packet.py b/tests/test_human_review_packet.py new file mode 100644 index 0000000..e762e62 --- /dev/null +++ b/tests/test_human_review_packet.py @@ -0,0 +1,60 @@ +import hashlib +import json +from pathlib import Path +import re +import sys +import unittest + + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "scripts")) +from run_review_comparison import _digest, load_corpus, review_packet + + +class HumanReviewPreparationTests(unittest.TestCase): + def setUp(self): + self.base = ROOT / "benchmarks" / "prospective-review-v1" + self.corpus = load_corpus(self.base / "context" / "corpus.json") + self.protocol = json.loads((self.base / "human-pilot" / "protocol.json").read_text(encoding="utf-8")) + self.case = next(case for case in self.corpus["cases"] + if case["id"] == self.protocol["selected_case_id"]) + + def test_readable_source_matches_retained_bytes_and_packet(self): + page = (self.base / "human-pilot" / "review.md").read_text(encoding="utf-8") + blocks = re.findall(r"```markdown\n(.*?)\n```", page, re.DOTALL) + self.assertEqual(len(blocks), 1) + source = blocks[0] + "\n" + self.assertEqual(self.case["files"], {"AGENTS.md": source}) + self.assertEqual(len(source.splitlines()), 28) + self.assertEqual(hashlib.sha256(source.encode("utf-8")).hexdigest(), + self.protocol["source_sha256"]) + packet = review_packet({**self.corpus, "cases": [self.case]}) + self.assertEqual(packet["packet_sha256"], self.protocol["canonical_packet_sha256"]) + self.assertEqual(packet["cases"][0]["id"], self.protocol["opaque_case_id"]) + self.assertIn(self.protocol["opaque_case_id"], page) + self.assertEqual(self.case["repository_context"]["tracked_path_facts"], []) + self.assertEqual(self.case["repository_context"]["omitted_documents"], []) + + def test_shortest_input_selection_preserves_frozen_unreviewed_parent(self): + sizes = sorted((sum(len(text.encode("utf-8")) for text in case["files"].values()), case["id"]) + for case in self.corpus["cases"]) + self.assertEqual(sizes[0], (self.protocol["selected_text_bytes"], self.case["id"])) + self.assertEqual(_digest(self.corpus), self.protocol["parent_corpus_sha256"]) + self.assertEqual(self.corpus["tier"], "candidate") + self.assertEqual(self.corpus["label_status"], "unreviewed") + self.assertEqual(self.corpus["labelers"], []) + self.assertTrue(all(case["label"] == "unresolved" for case in self.corpus["cases"])) + + def test_consent_and_preparation_do_not_become_completed_evaluation(self): + self.assertEqual(self.protocol["annotation_status"], "awaiting_annotation") + self.assertEqual(self.protocol["annotations"], []) + self.assertEqual(self.protocol["tool_predictions"], "not_run_for_this_case") + self.assertEqual(self.protocol["model_predictions"], "not_run_for_this_case") + self.assertIsNone(self.protocol["correctness_metrics"]) + page = (self.base / "human-pilot" / "review.md").read_text(encoding="utf-8") + self.assertIn("同意审阅不算已完成标注", page) + self.assertIn("不要运行其中的命令", page) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_pinned_tool_probe.py b/tests/test_pinned_tool_probe.py index 85f72f7..a26788c 100644 --- a/tests/test_pinned_tool_probe.py +++ b/tests/test_pinned_tool_probe.py @@ -328,7 +328,10 @@ def test_retained_acquisition_failures_are_not_real_case_predictions(self): self.assertEqual(len(failures["attempts"]), 7) self.assertTrue(all(row["completed_scans"] == 0 for row in failures["attempts"])) self.assertIsNone(failures["correctness_metrics"]) - self.assertFalse((directory / "report.json").exists()) + # A later successful replay may coexist with these unchanged failed + # attempts; failures themselves must never acquire scan predictions. + self.assertNotIn("native_discovery", failures) + self.assertNotIn("root_instruction_scope", failures) before = json.loads((PROBE.BASE / "tool-probe-protocol.transport-initial.json").read_text(encoding="utf-8")) after = {**self.protocol} for key in ("maximum_download_seconds", "transport_amendment"): diff --git a/tests/test_retained_tool_probe.py b/tests/test_retained_tool_probe.py new file mode 100644 index 0000000..f22113f --- /dev/null +++ b/tests/test_retained_tool_probe.py @@ -0,0 +1,70 @@ +import json +from pathlib import Path +import sys +import unittest + + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "scripts")) +from run_review_comparison import _digest, load_corpus + + +class RetainedToolProbeTests(unittest.TestCase): + def setUp(self): + self.base = ROOT / "benchmarks" / "prospective-review-v1" + self.record = self.read("tool-probe/replay-20261009.json") + self.report = self.read("tool-probe/report.json") + self.protocol = self.read("tool-probe/replay-protocol.json") + + def read(self, name): + return json.loads((self.base / name).read_text(encoding="utf-8")) + + def test_retained_report_and_protocol_are_bound_to_original_inputs(self): + self.assertEqual(_digest(self.report), self.record["report_normalized_json_sha256"]) + self.assertEqual(_digest(self.protocol), self.record["protocol_normalized_json_sha256"]) + self.assertEqual(self.protocol, self.read("tool-probe-protocol.json")) + self.assertEqual(_digest(self.read("tool-probe/failures.json")), + self.record["prior_failures_normalized_json_sha256"]) + for key in ("packet_sha256", "context_corpus_sha256", "tool_source_sha256"): + self.assertEqual(self.report[key], self.protocol[key]) + self.assertEqual(self.report["protocol_sha256"], _digest(self.protocol)) + self.assertEqual(self.report["packet_sha256"], self.read("pilot/review-packet.json")["packet_sha256"]) + self.assertEqual(self.report["context_corpus_sha256"], + _digest(load_corpus(self.base / "context" / "corpus.json"))) + + def test_snapshot_membership_counts_and_declared_limits_remain_consistent(self): + snapshot = self.report["snapshot"] + total = sum(snapshot[key] for key in ("regular_files", "directories", "symlinks")) + self.assertEqual(total, snapshot["tree_entries"]) + self.assertEqual(snapshot["tree_entries"], 50749) + self.assertEqual(snapshot["canonical_tree_object_sha"], self.protocol["canonical_tree_object_sha"]) + self.assertEqual(self.report["archive"]["archive_bytes"], 51988413) + self.assertLessEqual(self.report["archive"]["archive_bytes"], self.protocol["maximum_archive_bytes"]) + self.assertLessEqual(snapshot["verified_blob_bytes"], self.protocol["maximum_source_bytes"]) + self.assertEqual(self.report["input_verification"]["supplied_files_verified"], 11) + self.assertEqual(self.report["input_verification"]["physical_path_facts_checked"], 420) + self.assertEqual(self.report["input_verification"]["unknown_path_facts_not_asserted"], 36) + + def test_unscored_scope_observations_never_become_correctness_metrics(self): + self.assertEqual(self.report["status"], "actual_tool_probe_unscored") + self.assertEqual(self.report["package_version"], "0.22.0") + self.assertEqual(self.report["host_platform"], "Linux") + self.assertFalse(self.report["comparison_ready"]) + self.assertIsNone(self.report["correctness_metrics"]) + self.assertEqual(self.report["human_reference_status"], "unreviewed") + self.assertEqual(self.report["hybrid_status"], "not_run") + self.assertEqual(len(self.report["native_discovery"]["files"]), 27) + self.assertEqual(self.report["root_instruction_scope"]["files"], ["AGENTS.md"]) + for scope in ("native_discovery", "root_instruction_scope"): + observation = self.report[scope] + self.assertEqual(len(observation["findings"]), 9) + self.assertNotIn("root", observation) + self.assertTrue(observation["review_needed"]) + self.assertIsNone(self.record["correctness_metrics"]) + self.assertFalse(self.record["comparison_ready"]) + self.assertEqual([attempt["completed_scans"] for attempt in self.record["attempts"]], [0, 0, 0, 1]) + self.assertFalse(self.record["attempts"][2]["changed_case_or_protocol"]) + + +if __name__ == "__main__": + unittest.main()