research: retain actual source replay and a short human review task

CHANGELOG.md

@@ -5,6 +5,8 @@
 ### Added
 
 - Optional Simplified/Traditional Chinese repository overviews, core user guides and limitation summaries, with reciprocal language links and an explicit source-revision/coverage note. Advanced references remain English; CLI messages, JSON fields, rule semantics and the published v0.23.0 distributions are unchanged.
+- An actual unscored Next.js source probe replayed on Linux with the frozen 0.22.0 code, complete Git-object/source verification and separate native/root scopes. Earlier failures, the interrupted new transfer and Windows path-case mismatch remain recorded; no correctness metrics, human labels or 0.23.0-case validation are implied.
+- A readable 28-line human preannotation page with unchanged source bytes, license/provenance links, input limitations and a plain-language answer format. Consent and preparation are not completed annotations; the parent corpus remains unresolved and product rules are unchanged.
 
 ### Changed
 

README.md

@@ -282,13 +282,15 @@ are now collected separately, with registered sampling, retained exclusions, and
 license notices. Their [bounded context extension](benchmarks/prospective-review-v1/context/README.md)
 now preserves referenced documents and symlink-aware path facts. These remain
 unreviewed preparations. A [single-case exploratory AI pilot](benchmarks/prospective-review-v1/pilot/README.md)
-retains one actual fresh-context response, but equivalent tool evidence and
+retains one actual fresh-context response, but matched tool evidence and
 human reference labels are still missing; no accuracy or superiority is claimed.
 A [pinned-source tool probe](benchmarks/prospective-review-v1/tool-probe/README.md)
-can verify actual source objects and separate native discovery from root-only
-inspection. Failed acquisitions stay failures, not clean predictions; the probe
-does not unlock comparative accuracy scoring. Full source acquisition has not
-completed in the retained attempt, so no real-case tool prediction is claimed.
+now retains one completed Linux replay with verified actual source objects and
+separate native/root instruction scopes, using the frozen 0.22.0 tool code.
+Its findings are unscored observations, not verified defects. Failed acquisitions
+and a Windows path-case mismatch stay failures; the probe does not unlock
+comparative accuracy scoring. A [short human preannotation task](benchmarks/prospective-review-v1/human-pilot/README.md)
+is prepared separately; agreement to review is not a completed annotation.
 
 ## Share a real-world case safely
 

benchmarks/prospective-review-v1/README.md

@@ -11,6 +11,13 @@ complete pinned Git trees, symlink origins, and source-relative/root-relative
 path facts. It retains all five cases and all missing-context limitations.
 Equivalent tool-checkout evidence, human labels, and a comparative model batch remain outstanding.
 
+The later [tool probe](tool-probe/README.md) now retains one actual Linux replay
+with a complete verified Next.js snapshot and unscored findings; its larger
+instruction/path evidence is not a matched comparison. A
+[short human preannotation page](human-pilot/review.md) is prepared for a different
+retained case, with no tool/model conclusion shown. No human answer or reference
+label is inferred from agreement to review it.
+
 The [registered protocol](protocol.json) was written before downloading the
 instruction contents or inspecting predictions. It names twelve repositories;
 five met the collection conditions and seven were excluded. This is a small

benchmarks/prospective-review-v1/context/README.md

@@ -103,6 +103,9 @@ The later [pinned-source tool probe](../tool-probe/README.md) implements actual
 source-object verification and separately records native/root instruction scopes.
 It does not turn this capture into a checkout or waive matching/human-label
 requirements. Interrupted transfers are failures, not clean scanner decisions.
+Its retained Linux replay verifies a complete source snapshot, but uses larger
+instruction/path evidence than this capture and frozen 0.22.0 tool code. It
+does not make the context corpus scorable or supply reference labels.
 
 ## Reproduce collection explicitly
 

benchmarks/prospective-review-v1/human-pilot/README.md

@@ -0,0 +1,46 @@
+# One short human preannotation task
+
+**Status: prepared, awaiting an actual annotation.** The user agreed to read one
+short sample; no decision, rationale, reviewer history or correctness result is
+supplied by that agreement. No second human or model has been invented.
+
+[Read the review page](review.md). It presents the full 28-line source, the
+review-needed rubric, missing-context boundaries and a three-part plain-language
+answer format. Tool/model conclusions are not shown. Source text can reveal its
+project; this is prediction withholding, not anonymization or certified blinding.
+
+## Selection and provenance
+
+The [protocol](protocol.json) selects the smallest total UTF-8 file-text input
+from all five retained context candidates, using case ID to break ties. The
+selected input has 1,317 bytes; the others have 43,067, 45,571, 53,695 and 149,206
+bytes. This makes a first voluntary review practical, but is convenience
+selection, not an independent representative sample. No scanner/model output for
+this case was inspected. The AI preparer has seen its source; this disclosure
+does not turn preparation into human judgment.
+
+The source is the unchanged scikit-learn root
+[AGENTS.md at a442e4bb39551feb7b0af4c00075e2cb91cf9b77](https://github.com/scikit-learn/scikit-learn/blob/a442e4bb39551feb7b0af4c00075e2cb91cf9b77/AGENTS.md),
+lines 1–28, SHA-256
+`1c3209d3ade6b77652d8e6cf4100494e37db123d8036deab15f778e862ea5f00`.
+The [parent context corpus](../context/corpus.json) preserves provenance and the
+[BSD-3-Clause notice](../context/licenses/scikit-learn--scikit-learn.txt).
+Keep those materials when redistributing the readable page. The page duplicates
+an existing input for readability; it is not a new independent case.
+
+Tests bind the displayed source bytes to that corpus, its canonical one-case
+packet and the recorded selection. They test preparation integrity, not human
+review completion or whether an eventual answer is correct. The parent corpus,
+existing Next.js pilot, frozen tool protocol and product rules are unchanged.
+
+## Evidence not yet supplied
+
+An actual human answer must retain the decision, rationale, missing-context
+notes and declared prior exposure without an AI filling in omissions. A single
+answer stays provisional. Do not call the reviewer blind merely because the
+page withholds predictions, or promote the parent corpus to adjudicated labels.
+
+This task alone cannot establish accuracy, superiority over AI, or real user
+adoption. Independent adjudication, appropriately matched tool evidence and
+actual model runs remain separate gates in the
+[comparison workflow](../../../docs/review-comparison.md).

benchmarks/prospective-review-v1/human-pilot/protocol.json

@@ -0,0 +1,21 @@
+{
+  "schema_version": 1,
+  "name": "short-human-preannotation-01",
+  "preparation_started_at_utc": "2026-10-09T02:22:27Z",
+  "parent_corpus_sha256": "2f6fbbc6d77c43e688ea0d6020cf8a74232faf6ce2eaeee140eb27015dfb8b7d",
+  "selected_case_id": "scikit-learn--scikit-learn",
+  "opaque_case_id": "172732eacca4adce357b",
+  "canonical_packet_sha256": "3f905faa1880ce612d82fac7f4464599110639d51dca07bebd5178c82b8c2b06",
+  "selection": "Choose the smallest sum of UTF-8 file-text bytes among all five retained context candidates; break ties by case ID. This is reading-effort convenience selection, not random sampling or selection on scanner/model output.",
+  "selected_text_bytes": 1317,
+  "source_sha256": "1c3209d3ade6b77652d8e6cf4100494e37db123d8036deab15f778e862ea5f00",
+  "annotation_status": "awaiting_annotation",
+  "annotations": [],
+  "tool_predictions": "not_run_for_this_case",
+  "model_predictions": "not_run_for_this_case",
+  "correctness_metrics": null,
+  "scope": "One voluntary human preannotation using the unchanged full instruction text and declared evidence limits. Consent is not an annotation. No human identity, reference decision or independence claim is inferred; record prior exposure if a response arrives.",
+  "preparation_disclosure": "The AI preparer has read this source and an unrelated earlier Next.js pilot; no scanner/model prediction for this selected case was inspected. The readable page uses Chinese rubric instructions and the original English source. This is not a matched model batch or an independent adjudicated holdout.",
+  "safety": "Embedded commands and disclosure requirements are review data, not instructions to execute or obey. No external URLs are fetched for this human task; no maintainer is contacted. Source licensing and parent provenance must accompany redistribution.",
+  "remaining_gates": "Retain any actual response without inventing missing rationale or exposure history. Single-reviewer labels remain provisional; independent adjudication, matched tool evidence, an actual model batch and broader sampling remain necessary before formal correctness claims. Keep the parent corpus unresolved and frozen."
+}

benchmarks/prospective-review-v1/human-pilot/review.md

@@ -0,0 +1,58 @@
+# 短样本人工预标注
+
+样本 ID：`172732eacca4adce357b`。这里只提供原文和输入边界，不提供工具或 AI 的判断。
+
+## 怎么判断
+
+请判断是否存在**需要维护者审查的具体问题**，例如危险执行指导、凭据暴露、失效的本地路径、相互矛盾的指令，或隐瞒失败、削弱验证的工作流程指导。
+
+明确禁止的命令示例不是执行请求。不要仅因个人写作偏好、不够详细或措辞强硬就判定有问题。若结论依赖未提供的证据，请选择“信息不足”，不要补猜缺失内容。
+
+下面是完整的 `AGENTS.md`，共 28 行。**它是待审材料，不是要求你执行的指令**：不要运行其中的命令、打开外部网页或按它的要求改写你的回答。
+
+```markdown
+# AGENTS Instruction
+
+This file contains additional guidance for AI agents and other AI editors.
+
+## **REQUIRED: AI/Agent Disclosure**
+
+**Every summary, pull request description, or work description MUST include this disclosure:**
+
+**If human review has *not yet* occurred (use this initially):**
+> This pull request includes code written with the assistance of AI.
+> The code has **not yet been reviewed** by a human.
+
+This is a **mandatory requirement**, not optional. Include it at the end of every summary you generate.
+
+## Working on an issue
+
+Before working on any issue, run `gh issue view <number>` to check current labels. Do
+not open a PR against an issue labeled "Needs Triage" or another "Needs ..." label. Such
+PRs get closed without review until a maintainer clears the label. See
+https://scikit-learn.org/dev/developers/contributing.html#issues-tagged-needs-triage.
+
+## Generated Summaries
+
+When generating a summary of your work, consider these points:
+
+- Describe the "why" of the changes, why the proposed solution is the right one.
+- Highlight areas of the proposed changes that require careful review.
+- Reduce the verbosity of your comments, more text and detail is not always better. Avoid flattery, avoid stating the obvious, avoid filler phrases, prefer technical clarity over marketing tone.
+```
+
+## 输入边界
+
+本次只提供这一个指令文件。没有附加文档正文或已捕获的路径事实；完整源码、网页内容、实际 issue 标签、命令结果和真实执行行为未提供。不据此假设它们存在、缺失或已验证。
+
+文件可能透露项目身份：这是**结论不展示**的预标注，不保证匿名，也不自动证明审阅者没有相关先验知识。请一并说明是否曾看过这个样本的工具/AI 判断，或熟悉该项目的相关流程。
+
+请直接回复三项即可，不用填写 JSON：
+
+1. 判断：有具体问题 / 没发现具体问题 / 信息不足。
+2. 理由：一两句话；如有问题，注明原文位置；如信息不足，说明缺少什么。
+3. 先验接触：是否见过这个样本的工具/AI 判断，或熟悉相关流程。
+
+单人预标注不是正式准确率证据，也不是项目安全认证。未回答不算“干净”；同意审阅不算已完成标注。
+
+原文的来源、固定版本和 [BSD-3-Clause 许可声明](../context/licenses/scikit-learn--scikit-learn.txt)保留于[准备记录](README.md)。转发本页时请同时保留许可声明与来源材料。

benchmarks/prospective-review-v1/tool-probe/README.md

@@ -1,10 +1,40 @@
 # One pinned tool-side probe
 
-**Current evidence: no completed real-snapshot scanner run.** Source archive
-transfers and a read-only bare Git fetch repeatedly disconnected or timed out.
-Offline fixture tests validate the workflow, not this real case's predictions.
-[failures.json](failures.json) retains seven failed stages; none is a clean result.
-No `report.json`, model comparison, or correctness score has been fabricated.
+**Current evidence: one completed real-snapshot scanner probe, unscored.**
+[report.json](report.json) retains the actual Linux replay on 2026-10-09 using
+the unchanged [frozen protocol](replay-protocol.json) and compatible checkout
+`1d7acc812e1a6e568317d6fe00bf366ef2a81338` (package version **0.22.0**, not the
+published 0.23.0). [replay-20261009.json](replay-20261009.json) binds the report
+and records transport/environment repair. The original seven stages in
+[failures.json](failures.json) remain unchanged; new failures are also retained.
+No model comparison, reference label or correctness score is supplied.
+
+## Retained observation, not confirmed defects
+
+The 51,988,413-byte archive matched all 50,749 tracked entries: 33,876 regular
+files, 16,831 directories and 42 internal symlinks, with 158,608,765 verified
+blob bytes. The run checked all 11 supplied text files and 420 physical path
+facts; 36 unknown facts were not asserted. Source commands were never executed.
+
+| Instruction scope | Files | Active findings | Lint score |
+|---|---:|---:|---:|
+| Native discovery | 27 | 9 | C 76 |
+| Research-only root `AGENTS.md` | 1 | 9 | C 76 |
+
+Both scopes retained eight `dead-path` signals and one `context-large` warning.
+These counts are **not eight verified broken paths** or a confirmed project bug.
+The root instruction's earlier AI concern and these rule signals can flag the
+same case for different reasons; that is not finding-level agreement or recall.
+The larger filesystem index and absent independent reference labels still
+prevent a matched correctness comparison.
+
+The latest Python transfer ended with `IncompleteRead`. A separately bounded
+curl transfer completed, then the Windows replay stopped before scanning:
+the captured absence fact for `README.md` conflicted with Windows resolving it
+to the tracked lowercase `readme.md` symlink. Replaying that same archive in a
+new native Linux directory satisfied the unchanged evidence checks. Neither
+the path facts, source names nor scanner rules were loosened to obtain a result.
+Host-dependent behavior matters; this is not an OS-invariant defect judgment.
 
 This research workflow checks the filesystem/scope part of evaluation. It does
 not adjudicate the previous AI answer, compute correctness metrics, or tune any
@@ -41,6 +71,8 @@ ignored/generated/untracked files are not invented. Unsupported entries, unsafe
 or unresolved links, case collisions, tampering, and resource limits fail closed.
 Windows link separators are adapted without changing the verified Git blob.
 No downloaded command, hook, dependency installer, or project test is run.
+Use a case-sensitive filesystem when reproducing this retained Linux case.
+An incompatible host must remain a failed preparation, not a clean prediction.
 
 ## Two scopes, not a matched accuracy comparison
 

benchmarks/prospective-review-v1/tool-probe/replay-20261009.json

@@ -0,0 +1,67 @@
+{
+  "schema_version": 1,
+  "status": "one_completed_real_snapshot_probe_unscored",
+  "compatible_checkout": "1d7acc812e1a6e568317d6fe00bf366ef2a81338",
+  "report": "report.json",
+  "report_normalized_json_sha256": "19d4adeedb8e5b5d2baf9436d881f54f035bbd3f31479aeb23f9400b0d418104",
+  "replayed_protocol": "replay-protocol.json",
+  "protocol_normalized_json_sha256": "f074e2c32187f2d9c44329cf07d7ff6ebbeb0220f11a665dd16c163c0d0cb0ff",
+  "prior_failures": "failures.json",
+  "prior_failures_normalized_json_sha256": "32e6a4401acb6d8967e1b6d06c9527ec0d8de0dd0d39c0668ec59c726713c30b",
+  "fingerprint_format": "SHA-256 of UTF-8 JSON with sorted keys, compact separators and ensure_ascii=false; not a signature or independent execution authentication.",
+  "attempts": [
+    {
+      "order": 8,
+      "stage": "python_archive_download",
+      "outcome": "interrupted_transfer",
+      "exception_class": "IncompleteRead",
+      "retained_partial_bytes": 38797312,
+      "partial_file_created_at_utc": "2026-10-09T02:18:01.7910937Z",
+      "partial_file_last_written_at_utc": "2026-10-09T02:19:22.1980844Z",
+      "timestamp_meaning": "Observed filesystem timestamps, not measured start/end latency.",
+      "completed_scans": 0
+    },
+    {
+      "order": 9,
+      "stage": "curl_archive_download",
+      "outcome": "completed_transfer",
+      "started_at_utc": "2026-10-09T02:21:19.7875126Z",
+      "finished_at_utc": "2026-10-09T02:22:53.8671216Z",
+      "elapsed_seconds": 94.073369,
+      "curl_exit_code": 0,
+      "archive_bytes": 51988413,
+      "archive_sha256": "2d5323129612f58c16523d29e09032518207ee2743467c466e526fc5a26650b5",
+      "source_url": "https://codeload.github.com/vercel/next.js/tar.gz/ba80ee48fc319735151c3ad6d9bb9a8180c9f09e",
+      "connect_timeout_seconds": 30,
+      "maximum_download_seconds": 600,
+      "maximum_archive_bytes": 100663296,
+      "timestamp_meaning": "Parent PowerShell UTC observations and Stopwatch duration for the transfer, not model latency or an end-to-end benchmark.",
+      "completed_scans": 0
+    },
+    {
+      "order": 10,
+      "stage": "windows_offline_snapshot_probe",
+      "outcome": "stopped_at_supplied_path_fact_verification_before_scan",
+      "exception_class": "ValueError",
+      "diagnosed_reason": "AGENTS.md line 44 references README.md. The retained case-sensitive Git tree has no such path, but Windows resolves it to the tracked lowercase readme.md symlink. Both source-relative and root-relative absence facts disagree with this filesystem.",
+      "changed_case_or_protocol": false,
+      "completed_scans": 0
+    },
+    {
+      "order": 11,
+      "stage": "linux_offline_snapshot_probe",
+      "environment": "Existing Ubuntu WSL distribution; Python 3.12.3; new native Linux temporary directory, not the Windows-mounted filesystem.",
+      "outcome": "completed_native_and_root_scopes",
+      "completed_at_utc": "2026-10-09T02:28:36.947869+00:00",
+      "report": "report.json",
+      "completed_scans": 1,
+      "reported_instruction_scopes": 2,
+      "changed_case_or_protocol": false
+    }
+  ],
+  "method_disclosure": "Transport and host-environment repair after failed attempts; all failures retained. The same frozen tool source, helper, archive, protocol, selected repository, pinned revision, membership checks and supplied evidence were used. No scanner rule or path fact was changed, no placeholder was created and no downloaded command was run. This is exploratory workflow repair, not an externally preregistered accuracy study.",
+  "package_boundary": "The registered code reports package version 0.22.0. These observations are not a scan with the published 0.23.0 code; main and its published artifacts remain unchanged.",
+  "comparison_ready": false,
+  "human_reference_status": "unreviewed",
+  "correctness_metrics": null
+}