fix(validate-doc-claims): name repo-scoped search base in path-not-found flag - #1552
Amanjyot-P wants to merge 2 commits into
Conversation
…und flag The 'not found in working tree' flag for a cited path implied a global search when the check only looks in one repo. An agent reading the flag could not distinguish 'this citation is wrong' from 'this citation points somewhere this check did not look' (e.g. a second docs store in another repo on the same machine). Name the search base (repo root or upstream ref) in the flag text and state that the check is repo-scoped, so the agent knows to check other stores before treating the citation as wrong. Ref: EveryInc#1545
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a121a1d03f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| + ". This check only looks in this repository; verify other " | ||
| "repos or stores before treating the citation as wrong. Fix " |
There was a problem hiding this comment.
Describe non-git search bases without calling them repositories
When the document and current directory are outside Git, base is merely os.getcwd(), but this new sentence still calls it “this repository.” Reproducing from a temporary non-git directory yields both INFO: not a git repository and This check only looks in this repository in the same report, undermining the clarification this change intends to provide. Make the scope description conditional on in_git, referring to the named directory rather than a repository in this supported fallback mode.
Useful? React with 👍 / 👎.
Amanjyot-P
left a comment
There was a problem hiding this comment.
OMP Swarm PR Review
Verdict: Ready with fixes — P0: 0 · P1: 0 · P2: 0 · P3: 0
Ready with fixes: the implementation is consistent across both validator copies, but the changed test does not assert the upstream-present suffix or the no-upstream branch. The validator independently confirmed this testing gap; it is retained as coverage debt rather than a primary P2 finding because the quote-the-line gate reduced confidence to 50 and the testing soft-bucket rule applies.
No confirmed PR-introduced findings survived evidence, scope, deduplication, cross-model, and validator gates.
Coverage: correctness, security, testing, maintainability, project_standards; independent validator completed before lead synthesis.
Skipped: performance (no performance-sensitive runtime path); adversarial (no high-risk adversarial trigger); api_contract (no external contract surface); previous_comments (no prior PR feedback).
Audit details
- Five selected reviewer artifacts were present, parseable, and matched one-for-one to the selected roster.
- The testing candidate was CONFIRMED by the validator with confidence 100, but its quoted evidence was not the exact motivating changed line; the quote-the-line gate reduced it from 75 to 50.
- The single-reviewer P2 testing-only candidate was moved to the testing-gap soft bucket and is therefore not emitted as a primary finding.
- No confidence promotion occurred: the candidate was reported only by testing, so there was no independent corroboration from a different model family.
- Validator batch: 1 verdict, 1 CONFIRMED, 0 REJECTED, 0 infrastructure failures.
- Suppressed/demoted: 1 candidate at confidence 50. No P0, P1, or retained P2 findings remain.
The complete audit remains in FINAL_REVIEW.md in the OMP run workspace.
|
@Amanjyot-P please look at the codex feedback. If you've got agents autonomously working, you need to have them review the feedback please. |
…k saw The flag text now says the search is repo-scoped, but the Step 1 table is what the adjudicating agent actually reads to decide what to do, and its row still named one conclusion — "typo, or drafted from memory" — with "fix the citation or remove the claim". That is the global reading this PR exists to stop asserting, left sitting in the deciding position. The row now states what the observation supports and routes the agent to check the other stores and the citing sentence before treating the citation as wrong. Issue EveryInc#1545 measured this: across 767 backticked .md citations, 17 did not resolve and none was a mistake — six were correct paths on another machine, three pointed outside the search roots, and eight said in the same sentence that the target was gone or external. The manual-fallback sentence below the table carries the same global framing and is deliberately untouched here: it was rewritten on main in EveryInc#1608, so editing it on this branch would hand this PR a conflict. Claude-Session: https://claude.ai/code/session_01HSahUJjXzZpJt5ztL7CUyt
|
Pushed one commit to this branch (004037c) — flagging it since it's your PR. Your change makes the flag text disclose that the search is repo-scoped. The gap it left is one layer over: The row now states what the observation actually supports and routes the reader to check the other stores and the citing sentence before treating the citation as wrong. #1545's own numbers are what justify it: across 767 backticked Two things I deliberately did not touch:
One small thing worth your call, not a blocker: the flag now embeds an absolute machine-local path. That's exactly what the issue asked for and it's right for terminal output — just noting that this output is sometimes read by an agent that then writes text into a learning doc, so a repo-relative form would carry the same meaning with less leakage. Fine either way. Tests and |
What
validate-doc-claims.pyflags a cited repo path with "not found in working tree" (or "working tree ororigin/main") without saying where it looked. An agent reading that flag can't tell "this citation is wrong" from "this citation points somewhere this check did not look" — e.g. a second docs store in another repo on the same machine.Fix
Name the search base (repo root, plus the upstream ref when one exists) in the flag text and state the check is repo-scoped:
Applied identically to both duplicated copies of the script (
skills/ce-compound/scripts/validate-doc-claims.pyandskills/ce-compound-refresh/scripts/validate-doc-claims.py), per the AGENTS.md duplication rule the existing test suite already enforces.Scope
Smallest correct fix per the issue: wording only, no cross-repo resolution (out of scope per the issue, tracked separately by #1505).
Tests
tests/doc-claims-validator.test.ts's "flags a cited path that exists nowhere" test to assert the new repo-scoped wording (search base + "This check only looks in this repository").bun test tests/doc-claims-validator.test.ts— 53/53 pass.bun run test— full suite passes except 5 pre-existing, unrelated timing-flaky tests intests/ce-babysit-pr-snapshot.test.ts/tests/skills/ce-code-review-cross-model-routes.test.ts, confirmed failing identically on a clean checkout of upstreammainbefore this change.Ref: #1545