Skip to content

fix(validate-doc-claims): name repo-scoped search base in path-not-found flag - #1552

Open
Amanjyot-P wants to merge 2 commits into
EveryInc:mainfrom
Amanjyot-P:fix-1545-repo-scoped-path-validation-message
Open

Amanjyot-P wants to merge 2 commits into
EveryInc:mainfrom
Amanjyot-P:fix-1545-repo-scoped-path-validation-message

Conversation

@Amanjyot-P

Copy link
Copy Markdown

What

validate-doc-claims.py flags a cited repo path with "not found in working tree" (or "working tree or origin/main") without saying where it looked. An agent reading that flag can't tell "this citation is wrong" from "this citation points somewhere this check did not look" — e.g. a second docs store in another repo on the same machine.

Fix

Name the search base (repo root, plus the upstream ref when one exists) in the flag text and state the check is repo-scoped:

FLAG path `x.md` (line N) — not found under /path/to/repo or origin/main. This
check only looks in this repository; verify other repos or stores before
treating the citation as wrong. Fix the citation, or annotate it as
historical (e.g. removed by this fix).

Applied identically to both duplicated copies of the script (skills/ce-compound/scripts/validate-doc-claims.py and skills/ce-compound-refresh/scripts/validate-doc-claims.py), per the AGENTS.md duplication rule the existing test suite already enforces.

Scope

Smallest correct fix per the issue: wording only, no cross-repo resolution (out of scope per the issue, tracked separately by #1505).

Tests

  • Updated tests/doc-claims-validator.test.ts's "flags a cited path that exists nowhere" test to assert the new repo-scoped wording (search base + "This check only looks in this repository").
  • bun test tests/doc-claims-validator.test.ts — 53/53 pass.
  • bun run test — full suite passes except 5 pre-existing, unrelated timing-flaky tests in tests/ce-babysit-pr-snapshot.test.ts / tests/skills/ce-code-review-cross-model-routes.test.ts, confirmed failing identically on a clean checkout of upstream main before this change.

Ref: #1545

…und flag

The 'not found in working tree' flag for a cited path implied a global
search when the check only looks in one repo. An agent reading the flag
could not distinguish 'this citation is wrong' from 'this citation points
somewhere this check did not look' (e.g. a second docs store in another
repo on the same machine).

Name the search base (repo root or upstream ref) in the flag text and
state that the check is repo-scoped, so the agent knows to check other
stores before treating the citation as wrong.

Ref: EveryInc#1545

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a121a1d03f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +278 to +279
+ ". This check only looks in this repository; verify other "
"repos or stores before treating the citation as wrong. Fix "

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Describe non-git search bases without calling them repositories

When the document and current directory are outside Git, base is merely os.getcwd(), but this new sentence still calls it “this repository.” Reproducing from a temporary non-git directory yields both INFO: not a git repository and This check only looks in this repository in the same report, undermining the clarification this change intends to provide. Make the scope description conditional on in_git, referring to the named directory rather than a repository in this supported fallback mode.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Amanjyot-P take a look at this

@Amanjyot-P Amanjyot-P left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OMP Swarm PR Review

Verdict: Ready with fixes — P0: 0 · P1: 0 · P2: 0 · P3: 0

Ready with fixes: the implementation is consistent across both validator copies, but the changed test does not assert the upstream-present suffix or the no-upstream branch. The validator independently confirmed this testing gap; it is retained as coverage debt rather than a primary P2 finding because the quote-the-line gate reduced confidence to 50 and the testing soft-bucket rule applies.

No confirmed PR-introduced findings survived evidence, scope, deduplication, cross-model, and validator gates.

Coverage: correctness, security, testing, maintainability, project_standards; independent validator completed before lead synthesis.
Skipped: performance (no performance-sensitive runtime path); adversarial (no high-risk adversarial trigger); api_contract (no external contract surface); previous_comments (no prior PR feedback).

Audit details
  • Five selected reviewer artifacts were present, parseable, and matched one-for-one to the selected roster.
  • The testing candidate was CONFIRMED by the validator with confidence 100, but its quoted evidence was not the exact motivating changed line; the quote-the-line gate reduced it from 75 to 50.
  • The single-reviewer P2 testing-only candidate was moved to the testing-gap soft bucket and is therefore not emitted as a primary finding.
  • No confidence promotion occurred: the candidate was reported only by testing, so there was no independent corroboration from a different model family.
  • Validator batch: 1 verdict, 1 CONFIRMED, 0 REJECTED, 0 infrastructure failures.
  • Suppressed/demoted: 1 candidate at confidence 50. No P0, P1, or retained P2 findings remain.

The complete audit remains in FINAL_REVIEW.md in the OMP run workspace.

@tmchow

tmchow commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

@Amanjyot-P please look at the codex feedback. If you've got agents autonomously working, you need to have them review the feedback please.

…k saw

The flag text now says the search is repo-scoped, but the Step 1 table
is what the adjudicating agent actually reads to decide what to do, and
its row still named one conclusion — "typo, or drafted from memory" —
with "fix the citation or remove the claim". That is the global reading
this PR exists to stop asserting, left sitting in the deciding position.

The row now states what the observation supports and routes the agent to
check the other stores and the citing sentence before treating the
citation as wrong. Issue EveryInc#1545 measured this: across 767 backticked .md
citations, 17 did not resolve and none was a mistake — six were correct
paths on another machine, three pointed outside the search roots, and
eight said in the same sentence that the target was gone or external.

The manual-fallback sentence below the table carries the same global
framing and is deliberately untouched here: it was rewritten on main in
EveryInc#1608, so editing it on this branch would hand this PR a conflict.

Claude-Session: https://claude.ai/code/session_01HSahUJjXzZpJt5ztL7CUyt
@tmchow

tmchow commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Pushed one commit to this branch (004037c) — flagging it since it's your PR.

Your change makes the flag text disclose that the search is repo-scoped. The gap it left is one layer over: grounding-validation.md's Step 1 table is what the adjudicating agent actually reads to decide what to do with a flag, and its row still said the likely meaning was "Typo, or drafted from memory" with the resolution "Fix the citation or remove the claim." That's the global reading this PR exists to stop asserting, sitting in the deciding position — so an agent could read your new caveat and then apply the old conclusion straight from the table.

The row now states what the observation actually supports and routes the reader to check the other stores and the citing sentence before treating the citation as wrong. #1545's own numbers are what justify it: across 767 backticked .md citations, 17 didn't resolve and none was a mistake — six were correct paths on another machine, three pointed outside the search roots, and eight said in the same sentence that the target was gone or external.

Two things I deliberately did not touch:

One small thing worth your call, not a blocker: the flag now embeds an absolute machine-local path. That's exactly what the issue asked for and it's right for terminal output — just noting that this output is sometimes read by an agent that then writes text into a learning doc, so a repo-relative form would carry the same meaning with less leakage. Fine either way.

Tests and release:validate pass on the branch with the commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants