Fix framework-aware evidence remediation

[pengfei-threemoonslab]

Summary

• make the rank-1 incomplete_surface action framework-aware when the source has an explicit inventory route
• project the authoritative evidence-gap action consistently in console scan output, GitHub summaries, and text-mode verify
• present the manual evidence work first and the exact rerun command as the final executable action
• collapse duplicate semantic/extraction inventory guidance into one verifier fix-task instruction
• replace the synthetic semantic fixture with a real Google ADK extraction and verifier regression
• update the Unreleased changelog, current agent contract, generated llms-full.txt, and adopter-facing remediation docs

Root cause

The short-form renderers were initially corrected to read evidence_gaps[0].next_action, but the first gap for every low-confidence framework tool was incomplete_surface, whose action was still generic. The framework-keyed declare_tool_inventory action appeared later as low_confidence_tool, so the Google ADK route was structurally unreachable.

Impact

For supported frameworks, the decision engine now leads with the generated suggested-inventory.json artifact and the exact <framework>.tool_inventories manifest route. Console, GitHub, verifier, fix-task, handoff, and packet consumers inherit the same authoritative action. Human work precedes a separate final Run: command, and verifier instructions no longer prescribe the same inventory repair twice. Unidentified or unsupported source shapes retain generic source guidance; no binding, effect, authority, approval, or runtime evidence is invented.

Validation

• full non-packaging suite: PYTHONPATH=src pytest -q -n auto --ignore=tests/test_packaging.py
• isolated packaging suite: pytest tests/test_packaging.py -q (8 passed)
• focused renderer/decision/verifier/packet/handoff/agent-boundary suite
• generated-contract check for llms-full.txt
• Ruff and git diff --check
• committed-ref Agents Shipgate verification against origin/main: control.state=complete, merge verdict mergeable, release decision passed

Closes #318

[pengfei-threemoonslab]

Engineering review

Verdict: request changes. The direction is right and the projection change is a genuine improvement, but the fix does not reach its own headline case. On a real Google ADK repository the output is still the generic MCP/OpenAPI sentence, and the new regression test is green only because it hand-builds a report the engine cannot produce.

Base is clean (merge-base == origin/main, 0 commits behind), single commit, full suite green (-n auto, no failures), ruff clean.

---

BLOCKER — the framework-aware branch is structurally unreachable

primary_evidence_remediation_text renders evidence_gaps[0], but evidence_gaps is assembled as [*binding_gaps, *semantic_gaps, *policy_gaps, *_evidence_gaps(...)] (ci/release_decision.py:233). The only framework-keyed action in the engine — declare_tool_inventory, the one that names google_adk.tool_inventories — is produced in _evidence_gaps, the last group.

It can never be first, because the two gaps are co-generated by the same condition:

• low_confidence_tool fires on extraction_confidence != "high" (ci/release_decision.py:925)
• incomplete_surface fires on extraction_confidence != "high" or not _surface_is_complete(tool) (core/semantic_assessment.py:567)

So low_confidence_tool ⇒ incomplete_surface, and incomplete_surface's action is the generic sentence (ci/release_decision.py:725-735).

Reproduced end-to-end on a minimal ADK workspace (LlmAgent + one FunctionTool), same workspace both sides:

	Improve evidence: on a real ADK repo at insufficient_evidence
origin/main	provide MCP export, OpenAPI spec, explicit local tool inventory, or a broader OpenAI SDK source path; then rerun scan.
this PR	Run: agents-shipgate verify … . Provide a complete MCP export, OpenAPI spec, or reviewed explicit tool inventory, then rerun verification Target: shipgate.yaml#tool_sources.

Neither names google_adk.tool_inventories or suggested-inventory.json. The actual gap order on that scan:

[0] incomplete_surface         -> provide_complete_inventory  (generic)   <- rendered
[1] missing_authority_evidence -> declare_action_authority
[2] missing_effect_evidence    -> declare_action_effect
[3] low_confidence_tool        -> declare_tool_inventory  (google_adk.tool_inventories)

I then declared exactly what the engine asked for (effect + authority under action_surface.actions) and re-scanned — incomplete_surface survives (it keys off extraction confidence, not declarations), stays at index 0, output unchanged. There is no adopter state that reaches the ADK route.

Acceptance criterion 1 of #318 ("A Google ADK-only result leads with the generated local inventory/declaration artifact and the exact google_adk manifest/evidence route") is not met. What the PR does deliver is removing the incorrect openai_agents_sdk mention and centralizing the string — real, but not the filed bug.

BLOCKER — the regression test asserts an engine-impossible state

tests/test_ci.py:35-40 hand-builds a ToolSemanticAssessment with no issues and pass_eligible=True while extraction_confidence="medium". Running the real resolver on that same Tool:

resolver issue kinds  : ['incomplete_surface', 'missing_effect_evidence']
resolver pass_eligible: False

assess_tool_semantics makes pass_eligible require extraction_confidence == "high" (core/semantic_assessment.py:112-125) — the fixture contradicts the invariant directly. The test passes because it skipped the resolver, which is exactly the drift it was meant to catch. Drive it through attach_semantic_assessments, or better through a real ADK workspace scan.

Recommended fix

Fix it in the engine, not the projection: _semantic_gap has tool in scope, and _inventory_manifest_key(tool.source_type) already exists ~200 lines below. Making the incomplete_surface action framework-aware fixes console, GitHub summary, fix_task, packet and handoff at once — the one-decision-engine route, and the same move #296 already made for the low-confidence remedy. Keep the projection change; it is correct and should stay.

MEDIUM — sentence-boundary bug in the new helper

report/summary_text.py:59: text.rstrip('.') strips the terminating period and never restores it, so every path-carrying action renders a run-on — visible in the real output above (…rerun verification Target: …). It also eats an ellipsis whole ("See docs..." → "See docs Target: x.json.").

text = f"{text.rstrip('.')}. Target: {action.path}."

Scope — "Closes #318" is premature

#318's last criterion says the adopter-facing terminal output from #327 is the primary surface and "a demoted scan-only message is not considered sufficient." _print_cli_summary is called only from cli/_register_scan.py:414. verify's text output prints five lines and no remediation at all (cli/verify/command.py:426-430); the remediation reaches verify only through fix_task.instructions, which was already framework-aware before this PR (cli/verify/fix_task.py:472). The maintainer comment on #318 scoped this to the two renderers and the PR follows that faithfully — but the diagnosis in that comment was incomplete, and the checkbox list is broader.

Minor

• No ## Unreleased CHANGELOG entry; every recent behavior change in that section has one.
• Doc drift on the retired sentence: README.md:254, docs/quickstart.md:166, STABILITY.md:1105.
• The if not evidence.evidence_gaps fallback is unreachable for current reports — every evidence_below_ie_threshold trigger produces gap rows. The docstring already says "older reports," which is accurate; worth a one-line note that it is legacy-only.

What's good

Centralizing the text in report/summary_text.py next to evidence_coverage_text is the right home; _safe_markdown_text is correctly applied on the GitHub path and correctly absent on the console path; sanitize_report still runs before the render; and dropping the openai_agents_sdk reference is a real correctness win regardless of the rest.

[pengfei-threemoonslab]

Engineering review — second pass (b27d87b1)

All four points from the first review were addressed, and both blockers are genuinely fixed. CI is fully green (test, verify, verify-self); full suite passes locally, ruff clean across src/ and tests/.

Blocker 1 — resolved, and fixed in the right place

The fix went into the engine (ci/release_decision.py:725), making the incomplete_surface action framework-aware via the existing _inventory_manifest_key, with _semantic_gap_path routed to SUGGESTED_INVENTORY_FILENAME. Re-ran the original reproduction on the same ADK repo — gap ordering is now:

[0] incomplete_surface -> declare_tool_inventory  path=suggested-inventory.json

and verify --format text emits:

Improve evidence: Run: agents-shipgate verify … . Review the skeleton written next to report.json, save it in your repo, reference it from google_adk.tool_inventories in shipgate.yaml, then rerun verification. Target: suggested-inventory.json.

Sentence boundary is correct. Because it landed in the engine, console, GitHub summary, fix_task, packet and handoff all move together — no second projection to keep in sync. The generic branch is preserved and still covered (test_release_decision.py:209, test_evidence_packet.py:1050 both still assert provide_complete_inventory and pass).

I also swept every samples/* workspace for a gap pointing at a suggested-inventory.json that isn't written — the writer keys on low confidence, while incomplete_surface keys on extraction_confidence != "high" or source_type in _AST_ONLY_SOURCE_TYPES, so those two are not the same predicate. No divergence found: the AST-only loaders never emit high, and the inventory path rewrites source_type to *_inventory, so the phantom-artifact case is not reachable today. The new test pins it anyway with assert (tmp_path / "reports" / "suggested-inventory.json").is_file(), which is the right guard.

Blocker 2 — resolved

The fixture now runs a real run_scan over a real ADK workspace instead of hand-building a ToolSemanticAssessment, so it can no longer encode a state the resolver rejects. It asserts the gap ordering, action kind, path, manifest key, and the on-disk artifact. test_verify.py adds a real verify --format text end-to-end. Stronger than what the review asked for.

CHANGELOG, README, docs/quickstart.md, and STABILITY are all updated.

---

Three minor items remain

1. Run: is rendered first, but the rerun is the last step. primary_evidence_remediation_text unconditionally prefixes Run: {command}., so verify now leads with "Run: agents-shipgate verify …" immediately after the user ran verify. The real work — review the skeleton, save it, reference it — follows, and the rerun is the final step. It also flips the suggested invocation to --format json. This is the one place the new output reads backwards, and it is the same thing #327 invariant 1 rules out ("not a sentence containing a command"). Rendering the command last would fix it:

text = f"{action.expects} Then run: {action.command}"

2. One repair is now prescribed twice. incomplete_surface and low_confidence_tool now carry the same action.kind, the same path, the same manifest key and the same skeleton file — differing only in the trailing verb ("rerun verification" vs "rerun the scan") and the presence of a command. In verify's fix_task.instructions it surfaces as two bullets for one action. Not a contradiction (before this commit they disagreed, which was worse), just redundancy. fix_task._insufficient_evidence_remedies already skips low_confidence_tool/source_warning to avoid double-emitting against its own by_source loop (cli/verify/fix_task.py:445); extending that skip to incomplete_surface rows whose action is declare_tool_inventory would close it.

3. Agent contract doc is now inaccurate. docs/agent-contract-current.md:294 states "Semantic next actions use declare_action_effect, declare_action_authority, provide_complete_inventory, or resolve_semantic_conflict" — a semantic gap can now emit declare_tool_inventory. No schema bump is needed (declare_tool_inventory is already in the EvidenceGapAction.kind Literal, and all report/packet/verifier schemas are unchanged), but that sentence should list it. STABILITY.md:880 is phrased historically ("v0.29 adds…") so it is defensible as-is.

---

Verdict: approve once item 1 is addressed — it is the only one an adopter actually reads, and it is a two-line change. Items 2 and 3 are fine as follow-ups.