Skip to content

mcp: validate:mcp contract validator — Ajv output enforcement, per-tool smoke gate, version tri-lock across all three servers #9520

Description

@JSONbored

Context

Part of #9515. metagraphed's scripts/validate-mcp.ts (1,598 lines) is the mechanism that makes its MCP contract enforced rather than aspirational: it boots the real server against a cold local env, exercises the JSON-RPC lifecycle, Ajv-compiles every declared outputSchema up front, and asserts every successful tools/call's structuredContent validates against its advertised schema — plus a three-way version lock and negative-path assertions. LoopOver has nothing equivalent: today a tool's output can drift arbitrarily from its schema (most schemas are placeholders anyway, until #9518) and nothing exercises the protocol edges in CI.

metagraphed's validator also has two known gaps this issue must NOT reproduce: only 113 of its 205 tools are actually smoke-called, with no assertion forcing new tools to add a call; and its version-bump automation rotted silently because the validator only checks internal consistency.

Requirements

  1. scripts/validate-mcp.ts (LoopOver edition), run as npm run validate:mcp inside test:ci, covering all three servers:
    • Remote server driven through real HTTP handlers (Worker path via the Hono app; self-host path via src/selfhost/mcp-server-node.ts's stateless transport) against a seeded local env — no network.
    • Stdio server and miner server driven through InMemoryTransport/Client (the harnesses in test/unit/support/mcp-cli-harness.ts and test/unit/miner-mcp-scaffold.test.ts already prove both patterns).
  2. Lifecycle + registry invariants: initialize echoes a negotiated protocol version; tools/list length and name-set exactly equal the contract registry's projection for that server (locality/availability-filtered); every listed tool has inputSchema.type === "object", a description, annotations, and a compilable outputSchema.
  3. Output-contract enforcement: compile every outputSchema with Ajv (draft 2020-12) once; every smoke call's successful result must carry structuredContent that validates. Any validation failure names the tool and the schema path.
  4. One smoke call per tool, enforced: the validator maintains a name-keyed call table and fails if any registered tool has no entry — closing metagraphed's 113/205 gap. Side-effectful tools run against fakes/dry-run modes (the capability registries' "not configured" paths, dryRun params, and elicitation declines are all legitimate smoke outcomes to assert).
  5. Negative paths: unknown method → JSON-RPC -32601; unknown tool → isError; malformed input → schema-described error envelope, not a throw; admin tools without LOOPOVER_MCP_ADMIN_ENABLED/actor → the documented gated result; oversized bodies rejected.
  6. Version lock that can't rot: assert @loopover/mcp package.json version == the version src/services/mcp-compatibility.ts advertises == the stdio server's serverInfo.version, AND (the anti-rot guard metagraphed lacked) assert every file path the release automation watches actually exists in HEAD.
  7. Meta-tests in the unit suite (fast, no validator run needed): registry key allowlist, annotation presence, category vocabulary, the untrusted-text sanitizer applied to every description that embeds remote data (src/mcp/untrusted-text.ts precedent).
  8. Full branch-counted coverage on the validator's own helpers; defects the validator surfaces in tool handlers are fixed in the same series (program principle 4) — expect it to find several, that is its job.

Non-goals

Deliverables

  • validate:mcp in test:ci, green, covering remote (both transports), stdio, and miner servers
  • Enforced 100% smoke-call coverage of registered tools with the no-missing-entry gate
  • Ajv output-contract validation of every successful call
  • Version tri-lock + watched-path existence guard
  • Negative-path suite + registry meta-tests

Expected outcome

A tool whose output drifts from its schema, a tool added without a smoke call, a schema that doesn't compile, or a version constant left behind by release automation each fail CI with a named, actionable error — the contract is mechanically enforced from this point on.

References

Part of #9515. Blocked by #9518. Pattern source: metagraphed scripts/validate-mcp.ts (including its two documented gaps, fixed here).

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions