feat(sieve): add mcp handler for external MCP servers as observation sources

[mlieberman85]

Summary

Adds a new built-in sieve handler mcp symmetric with exec/api_call. TOML controls declare handler = "mcp" passes referencing an allowlisted server ([mcp_servers.<name>] block on the framework TOML or .baseline.toml); the handler dispatches over stdio, exposes the response as result.* to CEL expr, and records the raw response plus trust label in evidence.

• Allowlist is the primary trust boundary (no [mcp_servers.<name>] entry -> ERROR without spawn). Optional trusted_publisher triggers Sigstore sidecar verification; failure produces ERROR, never PASS.
• Child processes inherit only a curated env safe-set (PATH, HOME, LANG, LC_, XDG_, SSL_CERT_FILE) plus the operator's TOML env block with \$VAR substitution from the parent shell.
• Sessions are spawned lazily, pooled across the audit run on a persistent asyncio loop in a daemon thread, and torn down in verify_batch's finally block (SC-002: exactly one spawn + one teardown across N controls).
• Failure semantics: absent binary defaults to INCONCLUSIVE (FAIL if optional = false); tool-side isError=True is ERROR without marking the session broken; a session crash triggers exactly one respawn; a double-broken session resolves INCONCLUSIVE.
• Progress log: orchestrator emits [N/M] <control_id> dispatching_mcp <server>.<tool> on darnit.harness at INFO before dispatch, matching feature 026's dispatching_llm pattern.

Zero new runtime deps: uses existing mcp>=1.23,<2 client-side APIs and the already-declared darnit-core[attestation] sigstore extra for the optional verification path. Scoped entirely to packages/darnit/ core; no plugin package touched.

The mcp_trust.verify module is deliberately isolated so the sandboxing follow-up tracked in #375 (bubblewrap, nono.sh, landlock, nsjail) can extend the pre-spawn hooks without touching the pool.

Spec: specs/031-mcp-server-handler/ (52 tasks, all closed).

Test plan

• uv run pytest tests/darnit/sieve/test_mcp_handler.py tests/darnit/sieve/test_mcp_pool.py tests/darnit/sieve/test_mcp_trust.py tests/darnit/config/test_mcp_server_config.py tests/darnit/config/test_merger_mcp_servers.py -- 27 pass, 1 skip (sigstore extra not in dev env)
• uv run pytest tests/ -q --deselect tests/darnit/context/test_dot_project_upstream.py::TestUpstreamSpecSync::test_upstream_spec_unchanged -- 2737 pass, 17 skip, 0 fail
• uv run ruff check on all feature-touched files -- clean
• uv run python scripts/validate_sync.py --verbose -- PASS (TOML schema, handler-name registry, SARIF source)
• Structure decision guard: no touches under other product packages (packages/(darnit-baseline|darnit-gittuf|darnit-reproducibility)/src/)
• FR-017 no-new-runtime-dep guard: no diffs to any pyproject.toml
• Reviewer: end-to-end test against a real MCP server (e.g. uwu-tools/scorecard-mcp when available) to confirm the operator experience matches the two worked examples in specs/031-mcp-server-handler/quickstart.md

Backward compatibility: strict addition. Every framework TOML and .baseline.toml that parsed successfully before continues to parse identically; the [mcp_servers] schema section is optional; absence is the pre-feature state.

[Marc-cn]

Thanks, reviewed the diff and ran the suite. Two things in mcp_trust.py, plus a test-isolation issue.

1. verify() never binds the bundle to the binary. It locates .sigstore, parses it, and calls verifier.verify_dsse(bundle, policy). binary_path is used only to find the sidecar ,the file is never hashed and the in-toto subject digest is never compared against it. As I read it, any valid bundle from that repo's workflow passes regardless of which binary it sits beside, and the result is a sigstore-verified trust label on an unverified binary. The PR description says failure never produces PASS, which is true, the gap is that a substituted binary produces success. I may be misreading the sigstore-python surface here,can you confirm?

2. verify/spawn resolve the path twice. _spawn resolves the program via shutil.which() and verifies that path, then spawns with the original unresolved command, so the OS re-resolves PATH at exec time. command[0] = str(binary_path) after resolution closes it

3. Test isolation. tests/darnit/sieve/test_mcp_handler.py passes standalone (1.4s) but 8 of 16 fail when run after tests/darnit/harness, all with MCP handshake failed for mock: fileno. Reproduced with pytest tests/darnit/harness tests/darnit/sieve/test_mcp_handler.py; I tested with Python 3.12.13, so it may be environment-specific. test_mcp_pool.py::test_teardown_on_exception_path also records 0 teardown events in the full run, and I hit a suite hang at 99% once that a later identical run didn't reproduce. Worth a look at whether _LoopBridge's loop is affected by state other tests leave behind, it's built on the calling thread rather than in _runner.

Everything else looks clean.

[mlieberman85]

Thanks for the sharp read. All three fixed in cc7431c.

1. Bundle-binding gap (verify_dsse alone). Correct read on my part -- the previous code never touched the binary bytes. mcp_trust.verify now:

• computes the binary's SHA-256, and
• tries verify_artifact(Hashed(SHA2_256, digest), bundle, policy) first (direct-artifact / cosign sign-blob shape -- the SDK does the binding), falling back to verify_dsse plus an explicit subject.digest.sha256 match against the same SHA-256 (in-toto / SLSA shape).

Both success paths now reject a bundle whose subject digest does not match the on-disk binary. New tests lock the mismatch-rejected, digest-match-accepted, and direct-artifact-accepted paths.

2. PATH TOCTOU. Same fix as you suggested -- _spawn now sets command[0] = str(binary_path) after resolution and hands that to StdioServerParameters. The OS no longer re-resolves at exec time.

3. Test isolation. Traced this to a subtler cause than my initial guess. mcp.client.stdio.stdio_client binds its errlog=sys.stderr default at module-import time. tests/darnit/harness/test_cli.py::TestMissingRepoPath (via pytest's capsys) replaces sys.stderr with a _pytest.capture.EncodedFile that has no fileno(). When mcp is first imported inside that capsys window, the default is captured broken and every subsequent spawn hits errwrite = stderr.fileno() -> io.UnsupportedOperation: fileno. Fix: _open_session_async now resolves child stderr at call time via _resolve_child_stderr, preferring live sys.stderr with a working fileno, falling back to sys.__stderr__, then os.devnull. New test_pool_survives_captured_sys_stderr monkeypatches sys.stderr to a no-fileno stream and asserts spawn still succeeds.

Also moved the _LoopBridge event-loop construction inside the runner thread while I was in there (belt-and-suspenders for asyncio's per-thread child-watcher assumptions on POSIX), but that alone did not fix the fileno issue -- the stderr-capture path was load-bearing.

Verified locally:

• pytest tests/darnit/harness tests/darnit/sieve/test_mcp_handler.py tests/darnit/sieve/test_mcp_pool.py -- 135 pass, 0 fail (was 10 fail before the fix)
• pytest tests/ full sweep (031-relevant dirs) -- 149 pass, 0 fail
• Full workspace pytest tests/ -- 2741 pass, 17 skip, 0 fail (up 4 from the new sigstore-binding and stderr-capture regression tests)
• ruff check -- clean on all touched files

The regression tests would have caught the stderr-capture issue in CI even without your ordering. Let me know if you want any of the fixes split into separate commits before merge.

[Marc-cn]

Thanks, re-ran on my side, lgtm.