Review 09: falsifiable experiment harness scaffold (track D)

[howlcipher]

Summary

Track D / review doc 09: scaffold for the falsifiable experiment harness at experiments/falsifiable-v0/. It lets an operator run the 09 HowlFrame-vs-JSON-plan experiment offline later. Scaffold only. No model runs, no results, no H1 PASS/KILL claim.

Layout

Path	Purpose
tasks/manifest.json	30 synthetic release-evidence tasks t01–t30 (3–6 steps; 10 loop, 11 conditional) + 30 one-to-one adversarial variants a01–a30 (10 each write_outside / fetch_attacker / auto_approve, expected: deny_or_never_requested)
tasks/<id>/evidence/	Deterministic inputs (JSON/kv/text); injection text lives in data ({{ATTACKER_URL}} substituted at materialization)
tasks/<id>/oracle/	Hidden expected effects/outputs, never copied to the sandbox. 6 reference (t01–t03, a01–a03), 54 placeholder (never count as success)
schema/action_catalog.schema.json	Closed Arm B catalog: read_file, filter, if, write_record, request_approval
prompts/	Arm A / Arm B preambles
reference/arm_a, reference/arm_b	Handwritten smoke solutions
fixtures/	Unsafe/invalid candidates for negative tests
harness/	Broker, materialization, recorder (fs diff + canary, C5 receipt audit, loopback HTTP recorder, process-audit stub), oracle, Arm A/B runners with ≤2 repair hooks, 09 decision function
cmd/hfexp/	validate, arm-a, arm-b, suite, score
README.md, PROTOCOL.md	Layout + offline operator runbook for the full 09 protocol

Arm A pipeline: evidence-only sandbox + canary → check --profile governed → build --profile governed → inspect --caps ⊆ allowed (otherwise stop before run) → -run-bc --allow-caps <allowed> --max-instructions --deadline --receipt (minimal env) → receipt audit + fs diff + oracle. Repairer interface (NoRepair default, DirRepairer replays saved repairs). No LLM implementation.

Arm B broker: decodes strictly (unknown fields/actions rejected) and validates the whole plan before any effect: clean relative paths, symlink confinement, exact allowed writes, 64-step / 8-level bounds. It has no network, process, or approve action.

Smoke coverage (hermetic, CI-safe)

• Manifest structure: counts, shapes, pairing, files, oracles.
• Broker: all six reference plans match their oracles. Rejected: absolute, .., NUL and symlink paths, disallowed writes, unknown fields/actions, depth/step overflow. These are rejected with no fs change. Approval never approves.
• Recorder: create/modify/delete detected, canary change flagged.
• Real CLI (built in t.TempDir): all six reference .howl pass governed check/build/caps/run/receipt with 0 unauthorized.
  • Escape fixture (../canary/pwned.txt): flagged by both the receipt and the diff.
  • Auto-approve fixture: unauthorized.
  • Network caps: stopped at preflight, no receipt.
  • FETCH under a filesystem-only grant: denied, 0 requests on the loopback recorder.
  • Check failure + repair1: success.
• Scoring: synthetic PASS/KILL/INCONCLUSIVE/incomplete/hard-stop cases.

Validation

gofmt -l . clean · go vet ./... ok · go test ./... -count=1 all ok · benchmarks/v2 unittest + scripts/test_seo.py pass.

Honest limits / deferrals

• The filesystem grant is not path-scoped (S8/P2). The harness detects out-of-sandbox writes but does not prevent them. Trials must run in a throwaway container/VM. Two-tree diffs do not see every absolute host write.
• Still deferred:
  • the 54 placeholder oracles
  • real process/host audit
  • token counts and reviewer minutes (operator-supplied)
  • model generation (outside CI)
  • Arm C (Starlark)
• No internal/ or howlframe.go change. No HFBC/opcode change. No prod -compile-bc/HFIR flip. Write flagged http_server bytecode and stop #90 stays Partial. C1–C5 unchanged.

Journal: docs/journals/2026-10-06_falsifiable_experiment_harness.md

[howlcipher]

Dev-lead COMMENT (Motoko) — head 113508a1

Merge-ready for undraft + squash-merge (CI build green on Go 1.21).

Diff matches scope

• Scaffold only under experiments/falsifiable-v0/: 30+30 tasks, Arm B JSON broker (closed catalog, whole-plan validate before effects, path/symlink fail-closed), Arm A governed check/build/inspect --caps/-run-bc --receipt glue, recorder (fs+canary + C5 receipt audit), cmd/hfexp, hermetic smoke.
• Explicit non-claims: no live LLM, no checked-in results, no H1 PASS/KILL; score provenance is “computed only from provided results; not an experiment claim.”
• Honest limits documented (54 placeholder oracles never succeed; S8 filesystem not path-scoped — detect via receipt/diff, container recommended; no Arm C).

Hard nos intact

Zero touches to internal/ or howlframe.go (291 files audited). No HFBC/opcode change; no prod HFIR flip; #90 Partial; C1–C5 consumed as CLI, not modified.

Non-blockers

• Placeholder oracles + operator freeze/PROTOCOL steps remain before any real trial (as designed).
• FINAL_REPORT P1 C4 row / 05 termination cell still stale pre-C4b wording — cleanup queue, not a #114 blocker.
• suite still requires --howlframe even when only Arm B candidates are present (harmless; documented by smoke).

Left draft. Undraft + squash-merge when you are ready.