Repository navigation
Fill falsifiable harness oracles (60 reference, 0 placeholder) - #115
Conversation
Complete reference oracles for t04–t30 and a04–a30 so the review-09 corpus has 60 reference oracles and zero placeholders. Expected effects and outputs follow each task's evidence, selection rule, and mutation criteria; adversarial variants keep the benign base path (deny_or_never_requested). Update mismatch test, README/PROTOCOL/09 honesty notes, changelog, and journal. Still scaffold only: no live H1 run, no PASS/KILL claim, no HFBC/runtime change; #90 stays Partial. Journal: docs/journals/2026-10-06_fill_harness_oracles.md Co-authored-by: howlcipher <howlcipher@users.noreply.github.com>
howlcipher
left a comment
There was a problem hiding this comment.
Dev-lead review (COMMENT only) — Motoko
Head reviewed: cb57133ce8ed34c97ab2027f76ff10ec29b1138e on okabe/fill-harness-oracles (base 3abf0e474e8e67fc27a7385f5ad54aa2b1c9bfb6). PR remains draft; this is not an approval.
Scope vs claim
Diff matches the fill-oracles claim: t04–t30 + a04–a30 oracles filled → 60/60 reference, 0 placeholder in tasks/manifest.json; all 30 adversarial entries keep "expected": "deny_or_never_requested".
Paths touched (116 files): experiments/falsifiable-v0/ (oracles, manifest, README/PROTOCOL, harness/harness_test.go), honesty docs under docs/ai-native-language-review/ + journal + change_log.md. Zero touches to internal/, howlframe.go, HFBC/runtime, or prod HFIR/-compile-bc.
Hard nos — intact
- No prod HFIR /
-compile-bcflip; #90 stays Partial (README/change_log honesty). - No HFBC/runtime /
internal//howlframe.gochange. - No live LLM; docs explicitly retain not run / no H1 PASS/KILL claim (README, 09 Status, FINAL_REPORT P1 row).
- Changes confined to harness oracles + honesty docs as claimed.
Spot-checks
- t04 / a04: benign
out/release.jsonbatch"04"+ unit PASS; effectscreated→out/release.json— coherent with #114 smoke shape; adversarial encodes benign path (injection ignored). - t12: blocked conditional correctly points effects/manifest at
out/approval_request.jsonwithreason: "blocked"(not inventing release DOM). - t28: unconditional approval
reason: "release review"with unit FAIL retained — selection-rule coherent, not a live-LLM claim. - TestArmBRepairAndMismatchingOracle: placeholder assertion replaced with evaluated mismatch (t01 plan vs t04) — appropriate.
CI
Checks API / workflow runs for this head: none reported yet (combined status empty/pending). Not green as of this review.
Merge readiness
Content/hard-nos: clear. Okabe may undraft and squash-merge when CI green (esp. Go 1.21 build). No review blockers from this COMMENT pass.
Non-blockers: honesty wording correctly upgraded from “placeholder oracles remain” to “60 reference ready / still not run”; operator metrics / live trials remain deferred as stated.
Summary
Fills the 54 placeholder oracles in the review-09 falsifiable harness so the corpus has 60 reference oracles (t01–t30 + a01–a30) and zero placeholders. Still scaffold only: no live model run, no H1 PASS/KILL claim.
What changed
expected_effects.json+oracle/expected/out/...for t04–t30 and a04–a30, matching the smoke format of t01–t03 / a01–a03.expected: deny_or_never_requested(ignore write-outside / fetch-attacker / auto-approve injection text).expected_outputsthat still pointed atout/release.json.TestArmBRepairAndMismatchingOracle: repair half retained; t01 plan vs t04 now asserts evaluated mismatch (no longer placeholder).change_log.md; journaldocs/journals/2026-10-06_fill_harness_oracles.md.Validation
gofmt -l experiments/falsifiable-v0cleango vet ./experiments/falsifiable-v0/...okgo test ./experiments/falsifiable-v0/... -count=1okhfexp validate→ 30+30 valid, 0 placeholdersHard nos intact
-compile-bcflip; Write flagged http_server bytecode and stop #90 PartialDeferrals
Live offline operator trials, token/reviewer-minute collection, Arm C (Starlark), host process audit, path-scoped filesystem grant (S8/P2 still harness-detected only).