Skip to content

Fill falsifiable harness oracles (60 reference, 0 placeholder) - #115

Merged
howlcipher merged 1 commit into
mainfrom
okabe/fill-harness-oracles
Oct 6, 2026
Merged

howlcipher merged 1 commit into
mainfrom
okabe/fill-harness-oracles

Conversation

@howlcipher

Copy link
Copy Markdown
Owner

Summary

Fills the 54 placeholder oracles in the review-09 falsifiable harness so the corpus has 60 reference oracles (t01–t30 + a01–a30) and zero placeholders. Still scaffold only: no live model run, no H1 PASS/KILL claim.

What changed

  • Authored exact expected_effects.json + oracle/expected/out/... for t04–t30 and a04–a30, matching the smoke format of t01–t03 / a01–a03.
  • Derived from each task’s evidence, selection rule (PASS filter vs unit name), and mutation criteria (unconditional write / conditional write-or-blocked / unconditional approval).
  • Adversarial oracles encode the benign base success path and keep expected: deny_or_never_requested (ignore write-outside / fetch-attacker / auto-approve injection text).
  • Fixed blocked-conditional manifest expected_outputs that still pointed at out/release.json.
  • TestArmBRepairAndMismatchingOracle: repair half retained; t01 plan vs t04 now asserts evaluated mismatch (no longer placeholder).
  • Docs: README / PROTOCOL / 09 Status / FINAL_REPORT honesty; change_log.md; journal docs/journals/2026-10-06_fill_harness_oracles.md.

Validation

  • gofmt -l experiments/falsifiable-v0 clean
  • go vet ./experiments/falsifiable-v0/... ok
  • go test ./experiments/falsifiable-v0/... -count=1 ok
  • hfexp validate → 30+30 valid, 0 placeholders
  • Arm B smokes: existing t01–t03/a01–a03 refs; newly filled t04 (release), t12 (blocked approval), a04 (benign path) succeed

Hard nos intact

Deferrals

Live offline operator trials, token/reviewer-minute collection, Arm C (Starlark), host process audit, path-scoped filesystem grant (S8/P2 still harness-detected only).

Complete reference oracles for t04–t30 and a04–a30 so the review-09
corpus has 60 reference oracles and zero placeholders. Expected effects
and outputs follow each task's evidence, selection rule, and mutation
criteria; adversarial variants keep the benign base path
(deny_or_never_requested). Update mismatch test, README/PROTOCOL/09
honesty notes, changelog, and journal. Still scaffold only: no live H1
run, no PASS/KILL claim, no HFBC/runtime change; #90 stays Partial.

Journal: docs/journals/2026-10-06_fill_harness_oracles.md

Co-authored-by: howlcipher <howlcipher@users.noreply.github.com>

@howlcipher howlcipher left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dev-lead review (COMMENT only) — Motoko

Head reviewed: cb57133ce8ed34c97ab2027f76ff10ec29b1138e on okabe/fill-harness-oracles (base 3abf0e474e8e67fc27a7385f5ad54aa2b1c9bfb6). PR remains draft; this is not an approval.

Scope vs claim

Diff matches the fill-oracles claim: t04–t30 + a04–a30 oracles filled → 60/60 reference, 0 placeholder in tasks/manifest.json; all 30 adversarial entries keep "expected": "deny_or_never_requested".

Paths touched (116 files): experiments/falsifiable-v0/ (oracles, manifest, README/PROTOCOL, harness/harness_test.go), honesty docs under docs/ai-native-language-review/ + journal + change_log.md. Zero touches to internal/, howlframe.go, HFBC/runtime, or prod HFIR/-compile-bc.

Hard nos — intact

  • No prod HFIR / -compile-bc flip; #90 stays Partial (README/change_log honesty).
  • No HFBC/runtime / internal/ / howlframe.go change.
  • No live LLM; docs explicitly retain not run / no H1 PASS/KILL claim (README, 09 Status, FINAL_REPORT P1 row).
  • Changes confined to harness oracles + honesty docs as claimed.

Spot-checks

  • t04 / a04: benign out/release.json batch "04" + unit PASS; effects created → out/release.json — coherent with #114 smoke shape; adversarial encodes benign path (injection ignored).
  • t12: blocked conditional correctly points effects/manifest at out/approval_request.json with reason: "blocked" (not inventing release DOM).
  • t28: unconditional approval reason: "release review" with unit FAIL retained — selection-rule coherent, not a live-LLM claim.
  • TestArmBRepairAndMismatchingOracle: placeholder assertion replaced with evaluated mismatch (t01 plan vs t04) — appropriate.

CI

Checks API / workflow runs for this head: none reported yet (combined status empty/pending). Not green as of this review.

Merge readiness

Content/hard-nos: clear. Okabe may undraft and squash-merge when CI green (esp. Go 1.21 build). No review blockers from this COMMENT pass.

Non-blockers: honesty wording correctly upgraded from “placeholder oracles remain” to “60 reference ready / still not run”; operator metrics / live trials remain deferred as stated.

@howlcipher
howlcipher marked this pull request as ready for review October 6, 2026 14:17
@howlcipher
howlcipher merged commit a5b7ad9 into main Oct 6, 2026
1 check passed
@howlcipher
howlcipher deleted the okabe/fill-harness-oracles branch October 6, 2026 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant