Add the #485 design simulations behind an invariant test

.github/workflows/live-tests.yml

@@ -37,8 +37,7 @@ jobs:
 
   # There is no live-anthropic job: Anthropic is not a provider of this
   # product (ADR 0001, decided 2026-08-21). The frozen adapter retires under
-  # #430. The next live job to add is live-huggingface, with the provider
-  # itself (#484).
+  # #430.
 
   live-dartmouth:
     name: Live Dartmouth provider (free models)
@@ -74,3 +73,37 @@ jobs:
           # otherwise pass having proved nothing. Require real passes.
           grep -qE "[1-9][0-9]* passed" dartmouth-live.txt \
             || { echo "No Dartmouth live test actually passed; coverage is not real."; exit 1; }
+
+  live-huggingface:
+    name: Live HuggingFace Inference API (free routes)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install
+        run: |
+          python -m pip install --upgrade pip
+          # No provider extra: the router is OpenAI-compatible and the
+          # adapter speaks it with aiohttp, which is already a core dependency.
+          python -m pip install -e .
+          python -m pip install pytest pytest-asyncio pytest-timeout
+
+      - name: Run live HuggingFace tests
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+          ORCHESTRATOR_REQUIRE_LIVE: "1"
+        run: |
+          if [ -z "${HF_TOKEN}" ]; then
+            echo "HF_TOKEN is not configured; cannot verify HuggingFace support."
+            exit 1
+          fi
+          python -m pytest -m live -k huggingface -v | tee huggingface-live.txt
+          # Free routes are promos and every one can legitimately be flapping
+          # at once, in which case every generation test skips and the job
+          # would otherwise pass having proved nothing. Require real passes.
+          grep -qE "[1-9][0-9]* passed" huggingface-live.txt \
+            || { echo "No HuggingFace live test actually passed; coverage is not real."; exit 1; }

.gitignore

@@ -286,3 +286,9 @@ tests/performance/results/
 tests/performance/alerts/
 tests/quality/results/
 tests/scenarios/results/
+
+# opencode/OMC session tooling state
+.omo/
+
+# minikernel prototype scratch
+scripts/prototypes/**/__pycache__/

README.md

@@ -30,12 +30,11 @@ provider to pass live acceptance tests** (`live-dartmouth`, 9 passed remotely
 on 2026-08-01) and is the one provider described here as supported.
 
 **Provider policy: Dartmouth Chat and HuggingFace (Inference API) only.**
-Anthropic, OpenAI, Google and Ollama adapters remain in the tree but are
-unsupported, are not advertised, and are being retired
-([#430](https://github.com/ContextLab/orchestrator/issues/430)). HuggingFace
-support means the hosted Inference API and is in progress
-([#484](https://github.com/ContextLab/orchestrator/issues/484)); it is not
-claimed to work until its live job passes.
+The Anthropic, OpenAI, Google and Ollama adapters were retired
+([#430](https://github.com/ContextLab/orchestrator/issues/430)) and are no
+longer shipped. HuggingFace support means the hosted Inference API and is in
+progress ([#484](https://github.com/ContextLab/orchestrator/issues/484)); it
+is not claimed to work until its live job passes.
 
 **The wider legacy test suite is not green.** Only the marked
 `unit`/`contract`/`e2e` layer gates the build. The remainder were written
@@ -47,8 +46,8 @@ That job is marked `continue-on-error`, which means GitHub reports it green
 **regardless of the result** — so its check mark says nothing about the
 suite. The real numbers are in the job's run summary, in a warning
 annotation on the run page, and in its `legacy-suite-results` artifact.
-As of the most recent run: **434 failed, 248 errors, 1827 passed, 227
-skipped**. Track it in
+As of the post-retirement re-baseline (full local run, 2026-08-21):
+**500 failed, 93 errors, 1707 passed, 109 skipped**. Track it in
 [#354](https://github.com/ContextLab/orchestrator/issues/354) rather than
 trusting a number maintained by hand here, which has been wrong before.
 
@@ -152,10 +151,9 @@ pip install "py-orc[all]"          # every runtime extra
 
 Neither supported provider needs an extra: Dartmouth Chat and the HuggingFace
 Inference API are both spoken over HTTP with `aiohttp`, already a core
-dependency. The `anthropic`, `openai` and `google` extras still exist for the
-frozen adapters, which are unsupported and being retired
-([#430](https://github.com/ContextLab/orchestrator/issues/430)) — do not build
-on them.
+dependency. The `anthropic`, `openai` and `google` extras were removed with
+the retired adapters
+([#430](https://github.com/ContextLab/orchestrator/issues/430)).
 
 A missing extra disables only the feature that needs it; it never breaks
 `import orchestrator`.
@@ -324,12 +322,10 @@ The supported providers need no configuration file:
   [#484](https://github.com/ContextLab/orchestrator/issues/484); `HF_TOKEN`
   will be the credential.
 
-A `~/.orchestrator/models.yaml` with `source:` entries (`ollama`,
-`huggingface`, `openai`, `anthropic`, `google`) is still read, but every one
-of those sources routes through the frozen adapter layer — unsupported, and
-being retired under
-[#430](https://github.com/ContextLab/orchestrator/issues/430). New work should
-not depend on them.
+A `~/.orchestrator/models.yaml` written before the provider retirement may
+still name `ollama`, `huggingface`, `openai`, `anthropic` or `google`
+sources; each such entry is skipped with a warning at population time — an
+old config file is not an error, but it no longer registers anything.
 
 ## Advanced Example
 
@@ -545,10 +541,10 @@ intent:
 |-|-|-|
 | Dartmouth Chat | — | **Supported** (free models). `live-dartmouth` green: 9 passed, 2026-08-01 |
 | HuggingFace (Inference API) | — | In progress ([#484](https://github.com/ContextLab/orchestrator/issues/484)) — not claimed to work until its live job passes |
-| Anthropic | `anthropic` | Not a provider of this product — frozen adapter, retiring under [#430](https://github.com/ContextLab/orchestrator/issues/430) |
-| OpenAI | `openai` | Not a provider of this product — frozen adapter, retiring under [#430](https://github.com/ContextLab/orchestrator/issues/430) |
-| Google | `google` | Not a provider of this product — frozen adapter, retiring under [#430](https://github.com/ContextLab/orchestrator/issues/430) |
-| Ollama (local) | — | Not a provider of this product — frozen adapter, retiring under [#430](https://github.com/ContextLab/orchestrator/issues/430) |
+| Anthropic | `anthropic` | Retired under [#430](https://github.com/ContextLab/orchestrator/issues/430) — no longer shipped |
+| OpenAI | `openai` | Retired under [#430](https://github.com/ContextLab/orchestrator/issues/430) — no longer shipped |
+| Google | `google` | Retired under [#430](https://github.com/ContextLab/orchestrator/issues/430) — no longer shipped |
+| Ollama (local) | — | Retired under [#430](https://github.com/ContextLab/orchestrator/issues/430) — no longer shipped |
 
 A provider is only called **supported** once the `live-tests` workflow passes
 for it remotely. "Verified locally" means its live tests were run by hand

docs/adr/0001-product-contract.md

@@ -80,10 +80,16 @@ passes.
     are not the supported path; the Inference API adapter is new work,
     tracked in #484.
 - **Anthropic, OpenAI, Google and Ollama are not providers of this product.**
-  Their adapters remain in the tree but are unsupported, must not be
-  advertised, and are retired under #430. The earlier plan to bring Anthropic
-  under live acceptance tests is withdrawn, and with it #432 (the
-  credit-blocked verification) and the `live-anthropic` CI job.
+  Their adapters were removed from the tree on 2026-08-21 (the first #430
+  cut: `integrations/` adapters, `models/anthropic_model.py`,
+  `models/openai_model.py`, `models/providers/anthropic_provider.py`, the
+  skills-era `models/registry.py` + `models/config.py` that only ever
+  supported Anthropic, `tools/update_models.py`, and the packaged default
+  model pool in `config/models.yaml`, which listed only retired providers).
+  A `models.yaml` written before the retirement is skipped entry-by-entry
+  with a warning, never raised on. The earlier plan to bring Anthropic under
+  live acceptance tests is withdrawn, and with it #432 (the credit-blocked
+  verification) and the `live-anthropic` CI job.
 - A provider earns the word **supported** only when its `live-tests` job
   passes remotely. As of 2026-08-21:
   - **Dartmouth Chat: supported.** `live-dartmouth` passed with 9 tests

docs/index.rst

@@ -57,4 +57,4 @@ Contribute
 Unsupported areas
 -----------------
 
-The supported providers are Dartmouth Chat (live-tested) and the HuggingFace Inference API (in progress). Anthropic, OpenAI, Google and Ollama adapters are present but frozen, and are being retired. Multimodal tools, MCP integration, monitoring, analytics, and deployment code are present but not part of the verified product surface. See the product contract for the precise boundary.
+The supported providers are Dartmouth Chat (live-tested) and the HuggingFace Inference API (in progress). The Anthropic, OpenAI, Google and Ollama adapters were retired and are no longer shipped. Multimodal tools, MCP integration, monitoring, analytics, and deployment code are present but not part of the verified product surface. See the product contract for the precise boundary.

pyproject.toml

@@ -48,9 +48,9 @@ dependencies = [
 
 [project.optional-dependencies]
 # --- Model providers -------------------------------------------------------
-anthropic = ["anthropic>=0.7.0"]
-openai = ["openai>=1.0.0"]
-google = ["google-generativeai>=0.3.0"]
+# None. The supported providers (Dartmouth Chat, HuggingFace Inference API)
+# are spoken over HTTP with aiohttp, a core dependency. The anthropic /
+# openai / google extras retired with their adapters (#430).
 
 # --- Graph runtime ---------------------------------------------------------
 langgraph = [
@@ -127,7 +127,7 @@ notebooks = [
 # import name (orchestrator). The previous value referenced a package that
 # does not exist on any index.
 all = [
-    "py-orc[anthropic,openai,google,langgraph,web,multimedia,viz,infra,crypto]",
+    "py-orc[langgraph,web,multimedia,viz,infra,crypto]",
 ]
 all-dev = [
     "py-orc[all,dev,docs,notebooks]",
@@ -226,6 +226,13 @@ markers = [
     "integration: needs local services (docker/redis/postgres); opt-in",
     "live: calls a real model provider API and costs money; opt-in",
     "e2e: full pipeline through the installed CLI/API",
+    # pytest 9 validates builtin marks applied through a module-level
+    # `pytestmark` list against this table, so `filterwarnings` has to be
+    # declared here even though pytest registers it itself. Without this the
+    # blocking suite fails at COLLECTION on test_failure_policy.py and
+    # test_supported_examples.py, which both promote teardown warnings to
+    # errors that way.
+    "filterwarnings: builtin; declared so --strict-markers accepts it in a module-level pytestmark list under pytest >= 9",
     # Resource requirements
     "local: marks tests as local-only (not run in CI)",
     "slow: marks tests as slow (deselect with '-m \"not slow\"')",
@@ -260,8 +267,6 @@ module = [
     "docker.*",
     "redis.*",
     "psycopg2.*",
-    "openai.*",
-    "anthropic.*",
 ]
 ignore_missing_imports = true
 

scripts/prototypes/README.md

@@ -0,0 +1,106 @@
+# `minikernel` — an executable schematic for #485
+
+A deliberately small, runnable kernel for the redesign proposed in
+[#485](https://github.com/ContextLab/orchestrator/issues/485). Not a product,
+and not on the ADR-0001 path: nothing under `src/` imports it, and it imports
+nothing from `src/`. It exists so the issue's claims can be **executed** rather
+than argued about.
+
+The three design reviews on #485 all reached their conclusions by modelling
+mechanisms in isolation (see `scripts/simulations/`). This does the other half:
+it puts every mechanism the issue proposes into one running system, in its
+cheapest honest form, and reports what breaks.
+
+```bash
+.venv/bin/python scripts/prototypes/run_scenarios.py       # 55 end-to-end checks
+.venv/bin/python -m pytest tests/test_minikernel.py -q     # 57 unit tests
+.venv/bin/python scripts/prototypes/measure_ambiguity.py   # measured f (cached)
+.venv/bin/python scripts/prototypes/probe_optimism.py      # is low f real? (cached)
+```
+
+## Modules
+
+| module | #485 component | what it is |
+|-|-|-|
+| `store.py` | 3 | one substrate: append-only event log, content-addressed blobs, sealed-segment journal, summary DAG, FTS. The scratchpad, insight pool, context tables and tool history are **queries over it**, not four stores. |
+| `ir.py` | 1 | typed plan IR (sequence / branch / bounded loop / call / decompose), its validator, `Authority`, and the `Budget` ledger. No `goto`. |
+| `capabilities.py` | 3 | one lifecycle for tools, skills and reusable plans: `draft → candidate → trusted`, `quarantined`, `revoked`, plus the bug-report/triage workflow. |
+| `library.py` | 1 + 3 | the solved-problem library — two-key retrieval (statement similarity **and** typed I/O signature) and, added after the harness demanded it, **negative results**. |
+| `review.py` | 1a | separation of duty, frozen criteria, concern ledger, evidential gate, insight pool with contradiction detection. |
+| `planner.py` | 1 | `StubPlanner` (deterministic, for sweeps) and `LLMPlanner` (a real model over stdlib `urllib`, for measurement). |
+| `runtime.py` | 2 | the durable executor: nested runs, crash-resume, budget escalation, addressed message bus, admission control. |
+
+## What the harness is for
+
+Each scenario in `run_scenarios.py` asserts a property the design needs. When a
+scenario failed, the **kernel** was changed, not the assertion — and six of
+those changes are design findings, not typos:
+
+1. **The library must store negative results.** A mission that ends at the depth
+   cap taught the system nothing, so re-running it cost exactly as much,
+   forever. The one regime where learning matters was the one regime where
+   learning could not start. With dead-end memory a repeated unreachable
+   mission costs 5 nodes instead of 20 — and still never reports success.
+2. **An escalating sibling must not cancel the others.** Returning on the first
+   unreachable subtree threw away every sibling that was still solvable, and
+   with them everything the run would have learned.
+3. **A wildcard in a stored signature defeats the two-key match.** A solution
+   published as `any->any` matches every later query, so text similarity
+   silently becomes the only key. Untyped solutions are no longer published.
+4. **Budget exhaustion must not count against a cached plan.** It says nothing
+   about whether the plan was right, but counting it as a failure dropped
+   reliability below the retrieval floor after one unlucky mission.
+5. **A planner must be a function of its inputs.** The first `StubPlanner`
+   carried one RNG stream across calls, so asking the same question twice gave
+   different plans and the library's benefit was unmeasurable.
+6. **`atomic` has to be a checked claim, not a label** — see below.
+
+## The measurement that matters
+
+`f` — the fraction of steps a planner marks ambiguous — is the load-bearing
+parameter: recursion is finite in expectation iff `m = b·f < 1`. Every prior
+review had to guess it. Measured on 16 real problems across four library tiers
+(`measurements/ambiguity.json`):
+
+```
+model            library      #caps  mean b  mean f  m (correct)   regime
+gpt-5.4-mini     L0_bare          0    2.50   1.000         2.50   SUPERCRITICAL
+gpt-5.4-mini     L1_minimal       4    2.73   0.483         1.33   SUPERCRITICAL
+gpt-5.4-mini     L2_working      12    3.38   0.302         0.88   subcritical
+gpt-5.4-mini     L3_mature       30    4.31   0.489         1.88   SUPERCRITICAL
+gpt-5.6-sol      L2_working      12    4.80   0.013         0.07   subcritical
+```
+
+Two things to note. **The library helps and then hurts** — a bigger menu drives
+`f` down but drives `b` up, and `m` is their product. And **the estimator
+matters**: `m` is `E[ambiguous children]`, not `mean(b)·mean(f)`; the product of
+the means called L2 critical when it is subcritical.
+
+Then `probe_optimism.py` asks an independent judge model whether each
+*declared-atomic* step can really be done by the capability it names:
+
+```
+planner        plans  mean b  declared m  overclaim rate  corrected m
+gpt-5.4-mini       8    3.00        1.88  4/9  = 44%             2.38
+gpt-5.6-sol        8    4.38        0.12  12/34 = 35%            1.62
+```
+
+The stronger planner's apparent `m` of 0.12 is really **1.62**. It was not less
+ambiguous; it was more optimistic. Declared `f` is not a safety metric, and the
+cheapest way for any planner to look like it terminates is to lie about what is
+easy. That is why `Runtime` takes an `admission` hook, and why scenario S10
+shows the same run reporting `completed` with a placeholder answer without it
+and `escalated` with it.
+
+## Assumptions
+
+- Token counts are a `len/3.5` estimate, deliberately over-counting. A real
+  implementation must use the selected model's tokeniser.
+- `ReviewBoard.detection_rate_prior` (0.6) is a **guess**, carried through to
+  every reported `residual_risk`. It should be replaced by a measured
+  per-reviewer detection rate as soon as there is one.
+- The measured `f` numbers are properties of *(model, problem distribution,
+  library contents)* and of these 16 problems in particular. Quote all three.
+- The judge in `probe_optimism.py` is one model (`gpt-5.5`). Its own error rate
+  is unmeasured; the overclaim rates are therefore lower-confidence than the
+  direction of the effect.