Document OpenRouter planner model evaluations

.context/reports/2026-09-09-gpt-5-6-luna-vs-gpt-5-4-mini-eval.md

@@ -0,0 +1,109 @@
+# GPT-5.6 Luna vs GPT-5.4 Mini planner evaluation
+
+- Evaluation date: 2026-09-09
+- Evaluated runtime revision: `816db61555a09c4e0ccafdd7827b6ca43a797bf3`
+- Harness: Discord-agent canonical live-planner eval
+- Route: direct OpenAI (`https://api.openai.com/v1`) using `OPENAI_API_KEY`
+- Models: `gpt-5.6-luna` and `gpt-5.4-mini`
+- Published evidence: [sanitized audit snapshot](../../tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/README.md)
+
+## Run method and selection semantics
+
+The two 27-scenario suites ran serially. The initial request for every planner
+attempt used JSON-object `response_format`; Luna used `max_completion_tokens`,
+`reasoning_effort=low`, and `verbosity=low`, with `temperature` omitted.
+
+The 90-second timeout applies to each HTTP request, not to an entire scenario.
+Any HTTP error causes an immediate second request without `response_format`,
+also with a 90-second timeout. A scenario whose production result is
+`failed` or whose provider draft fails gets one full harness retry; that retry
+repeats the same request behavior and replaces attempt 1 only when both its
+production and provider-draft checks pass. An affected scenario can therefore
+make up to four provider requests. The `retries` metric counts only the full
+scenario retries, and the retained observations record only the selected
+attempt; unselected retry and inner-fallback usage is not available.
+
+## Result
+
+Both models preserved the production safety contract and parsed all selected
+provider drafts. Mini had three more retained provider-draft passes and was
+faster; Luna used fewer output tokens and has a substantially lower
+selected-usage rate-card proxy.
+
+| Metric | GPT-5.6 Luna | GPT-5.4 Mini |
+| --- | ---: | ---: |
+| Production scenarios passed | 27 / 27 | 27 / 27 |
+| Provider-draft parses | 27 / 27 | 27 / 27 |
+| Retained provider-draft passes | 18 / 27 | 21 / 27 |
+| Provider-draft failures | 9 | 6 |
+| Average selected latency | 1,973.7 ms | 1,243.2 ms |
+| Maximum selected latency | 3,993 ms | 2,188 ms |
+| Total wall time | 74,474 ms | 40,418 ms |
+| Selected input tokens | 32,630 | 32,630 |
+| Selected cached-input tokens | 29,474 | 0 |
+| Selected output tokens | 2,464 | 2,982 |
+| Harness selected-use cost | not available | $0.0378915 |
+| Harness retries | 10 | 6 |
+
+Luna's selected average latency was 58.8% higher and its end-to-end suite time
+was 84.3% longer. It generated 17.4% fewer selected output tokens. The audited
+observation files and compact score summaries behind every value in this table
+are tracked in the linked snapshot.
+
+## Cost interpretation
+
+The direct OpenAI billing API was unavailable to this run, and no official
+direct Luna price was added to the local catalog. For a consistent comparison,
+the public [OpenRouter Luna](https://openrouter.ai/openai/gpt-5.6-luna) and
+[OpenRouter Mini](https://openrouter.ai/openai/gpt-5.4-mini) rates captured on
+the run date are used only as a rate proxy:
+
+| Model | Input / M | Cache read / M | Output / M | Selected-use proxy |
+| --- | ---: | ---: | ---: | ---: |
+| GPT-5.6 Luna | $0.20 | $0.02 | $1.20 | $0.00417748 |
+| GPT-5.4 Mini | $0.75 | $0.075 | $4.50 | $0.03789150 |
+
+The selected-usage proxy puts Luna at 9.07x lower cost (an 89.0% reduction).
+This is neither an invoice nor a lower bound: direct-provider billing can
+differ, and the published selected attempts omit any unselected retry or inner
+HTTP-fallback requests.
+
+## What the behavioral result does and does not show
+
+The fresh pair does not support the earlier provider-draft tie: Mini retained
+21/27 passes and Luna 18/27. Luna's extra failures include task-assignment and
+member-agreement status/action assertions in addition to GitHub query fields;
+Mini's only non-intent failure was the default GitHub state assertion. Both
+models also have free-form intent-label mismatches, which are useful diagnostics
+but are not executable-plan failures.
+
+All 27 production outcomes passed because the canonical suite is primarily a
+policy and deterministic-routing regression suite: 26 of its 27 fixtures have
+a deterministic ownership path. The result demonstrates that Luna integrates
+with the production safety boundary. It is not a broad quality benchmark for
+arbitrary planning, job matching, or resume extraction.
+
+## Historical OpenRouter attempt
+
+The initial OpenRouter preflight for both models returned HTTP 403 before any
+provider draft or usage. The account-level message cited a provider Terms of
+Service restriction and named no provider. That route remains blocked, but it
+does not affect this direct-OpenAI evaluation.
+
+## Decision and activation
+
+The workspace is configured for Luna as a cost-first, direct-OpenAI choice at
+the user's request. The fresh audit shows this saves substantial proxy cost but
+does not establish quality equivalence with Mini; Mini performed better on this
+single provider-draft pair. The ignored local `.env` pins these selectors to
+`gpt-5.6-luna`:
+
+- `OPENAI_MODEL` for job-requirement extraction and candidate reranking.
+- `AGENT_FALLBACK_MODEL` for the Discord agent and lead-classification fallback.
+- `RESUME_AI_MODEL` for resume extraction.
+- `AGENT_EVAL_OPENAI_MODEL` for future primary eval runs.
+
+This local activation does not modify any deployment dashboard or other
+environment. Do not treat the planner-only result as resume-workflow evidence;
+mirror the variables in a deployment only after accepting that scope and
+monitoring output quality.

packages/shared/src/five08/data/model-profiles.json

@@ -2,6 +2,24 @@
   "version": "model-profiles.v1",
   "pricing_source": "official provider pricing docs; OpenAI org usage/cost API requires api.usage.read",
   "models": {
+    "gpt-5.6-luna": {
+      "name": "gpt-5.6-luna",
+      "provider": "openai-compatible",
+      "model": "gpt-5.6-luna",
+      "api_key_env": "OPENAI_API_KEY",
+      "base_url": "https://api.openai.com/v1",
+      "request_options": {
+        "response_format": {
+          "type": "json_object"
+        }
+      },
+      "chat_completion_options": {
+        "max_tokens_parameter": "max_completion_tokens",
+        "reasoning_effort": "low",
+        "verbosity": "low",
+        "supports_temperature": false
+      }
+    },
     "gpt-5.5": {
       "name": "gpt-5.5",
       "provider": "openai-compatible",

packages/shared/src/five08/llm_model_profiles.json

@@ -1,5 +1,16 @@
 {
   "models": {
+    "gpt-5.6-luna": {
+      "name": "gpt-5.6-luna",
+      "provider": "openai-compatible",
+      "model": "gpt-5.6-luna",
+      "request_options": {
+        "temperature": false,
+        "response_format": true,
+        "reasoning_effort": true,
+        "verbosity": true
+      }
+    },
     "gpt-5.5": {
       "name": "gpt-5.5",
       "provider": "openai-compatible",

tests/evals/discord-agent/README.md

@@ -45,6 +45,13 @@ The CLI loads `.env` by default without overriding exported environment values.
 Use `--no-env-file` to disable that behavior or `--env-file <path>` to point at
 another file.
 
+Generated reports remain ignored because they are ephemeral and can contain
+debugging detail. A reviewed historical run can instead publish a sanitized,
+immutable audit snapshot under `artifacts/<run-id>/`; see the
+[`2026-08-28` OpenRouter flash comparison](artifacts/llm-workflow-flash-comparison-2026-08-28/README.md).
+Those snapshots are evidence for a specific run, not a replacement for a fresh
+evaluation.
+
 ## Viewing Results
 
 Open `tests/evals/discord-agent/reports/index.html` after a run. GitHub Actions

tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/README.md

@@ -0,0 +1,37 @@
+# Direct OpenAI Luna/Mini audit snapshot
+
+This tracked, immutable snapshot supports the findings in [the direct OpenAI
+planner evaluation report](../../../../../.context/reports/2026-09-09-gpt-5-6-luna-vs-gpt-5-4-mini-eval.md). It is deliberately separate from
+`tests/evals/discord-agent/reports/`, which remains ignored scratch output.
+
+## Scope
+
+- Evaluated runtime revision: `816db61555a09c4e0ccafdd7827b6ca43a797bf3`.
+- Route: direct OpenAI (`https://api.openai.com/v1`) using the primary eval
+  profile and `OPENAI_API_KEY`.
+- Full sweeps: one serial 27-scenario canonical run for `gpt-5.6-luna` and one
+  for `gpt-5.4-mini`, each with a 90-second timeout for each HTTP request.
+- The full-sweep files preserve selected scenario checks, provider-draft
+  results, selected latency, token usage, and raw synthetic model drafts.
+
+The first request in an attempt uses JSON-object `response_format`. Any HTTP
+error triggers an immediate second request without that format; a scenario
+whose production result is `failed` or whose provider draft fails gets one
+full retry. The retry replaces the first result only if both production and
+provider-draft checks pass. Thus an affected scenario can make up to four
+provider requests, while the published observation represents only the
+selected scenario result. The `retries` metric counts full scenario retries,
+not inner HTTP fallbacks.
+
+The `api_key_configured` boolean was removed from every published observation.
+No credentials, request headers, HTML viewer, CTRF output, or debug trace is
+retained here. These are historical observations, not a claim that future
+provider responses will be identical.
+
+## Integrity check
+
+From the repository root, verify the snapshot with:
+
+```bash
+sha256sum -c tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/SHA256SUMS
+```

tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/SHA256SUMS

@@ -0,0 +1,5 @@
+cc6ac1934afc35414cc67195e5e5d4e8078324480055538eace418de9acb6b58  tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/README.md
+4a52d56863ecc352abb0fee5ffccda2b8729f4cb3729f82543c15a85f8178da9  tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/full-sweeps/gpt-5.4-mini.observed.json
+92ea3a093c43ae4709425324e6a0e601815d86e49e1827866e152b884e7c4ae7  tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/full-sweeps/gpt-5.4-mini.score.md
+bb01b6b1f1345592e92b59fe3a7ea0ab2383f0ae0ae90d5c3f2ae0f7831de45f  tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/full-sweeps/gpt-5.6-luna.observed.json
+a137b5b4922ef0f738ac7c2da1f65eef5c85ef7eb166a3d64750a78cf271cf83  tests/evals/discord-agent/artifacts/gpt-5-6-luna-vs-gpt-5-4-mini-2026-09-09/full-sweeps/gpt-5.6-luna.score.md