[Recipe] Run CLM behind the frontend, and compare engine responses

[xiaoyu-xyz]

CLM is the second model in #9: a frozen Qwen3-8B encoder behind an OpenAI-compatible /v1/embeddings endpoint, two projection heads, and a cosine score.

clm-serve is an HTTP client of that endpoint, so the whole path runs on CPU against a stub encoder — no GPU, no vLLM, no 8B weights. recipe/clm/ has the stub, the launch commands and what the run shows. Point it at the real encoder with serve_qwen3_8b.sh and nothing downstream changes; that is the property worth having a recipe for.

The frontend needs no adapter. Status, content type and all three answer types pass through unchanged, including CLM's X-CLM-Latency-Ms:

PASS health  PASS department  PASS urgency  PASS refund  PASS combined

The response comparison did need a change. It compared the whole body byte-for-byte, which holds for LAYA because its body is a pure function of the request. A backend that reuses encoder state across requests reports usage only for the calls it paid for, so two identical requests differ:

same state, three times:   noul=0.977197  usage.input_tokens=0
a state not seen before:   noul=0.280477  usage.input_tokens=26
that same state again:     noul=0.280477  usage.input_tokens=0

The decision is deterministic; input_tokens counts cache misses. Comparing the full body fails a correct response, and whether it fails depends on which call happened to warm the cache — so the same run passes or fails on ordering. The comparison now checks status, content type and the answers subtree, and prints the usage difference rather than asserting on it. Strict equality is still tried first, so a backend whose body is a pure function is unaffected. The relaxation applies only to a successful status: it used to accept any two equal statuses, so a matching pair of 500s carrying an equal answers subtree passed.

compare_with_backend.py and its test move to recipe/ because both recipes use them now.

Tests

recipe/test_compare_with_backend.py runs two throwaway HTTP servers, no model and no Rust binary:

ok   identical responses pass
ok   usage-only difference passes and is reported
ok   a differing answer fails
ok   a status difference fails
ok   a non-200 backend fails
ok   equal answers on matching errors fail

Also verified against CLM with a cold cache, where the usage difference is real rather than zero on both sides.

Open question

Which response fields may differ between two otherwise identical requests? billing_units looks stable, input_tokens does not. If the project wants the strong form — the whole body identical — then input_tokens has to mean "tokens the request required" rather than "tokens this call paid for", which is a decision for the engine, not the frontend.

[hsliuustc0106]

Reviewed commit d61c0d139621fd0f8941360009859c38c45b5e17.

[P2] Preserve success-status requirement in relaxed comparisons. recipe/compare_with_backend.py:90 sets ok=True for equal answer subtrees whenever statuses match, even when both statuses are 500. Reproduced all PASS and exit 0 with (500, application/json, {"answers":{},"detail":"failed"}). Require HTTP 200 in this fallback. Existing five tests pass but do not cover errors containing answers.

[xiaoyu-xyz]

Fixed in fe319e1. The relaxed fallback now requires the worker's status to be 200 — since the same block already forces the two statuses equal, a matching pair of 500s carrying equal answers now fails instead of passing.

The error body on the test server is configurable, so the case the existing five did not cover is pinned by a sixth test. It fails against the previous revision.

[Levius-Fubuki]

Reviewed fe319e16838a7d408ac330afae2d3ec9d3bb0780, including the complete recipe diff and the follow-up to the previous HTTP-status finding.

The original finding is fixed. An independent probe returning identical HTTP 500 JSON bodies with equal answers exits 0 on the previously reviewed revision and exits 1 on this head. The relaxed path now requires HTTP 200 and matching status/content-type. All six supplied HTTP comparison cases passed on Linux/Python 3.12; a first unittest-discovery invocation found no tests because this file uses its own runner, so validation used the documented python recipe/test_compare_with_backend.py command.

No new actionable findings. This is a COMMENT rather than merge approval: the branch currently conflicts with main. Please resolve those conflicts and rerun the comparison tests on the resulting head. I did not run CLM's full external server recipe or a GPU encoder for this recipe review.

[hsliuustc0106]

fix conflicts

[xiaoyu-xyz]

Rebased onto main (58b8cbe). The only conflict was recipe/README.md, where both sides added entries — both are kept.

python recipe/test_compare_with_backend.py is 6/6 on the rebased head.

[hsliuustc0106]

Independent review — verdict: ready to approve once rebased. The current head (9ddbca29) conflicts with main at 873655b: this PR and the just-merged #55 both add an entry to the recipe index, so recipe/README.md needs a one-line resolution. Everything below was verified at that head before main moved.

Verified locally (detached worktree, CPU only):

• python3 recipe/test_compare_with_backend.py — 6/6 ok on Python 3.9.6
• Upstream CLM's src/clm/embedder.py confirms the three assumptions the recipe rests on: the client sends "encoding_format": "base64" (so the stub's base64 float32 responses are exactly what a real vLLM pooling server returns for this client), it health-checks GET /v1/models (the stub serves it), and its LRU cache reports "encoder tokens spent on cache misses" — precisely the field the comparator relaxation excludes, and why.
• No stale references to recipe/laya/compare_with_backend.py remain; the README.md#compare-responses anchor still resolves.

Findings (non-blocking):

1. The relaxation compares the JSON-parsed answers subtree, so key-order/whitespace differences also pass, not just usage. Acceptable for a recipe tool (strict byte equality is tried first, so pure-function backends are unaffected) — noting it because the relaxation is slightly wider than "usage-only".
2. stub_embedder.py: Handler.calls is incremented but never read — dead code.
3. recipe/laya/apple-silicon.md:140 still describes compare_with_backend.py as "from the Laya text worker recipe" — stale after the move to recipe/.
4. The full three-terminal run (stub + clm-serve + frontend, X-CLM-Latency-Ms pass-through) is author-reported and was not re-run here; the comparator logic itself is covered by the tests above.

Process note: after the rebase, this fork PR's workflow runs will be held in action_required again and will need maintainer approval to run.

[xiaoyu-xyz]

Rebased onto main (6d92519); the only conflict was the recipe index, where #55 and this PR each added an entry, so both are kept.

Reran the three-terminal setup end to end on this head — stub embedder, clm-serve, omni-jev — since it was author-reported before. 5/5 PASS, exit 0:

PASS health, department, urgency, refund, combined

The three answer types come back through the frontend unchanged, including the score answer's legend, and the usage difference is reported rather than asserted — input_tokens 33 → 0 on the first decision, which is the cache-miss semantics the relaxation exists for.

Also corrected, from your findings and one more found while running it:

• CLM_CKPT_DIR was wrong. clm-serve reads CLM_CKPT (the file path) and --ckpt-dir; there is no CLM_CKPT_DIR. As written, the recipe's command ignored it, found nothing at the default ~/.cache/clm, and downloaded a second copy of the 75 MB checkpoint. Now CLM_CKPT=/tmp/CLM_v0.1-8B.pt, and the line above it says what the variable is.
• The comparator's docstring claimed a byte comparison. It is byte-first with a parsed-answers fallback, and that fallback also ignores key order and whitespace — wider than "usage only", now stated instead of implied.
• stub_embedder.py: dropped the Handler.calls counter nothing read.
• apple-silicon.md no longer calls the comparator the Laya recipe's; it is shared and lives in recipe/.

python3 recipe/test_compare_with_backend.py is 6/6 on the head.

[Levius-Fubuki]

lgtm