Skip to content

fix(v1): preserve cache usage in train responses - #2542

Draft
hallerite wants to merge 1 commit into
fix/eval-long-override-displayfrom
fix/train-cache-usage
Draft

fix(v1): preserve cache usage in train responses#2542
hallerite wants to merge 1 commit into
fix/eval-long-override-displayfrom
fix/train-cache-usage

Conversation

@hallerite

@hallerite hallerite commented Sep 5, 2026

Copy link
Copy Markdown
Member

Stacked on the compact-eval-overrides branch to preserve the frozen research base.

Change

Consume optional generate-endpoint usage through CompletionUsage and the existing
Usage.from_openai conversion. This keeps cached prompt tokens out of Usage.prompt_tokens and
preserves optional reasoning details. Responses without usage still use exact token lengths.
The existing serialize_completion already forwards those details to the harness SDK.

Requires renderers' companion fix/forward-generate-usage change for live metadata forwarding.
Without it the existing exact-length fallback remains. No sampling, rendering, graph, numerical
budget, or optimizer changes. The server already reports usage; no serving restart is needed.

Validation

  • 84 deterministic verifier tests passed; live-provider/sandbox e2e cases excluded.
  • Six model-free cases cover unknown/zero/partial/full cache, reasoning subsets, and missing usage.
  • Two tiny live synthetic requests through the actual TrainClient and SDK roundtrip passed.
    Repeated 1,045-token prompt: 1,040 cached + 5 uncached; with 3 output, charge is 8 new tokens.
  • Changed-file hooks passed. All-files markdownlint/format passed; unrelated pre-existing E402
    remains in verifiers/v1/harnesses/hermes_agent/program.py:16.
  • Type checking was attempted but ty is not installed in the active research environment.

Isolated commits are frozen for fresh local canaries. Existing full comparisons still use their
original dependencies; no result is silently resumed under the fix. Unsigned commit.

Note

Fix response_from_generate to preserve provider CompletionUsage in train responses

  • When the generate result includes usage, the parser now validates it as CompletionUsage and converts it via Usage.from_openai, preserving cache and reasoning-token fields from the provider.
  • Falls back to the existing prompt/completion token-id length calculation when usage is absent.
  • Risk: responses with malformed usage objects may now fail validation instead of silently falling back.

Macroscope summarized e4f51c0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant