Skip to content

Derive preset decode rate from E2E latency - #4230

Closed
SiluPanda wants to merge 1 commit into
dstackai:masterfrom
SiluPanda:issue_4225_correct_preset_tps_user
Closed

SiluPanda wants to merge 1 commit into
dstackai:masterfrom
SiluPanda:issue_4225_correct_preset_tps_user

Conversation

@SiluPanda

Copy link
Copy Markdown

Summary

  • require end-to-end latency in new preset-agent benchmark reports and derive per-user decode throughput from E2E latency, TTFT, and measured output-token totals
  • stop interpreting SGLang's chunk-level ITL as TPOT by allowing unavailable TPOT values to be reported as null
  • use the derived rate in both stored preset rows and in-progress session rows, while retaining a TPOT fallback for presets and sessions created by older clients
  • reject fresh reports whose claimed TPOT would require more decode time than the measured duration and concurrency can supply

The agent prompt now calls out SGLang's speculative-decoding behavior and explains which latency fields to record. At the storage boundary, the stricter fresh-report model is normalized back to the backward-compatible persisted model.

Fixes #4225.

Validation

$ uv run pytest -q src/tests/_internal/cli/services/presets src/tests/_internal/cli/models/test_presets.py src/tests/_internal/core/models/test_presets.py
305 passed, 2 skipped in 3.30s

$ uv run ruff format --check <changed Python files>
11 files already formatted

$ uv run ruff check <changed Python files>
All checks passed!

$ git diff --check
# no output

AI-assisted implementation; I reviewed the diff and ran the checks above locally.

@r4victor r4victor closed this Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Preset reported tps/user ~10x too low with sglang bench and speculative decoding

2 participants