Repository navigation
Conversation
…_S (0.1.44.1) Results-only community benchmark; details and limitations in the report README.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #2581 (closed in the 0.2.0-beta tracker reset; branch rebased onto current
main).Area: [Benchmark]
Problem and result
There was no community datapoint for Qwen3.6-35B-A3B on an 8 GiB consumer card. This adds one: a Strata 0.1.44.1 run on an RTX 3070 Ti + Ryzen 5 7600X with a community abliterated Q5_K_S build substituted into the install slot. Before: no result in
bench/results/for this hardware class. After: a result directory in the same shape as the existing community entries.Related discussion
none
Approach
Two files, both new, no code touched:
bench/results/2026-10-15-community-rtx-3070-ti-qwen3-6-35b-a3b-ud-q4-k-xl/README.md— hardware and software, model and configuration, method, results, correctness notes, limits.…/results.json— per-request data from the suite.Headline numbers (medians, 3 runs): short chat decode 50.6 tok/s (46.6–60.7); 4K fresh prompt 1,189 tok/s prompt / 3.40 s TTFT; 16K fresh prompt 1,100 tok/s prompt / 14.62 s TTFT; 32K-context agent session decode 35.5 tok/s (22.4–47.3) with TTFT 8.28 s (3.31–14.05). VRAM steady at 7,666 of 8,192 MiB; all routed experts run from pageable host RAM at this quant.
Validation
Revision: rebased onto
mainat011c9e60(the 0.2.x line). This PR is documentation-only — no code, no runtime path — so the checks that apply are the added JSON parsing and the file shape.Passed:
python -m json.tool results.jsonon the added file (valid). The benchmark itself was run withpython tools/strata_bench.pyagainst a live HTTP server (model load not timed, one warm-up plus a 1K prompt discarded, unique prompt tags so the conversation cache cannot help). Results parsed from the engine's own timings; per-request data inresults.json. A separate 5/5 correctness battery (arithmetic, the five-widgets puzzle, executed fizzbuzz, capital of Australia, exact bullet format) is recorded in the README, not part of the suite.Failed: none observed.
Not run: no rebuild of the engine (official Windows CUDA 13 release binary used as-is); no CUDA/HIP/SYCL/Metal variants; single machine, single session; no correctness suite integrated into the benchmark.
Limits
Numbers are single-session medians on a box at its RAM edge (free RAM ranged 0.2–3.8 GiB during the runs; chat decode dips to ~14 tok/s when Windows trims model pages), so expect variance with different background load. The model is a community abliterated fine-tune (Q5_K_S, MTP block preserved) substituted into the UD-Q4_K_XL slot — the server label comes from that slot. A second suite run and a 4-client concurrency run on the same day are documented in the README as additional measurements, not part of the primary table.