Repository navigation
docs: add the 2026-10-21 Tic Tac Toe series results - #11
Merged
Merged
Conversation
Haiku, Sonnet and Opus, each at effort low, medium and high, with both prompts on all four profiles: 72 tests, ordered by model, effort, profile and prompt. Assisted-by: Claude Opus 5.5 (code generation)
216 runs: Haiku, Sonnet and Opus 5.5 at effort low, medium and high, with both prompts on all four profiles, three runs each. The folder keeps the raw results and the prompts, follow-ups, goal, assessment and matrix the series ran with. The README summarizes them by model and effort, and compares models, efforts, prompts and profiles each over all other parameters. Assisted-by: Claude Opus 5.5 (code generation)
- The results README now rounds from exact medians (39 % and 79 % for the prompts, 194 s and 174 s), ends the series at 14:26, puts Sonnet at about 43 % of Opus and the spread at up to 7.5×, and states the prompt and effort reliability differences as tendencies. - The profile statement now says what the data shows: more start context and cost, while output and duration do not follow the profile size. - README and lab.py help no longer name test1 and test3, which the new matrix does not have; they use a test id from lab/matrix.conf. Assisted-by: Claude Opus 5.5 (code generation)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Results of the first Tic Tac Toe series: Haiku, Sonnet and Opus 5.5 at effort low, medium and high, with both prompts on all four profiles, three runs each (216 runs).
Changes
lab/matrix.conf: the series' 72 tests, each line with its model and effort, ordered by model, effort, profile and prompt.lab.pyhelp: the run and session examples use a test id fromlab/matrix.confinstead oftest1andtest3, which the new matrix does not have.docs/results/2026-10-21-tic-tac-toe/: the rawresults.csvand copies of the prompts, follow-ups, goal, assessment and matrix the series ran with. The README describes the setup, gives medians by model and effort, compares models, efforts, prompts and profiles each over all other parameters, and lists observations and limitations.Decisions
run allrepeats the series. The previous nine tests (test1totest9) go.Tests
No code change; the help-text test expects the new example.
Assisted-by: Claude Opus 5.5 (code generation)
Assisted-by: Claude Opus 5.5 (code review)