Nightly fuzzing with a moving seed (#98) - #141
Merged
Merged
Conversation
tests/test_fuzz.py's "netprotocols" Hypothesis profile is deliberately deterministic (200 examples, derandomize=True) so a pull request stays reproducible — but that also means every build since the profile was written has run the same 200 inputs. Adds a second "nightly" profile (10,000 examples, a real random seed each run), selected at import time via HYPOTHESIS_PROFILE, plus a new scheduled workflow (.github/workflows/fuzz.yml, 03:00 UTC daily + manual dispatch) that runs the whole suite under it. Kept out of ci.yml deliberately: ci.yml is reused by release.yml via workflow_call specifically so the release gate and the PR gate never drift apart, and a schedule trigger there would fire that whole ladder on a cron too. .hypothesis/'s example database accumulates across nightly runs via actions/cache, keyed by run id with a prefix restore-key (a fixed key would restore forever but never actually save a new entry, so nothing would accumulate); a failure additionally uploads it as a workflow artifact so a counterexample is recoverable without repo write access. A failed scheduled run's own red build is the notification GitHub already sends by default, so nothing here auto-files an issue on top of it. Also adds a targeted Hypothesis strategy building well-formed TCP SYN options (MSS, window scale, SACK-Permitted, SACK, timestamps): the real-capture corpus never caught a SYN, so unlike NOP/Timestamps these option kinds were otherwise exercised only by hand-built examples in test_tcp.py, dependent on max_examples alone landing on a well-formed TLV by chance under the generic fuzz strategies. Flips docs/CLAIMS.md 5.2's "PARTLY TRUE" language now that both halves of the claim (universal round-trip generation, and a moving seed) are landed. Closes #98. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MGDTcK51CWcy6PrNetN213
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tier 4 (#106), item 2 of 3: replaces the fuzz suite's fixed, deterministic 200-example run with a scheduled nightly run that explores fresh ground every time. Closes #98.
What's included
tests/test_fuzz.py: the existing"netprotocols"profile (200 examples,derandomize=True, unchanged — every push/PR stays reproducible) and a new"nightly"profile (10,000 examples,derandomize=False— a real random seed each run). Selected viaHYPOTHESIS_PROFILE, read once at import time, soHYPOTHESIS_PROFILE=nightly pytestreproduces a nightly run locally..github/workflows/fuzz.yml:schedule(0 3 * * *UTC) +workflow_dispatch, Python 3.12 only (an exploratory job, not a compatibility gate — 3x the interpreters would 3x its cost for no extra coverage of what it's looking for). Kept out ofci.ymldeliberately:ci.ymlis reused byrelease.ymlviaworkflow_callspecifically so the release gate and the PR gate never drift apart, and aschedule:trigger there would fire that whole ladder on a cron too..hypothesis/cache:actions/cache, keyed bygithub.run_idwith ahypothesis-nightly-prefixrestore-keysfallback. (Note for reviewers: the original plan called for "a cache key that doesn't change per-run" — that doesn't actually accumulate anything, sinceactions/cachenever re-saves once an exact key already exists. The run-id+prefix pattern here is the standard fix and does what was intended: each run restores the most recent cache and saves a new entry.) On failure,.hypothesis/is additionally uploaded as a workflow artifact so a counterexample is recoverable without repo write access. No custom issue-filing — a failed scheduled run's own red build is GitHub's default notification to watchers, which is the simpler of the two options the issue named.tcp_syn_optionsintest_fuzz.py): MSS, window scale, SACK-Permitted, SACK, and timestamps, well-formed and NOP-padded. The real-capture corpus never caught a SYN (tests/fixtures/MANIFEST.md), so unlike NOP/Timestamps (kinds 1, 8, exercised via other captured traffic), these option kinds were otherwise reachable only throughtest_tcp.py's hand-built single examples — the existing genericfuzz_inputstrategy essentially never lands on a well-formed multi-option TLV sequence by chance.docs/CLAIMS.md5.2 flipped fromPARTLY TRUEtoVERIFIED— both halves of the claim (universally-generated round-trips from Generalize the round-trip property to every protocol #97, and a moving seed here) are now landed.CHANGELOG.mdentry under## [Unreleased].Not included (used judgment per the handoff, not required): a completeness assertion reconciling
test_fuzz.py::ALL_PROTOCOLSagainststrategies.py::ROUND_TRIP_STRATEGIES— the two lists differ by design (DNSOverTCPis round-trip-tested but deliberately excluded from the chain-walk fuzz properties), so a correct assertion needs more scaffolding than a drive-by warrants; skipped rather than ship something fragile.Verification
uv run ruff checkanduv run ruff format --checkare cleanuv run mypyis clean (strict)uv run pytestpasses locally — 1057 passed (200-example profile)CHANGELOG.mdhas an entry under## [Unreleased]HYPOTHESIS_PROFILE=nightly uv run pytest— the actual nightly job's workload, run locally end-to-end: full suite, 10,000 examples, real random seed — passes clean in ~7m14s. That's also useful signal on the scheduled job's real runtime.Notes
fuzz.ymlonly runs onschedule/workflow_dispatch, so this PR's own CI run will not exercise it — by design, the same reason it isn't folded intoci.yml. It'll get its first real run at 03:00 UTC after merge (or dispatch it manually to check sooner).Closes #98.
🤖 Generated with Claude Code
https://claude.ai/code/session_01MGDTcK51CWcy6PrNetN213
Generated by Claude Code