QuantCheck

A filing can become public on May 10 even though its quarter ended on March 31. If a dataset labels that value as available on March 31, a mathematically correct calculation can still run on a historical state that never existed.

QuantCheck is a deterministic Python framework for testing whether timestamp leakage, unit drift, duplicate observations, or revision overwrites can manufacture a misleading research result. It is a research-integrity tool rather than a trading system.

Demo · Quick start · Benchmark · Evidence archive · Research

QuantCheck keeps period, filing, availability, research as-of, and runtime dates distinct. The v0.1 visibility contract is deliberately narrow and day-level:

visible ⇔ available_on <= as_of_date

How it works

The design constraint is that a detector must never see the answer key. Truth about an injected fault stays private until findings are already final.

1. Build a clean point-in-time snapshot.
2. Inject one deterministic, configured fault and keep its truth private.
3. Sanitize the corrupted snapshot into an AuditInputSnapshot.
4. Run detectors without the manifest, clean values, target IDs, seed, or injector metadata.
5. Finalize findings, then score them against the private manifest.
6. Compare one controlled research output across clean, corrupted, and replayed states.

The public presentation layer reads public artifacts only. It does not rerun scientific logic or require the private artifact tree.

Controlled fault families

The frozen v0.1 benchmark covers one narrow subtype per family. Each is a controlled contract, scoped to the question in the right-hand column, rather than general timestamp, anomaly, deduplication, entity-resolution, or restatement detection.

Family	Subtype	Detector question
Look-Ahead Timestamp	period_end_substitution	Did a period-end date make a fact visible before filing?
Unit Drift	value_scaled_unit_unchanged	Is there a scale discontinuity in an exact comparable series?
Duplicate Observations	exact_occurrence_copy	Does one exact public fingerprint occur more than once?
Revision Overwrite	later_vintage_in_earlier_state	Does an earlier state contain a later declared revision?

Evidence at a glance

The immutable v0.1 controlled benchmark has 124 configured cases: 94 successful, 30 structural no-target, and 0 incomplete. Among successful scored cases it recorded 130 injected faults, 130 exact matches, 78 strict cross-detector false-positive findings, precision 0.625, recall 1.000, and F1 0.769.

These are synthetic fixture measurements. They characterise the framework on its reviewed fixture and say nothing about production performance. The numbers above are read back from the canonical aggregate report; the benchmark summary walks through where every false positive comes from and why three profile/severity cells have no eligible target at all.

Two smaller studies sit alongside it, and neither one generalises the benchmark:

• The observational SEC study processed 472 selected observations from five issuers and emitted no findings. Only Exact Duplicate had applicable opportunities. It shows that the public-source pipeline runs end to end under real constraints, which is a much narrower claim than natural-error discovery or broad detector validation.
• The SEC-derived adversarial study manufactured 120 faults on preserved SEC observations. Those 120 faults were injected by design, so they are evidence about detector behaviour on real-shaped data rather than a count of natural SEC defects.

Quick start

Requires Python 3.12 and uv.

uv sync --frozen --all-groups
uv run quantcheck benchmark smoke --output /tmp/qc-smoke
uv run python scripts/render_html_summary.py /tmp/qc-smoke \
  --output /tmp/qc-smoke/summary.html

The smoke path runs 12 deterministic offline cases. For the read-only dashboard:

uv run --group dashboard streamlit run dashboard/app.py -- --artifacts /tmp/qc-smoke

Every ordinary test and quick-start command runs offline. None of them makes a live network request.

Reproduce the frozen baseline

The held-out baseline lives in the immutable v0.1.0 release. The complete procedure is in Reproducibility. The current worktree is 0.2.0.dev0, which is development state and carries no release claim of its own.

uv sync --frozen --all-groups
uv run python scripts/release_freeze.py --check
uv run python scripts/run_release_benchmark.py --output release_evidence/final
uv run python scripts/verify_release_evidence.py \
  --source release_evidence/final \
  --public-only release_evidence/public_only \
  --html release_evidence/public_only/summary.html

The freeze check has to pass before the benchmark will dispatch a single case.

Demo

Review the production notes. The immutable release-hosted narrated demo is available as the QuantCheck_demo.mp4 release asset. The accompanying real-data evidence archive contains the raw study payloads that are deliberately kept out of this tip.

My role

I framed the problem, defined the scientific contracts and evidence boundaries, set the verification gates, and made the final publication calls. Implementation used AI-assisted development under acceptance criteria, tests, and review that I wrote and enforced, so I am not claiming that every line was typed by hand. What the repository actually contributes is the contract-and-verification design: results that came out inconvenient are still in the evidence record, unedited.

Limitations

The primary benchmark is synthetic and fixture-bounded. The observational SEC run has no ground truth, and only one detector there has a nonzero opportunity denominator. The adversarial SEC-derived study injects manufactured faults into a reused substrate.

On the release side, candidate source commit 3b47da9 has a green Security workflow run 32693592309 covering exact two-build OCI identity plus dependency, secret, image, and SBOM gates. Still missing from the current worktree: a customer pilot, customer adjudication, production deployment, a published image digest, a trusted candidate attestation, and a held-out v0.2 result.

QuantCheck makes no claim about alpha, returns, prevented losses, natural SEC defect prevalence, vendor-scale coverage, or automatic remediation.

See the current status and authoritative limitations.

Research

The research archive is retained in the current tree:

• Research paper
• Public evidence ledger
• Observational SEC results
• SEC-derived adversarial results
• Adversarial adjudication
• Real-data limitations

For implementation semantics, start with Methodology. The evidence archive holds the saved public and study-scoped artifacts. Private truth stays private and is never presented as a public performance number.