Skip to content

[CI] Make local, PR, hardware, and release quality gates truthful #140

Description

@proggeramlug

Parent: #126

Problem

The repository's quality contract is fragmented:

  • scripts/ci-check.sh describes itself as local parity but omits several CI/release-sensitive checks;
  • rustfmt/clippy are advisory in .github/workflows/test.yml;
  • example compilation and real browser startup are not a complete matrix;
  • reference validation is report-oriented and not a required gate;
  • FFI and file-size ratchets can fail independently of the command contributors are told to run.

An engine cannot protect high-end rendering work if “green locally” and “green in CI” mean different things.

Outcome

One versioned check manifest/entry point used by local development, PR CI, and release gating, with explicit quick/full/hardware lanes and no ignored required failures.

Design

Single orchestration source

Use one script/tool (the existing scripts/ci-check.sh may evolve) to define named lanes:

  • quick: formatting/lint baseline, unit tests, FFI/schema validation, file ratchets, representative example compile;
  • full: all host builds, shared/web checks, all example compile checks, goldens that can run on the host;
  • web: build/package plus browser startup/render smoke;
  • hardware: quality corpus and GPU/path-tracing cases;
  • release: full plus packaging/install smoke for artifacts.

CI invokes the same lanes rather than reimplementing command lists in YAML. Commands must emit machine-readable summaries and preserve normal console output.

Baselines and rollout

  • Fix existing violations or record a narrow ratcheting baseline with owner/removal issue.
  • continue-on-error is allowed only for a named migration with an expiry/issue; required checks cannot be advisory indefinitely.
  • A test may skip only with a structured reason and the lane declares which skip reasons are acceptable.
  • Dirty worktree diagnostics are reported locally but do not mutate source/baselines.

Required coverage

  • cargo fmt --check and clippy policy for maintained Rust crates;
  • shared unit/integration/golden tests;
  • FFI/schema parity and public API generation freshness;
  • file-size/monolith ratchets;
  • native platform builds available to the runner;
  • WASM check/build and real-browser smoke;
  • compile every canonical TypeScript example against its checked-in package/manifest;
  • release artifact contents/startup, including Windows shader compiler dependencies;
  • quality/hardware lane from the quality harness issue.

Acceptance criteria

  • ./scripts/ci-check.sh --quick (or replacement) runs every PR-required platform-independent check and exits non-zero on any failure.
  • CI YAML delegates to the same named lanes; a test verifies the command inventory cannot silently diverge.
  • No required lint/format/FFI/example check uses continue-on-error or || true.
  • All canonical examples compile in CI; at least one native and one web example reaches and renders a known frame in smoke testing.
  • PR failures upload useful artifacts (golden diffs, browser logs/screenshots, capability report) without requiring rerun access.
  • Release workflow gates the exact tag SHA on required lanes and package-install smoke.
  • A contributor setup section lists tool prerequisites and gives one quick and one full command.

Likely files

  • scripts/ci-check.sh
  • .github/workflows/test.yml, .github/workflows/release.yml
  • tools/validate-ffi.js, tools/check-file-lines.js, quality tooling
  • example package.json files and top-level docs

Non-goals

  • Running expensive GPU qualification on every hosted runner/commit.
  • Reformatting unrelated user work as part of the issue.
  • Treating compilation as sufficient browser/runtime validation.

Dependencies

Can start immediately. Add quality/hardware lane when that child issue lands.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions