From b203e4281eef062dd00ff67842b5bf3508fa7890 Mon Sep 17 00:00:00 2001 From: Esau Rodriguez Date: Mon, 28 Sep 2026 12:34:45 +0100 Subject: [PATCH] Bump libhegel to 0.44.0: nondeterminism handling and engine-owned final replays The engine now confirms a nondeterministic failure by replay and reports it with a caveat instead of aborting; Settings.nondeterminismStrictness (and the @HegelTest attribute / HEGEL_NONDETERMINISM_STRICTNESS) selects QUIET, WARN or the old ERROR abort. The engine also runs each reported failure's final replay itself, so the runner stops replaying blobs and builds every Failure from the case the engine stamped via hegel_test_case_should_capture. reproduceFailure now goes through hegel_run_start_blob, which replays until a replay fails. Failure.reproduceBlob() becomes Optional, exception() is always present, and flaky() is replaced by nondeterministic()/caveat(). Libhegel gains the new capture, caveat, blob-run and strictness functions; the retired RUN_STATUS_FAILED_NONDETERMINISTIC constant is removed. Co-Authored-By: Claude Fable 5.1 --- .claude/CLAUDE.md | 40 +++- README.md | 10 +- RELEASE.md | 79 +++++++ .../src/main/java/dev/hegel/JnaLibhegel.java | 69 ++++++ .../test/java/dev/hegel/JnaLibhegelTest.java | 23 ++ .../src/main/java/dev/hegel/lowlevel/Abi.java | 10 +- .../java/dev/hegel/lowlevel/Libhegel.java | 50 +++- .../src/main/java/dev/hegel/RealLibhegel.java | 71 ++++++ .../test/java/dev/hegel/RealLibhegelTest.java | 24 ++ pom.xml | 2 +- shared/src/main/java/dev/hegel/Failure.java | 79 ++++--- shared/src/main/java/dev/hegel/HegelTest.java | 9 + .../java/dev/hegel/HegelTestExtension.java | 1 + .../dev/hegel/NondeterminismStrictness.java | 45 ++++ .../main/java/dev/hegel/PrintingReporter.java | 13 +- shared/src/main/java/dev/hegel/Reporter.java | 51 ++-- shared/src/main/java/dev/hegel/RunReport.java | 14 +- shared/src/main/java/dev/hegel/Runner.java | 219 +++++++++--------- shared/src/main/java/dev/hegel/Settings.java | 36 ++- shared/src/main/java/dev/hegel/TestCase.java | 82 ++++--- .../src/test/java/dev/hegel/CoverageTest.java | 3 + .../java/dev/hegel/EnvironmentFixture.java | 14 +- .../test/java/dev/hegel/EnvironmentTest.java | 10 + .../src/test/java/dev/hegel/FakeLibhegel.java | 55 ++++- .../test/java/dev/hegel/LibraryApiTest.java | 16 +- .../src/test/java/dev/hegel/ReporterTest.java | 17 +- .../java/dev/hegel/ReproduceFailureTest.java | 96 +++++++- .../test/java/dev/hegel/RunReportTest.java | 120 +++++++--- .../src/test/java/dev/hegel/RunnerTest.java | 153 ++++++++---- .../src/test/java/dev/hegel/TestCaseTest.java | 26 ++- 30 files changed, 1101 insertions(+), 336 deletions(-) create mode 100644 RELEASE.md create mode 100644 shared/src/main/java/dev/hegel/NondeterminismStrictness.java diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 575f615..df5d39c 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -114,12 +114,26 @@ public `Generator`/`TestCase`/`Generators`/`Hegel`/`Stateful` surface stays in ` `AssumeRejected` → INVALID, `INVALID_ARG` → `IllegalArgumentException` with the engine's diagnostic, other negatives → `HegelException`) and short-circuiting once a case is aborted. - **Run loop** — `Runner` builds the settings handle and drives `hegel_run_start` → - `hegel_next_test_case` → `hegel_mark_complete`. The engine only *explores* (generation and - shrinking), so every pumped case is non-final; after the loop drains, `hegel_run_result` yields - PASSED, FAILED, or ERROR. On FAILED the runner replays each distinct counterexample's - **reproduce blob** (`hegel_test_case_from_blob`) with reporting enabled — capturing the shrunk - draws, notes, and the body's own exception into a `Failure` (a replay that no longer fails is a - *flaky* failure). The runner returns a `RunReport` (status, client-side `RunStatistics` counted + `hegel_next_test_case` → `hegel_mark_complete`. The engine owns the *whole* run — generation, + shrinking, the replays that confirm a failure, and (since 0.44) the final replay of every failure + it is about to report — so the runner never replays anything itself. It reads + `hegel_test_case_should_capture` once per case: a *stamped* case's `TestCase` records its + top-level draws and notes (`isFinal()` exposes the stamp), and every interesting case's exception + is kept per origin in a capture map (a stamped capture outranks an unstamped one, newest wins at + equal rank — the unstamped discovery of a failure that never fails again is all there is for an + *unconfirmed* nondeterministic failure). After the loop drains, `hegel_run_result` yields PASSED, + FAILED, or ERROR. On FAILED each distinct failure is built from its origin's capture: the recorded + draws/notes are replayed to the reporter (`caseStarted(true)` … `caseFinished(true)`), and the + `Failure` carries the body's own exception, the reproduce blob (`hegel_failure_reproduction_blob`, + null for an unconfirmed failure and for a blob replay) and the **caveat** + (`hegel_failure_caveat`, non-null for a nondeterministic failure — `Failure.nondeterministic()`; + the printing reporter prints it as `note: …`). A failure whose origin has no capture is an + internal error. `Settings.reproduceFailure` drives the same loop over `hegel_run_start_blob` + (the engine replays until a replay fails, under its budget): PASSED → `Runner.STALE_BLOB` + `HegelException`, ERROR → "blob is not valid" `HegelException`. There is no Java-side flaky + detection any more: the engine handles nondeterminism per `nondeterminismStrictness` + (`NondeterminismStrictness.QUIET`/`WARN` confirm-by-replay and report with a caveat; `ERROR` + aborts the run, surfacing as the engine's own `Flaky test detected` message). The runner returns a `RunReport` (status, client-side `RunStatistics` counted per `mark_complete` — the C ABI has no counter accessor — engine error, failures) and never prints: everything goes through the run's `Reporter` (`Reporter.printing(System.err)` is the default and reproduces the classic output; `Reporter.silent()` for library use). `Hegel.run` @@ -127,17 +141,17 @@ public `Generator`/`TestCase`/`Generators`/`Hegel`/`Stateful` surface stays in ` failure as-is (checked exceptions included), aggregates several into an `AssertionError`, and maps ERROR to `HealthCheckFailure`/`HegelException`. `HegelException` (binding/engine errors) is always thrown, never reported. `Settings` is the immutable config; its closed-state setting types live - alongside it — `Backend` (auto = leave it to the engine's profile, which picks urandom inside Antithesis / default / urandom), `Database`, `OptBoolean` — + alongside it — `Backend` (auto = leave it to the engine's profile, which picks urandom inside Antithesis / default / urandom), `Database`, `OptBoolean`, `NondeterminismStrictness` (`DEFAULT` = leave it to the engine) — plus `printBlob` (print a copy-pasteable reproducer per failure) and `reproduceFailure` (replay - a stored blob instead of running the property). `testCases`, `printBlob`, `derandomize`, `seed` - and `database` are nullable ("unset"): `Runner.applySettings` sends only what the user set, so + a stored blob instead of running the property). `testCases`, `printBlob`, `derandomize`, `seed`, + `nondeterminismStrictness` and `database` are nullable ("unset"): `Runner.applySettings` sends only what the user set, so the engine's resolution — its profile (`hegel.toml`, the shipped `development`/`ci`/`workload` profiles) with the `HEGEL_TEST_CASES`/`HEGEL_DATABASE`/`HEGEL_SEED`/`HEGEL_DERANDOMIZE`/ - `HEGEL_PRINT_BLOB` variables applied over it in `hegel_settings_new` — stands for the rest, and + `HEGEL_PRINT_BLOB`/`HEGEL_NONDETERMINISM_STRICTNESS` variables applied over it in `hegel_settings_new` — stands for the rest, and explicit Java settings win over both. There is no Java-side CI detection (the engine's `ci` profile covers it). `Libhegel.settingsNew` returns a raw result code because a malformed variable or `hegel.toml` fails it with `E_INVALID_ARG` (→ `IllegalArgumentException`); after - applying, the runner reads `settingsGetTestCases`/`settingsGetPrintBlob` back and hands the + applying, the runner reads `settingsGetTestCases`/`settingsGetPrintBlob`/`settingsGetNondeterminismStrictness` back and hands the *effective* `Settings` to the reporter and the run. Verbosity and `reportMultipleFailures` keep Java-side defaults and are always sent. `EnvironmentTest` covers the variables and `hegel.toml` through a child JVM (`EnvironmentFixture`), since the engine reads its own process environment. @@ -173,7 +187,9 @@ public `Generator`/`TestCase`/`Generators`/`Hegel`/`Stateful` surface stays in ` for `@Invariant(alwaysRun = true)`, sampled otherwise; the initial and final checks run every invariant unconditionally. The machine handle is freed in a `finally`. An engine-level `E_ASSUME` inside a rule (the data source is aborted) unwinds the whole case as invalid instead - of being reported as the rule's rejection. `Pool` tracks previously generated values over + of being reported as the rule's rejection. (Since engine 0.44 concurrency alone no longer marks + a run nondeterministic, and `HEGEL_RUN_STATUS_FAILED_NONDETERMINISTIC` is retired; this binding + drives machines sequentially anyway.) `Pool` tracks previously generated values over the engine's pool primitives so rules can reuse or consume them. - **Derivation** — `dev.hegel.generators.Derive` + `RecordGenerator` build generators from records, enums, scalars, and generic `List`/`Set`/`Optional`/`Map` by reflection. diff --git a/README.md b/README.md index a5bdd1d..87167d9 100644 --- a/README.md +++ b/README.md @@ -109,13 +109,15 @@ report.statistics(); // valid / invalid / overrun / interesti for (Failure f : report.failures()) { // one per distinct counterexample f.draws(); // labelled draws of the minimal example, as Java values f.exception(); // the body's own throwable - f.reproduceBlob(); // replay it later with Settings.reproduceFailure + f.reproduceBlob(); // Optional: replay it later with Settings.reproduceFailure + f.caveat(); // Optional: how reliably a nondeterministic failure reproduced } ``` -`TestCase.isFinal()` identifies the final replay of a counterexample, `TestCase.span` and `Label` -let custom composite generators tell the engine about their structure, and -`Settings.infrastructurePackages` keeps a frontend's own stack frames out of failure origins. +`TestCase.isFinal()` identifies the executions the engine stamped for the failure report (the +final replay of a counterexample among them), `TestCase.span` and `Label` let custom composite +generators tell the engine about their structure, and `Settings.infrastructurePackages` keeps a +frontend's own stack frames out of failure origins. ## Binding Hegel yourself diff --git a/RELEASE.md b/RELEASE.md new file mode 100644 index 0000000..271edb3 --- /dev/null +++ b/RELEASE.md @@ -0,0 +1,79 @@ +RELEASE_TYPE: minor + +This release upgrades the bundled libhegel engine from 0.43.4 to 0.44.0, which handles +nondeterministic tests instead of refusing them, and changes the `Failure` API accordingly. + +A test whose outcome changes when the same generated data is replayed — because it depends on hidden +global state, time, an outside service, or thread scheduling — previously aborted the run with a +`HegelException` reading "Flaky test detected". The engine now switches such a run to +nondeterministic handling: a discovered failure is confirmed by repeated replay before it is shrunk +or saved to the database, shrunk under a guard on how much reproduction reliability a shrink step may +trade away, and reported with a *caveat* quoting the run's own replay evidence. `Hegel.test` then +throws the test's own exception as for any other failure, and the printed report carries the caveat: + +``` +x = 11; +note: nondeterministic failure, confirmed: failed 7 of 20 replays at confirmation and 1 of 2 at report time + +To reproduce this failure, replay it with: + @HegelTest(reproduceFailure = "...") +``` + +A failure that never reproduces after its discovery still fails the run, reported as +`unconfirmed failure: failed 0 of 10 replays after the observed failure` and without a reproduce +blob. A reproduce blob from a nondeterministic failure records the failing runs the engine saw, and +`Settings.reproduceFailure` / `@HegelTest(reproduceFailure = ...)` now replays a blob until one of +its replays fails, under a bounded budget, instead of judging it stale after a single attempt; a +blob none of whose replays fail reports that "the supplied failure blob did not reproduce a failure". + +The new `Settings.nondeterminismStrictness` (and `@HegelTest(nondeterminismStrictness = ...)`) +controls the reaction: `NondeterminismStrictness.QUIET` (the engine's default) switches silently, +`WARN` prints a one-line notice once per run, and `ERROR` aborts the run with the previous +flaky-test error, for suites that use determinism as a lint. It is a profile setting like any other +(`nondeterminism_strictness = "error"` in `hegel.toml`), the `HEGEL_NONDETERMINISM_STRICTNESS` +environment variable (`quiet`, `warn` or `error`) overrides it for one run, and a value set in Java +wins over both. + +The engine now also runs every failure it is about to report one final time itself, so hegel-java no +longer replays reproduce blobs after a run: the reported draws, notes and exception come from the +engine's own final replay, and a failing test's body runs once less per reported failure. Two +visible consequences. `TestCase.isFinal()` is now true on every execution the engine stamps for the +failure report — the final replay of a counterexample, but also the short replays that confirm a +discovered failure — so a body that uses it to gate expensive diagnostics may run them a few times +per failure rather than once. And a `Reporter` receives the counterexample's `draw` and `note` +callbacks (flagged `finalReplay = true`) when the failure is reported, after the run loop, rather +than live while a replay executes; the order of callbacks is unchanged. + +`Failure` changes shape to carry the caveat and to admit failures without a blob: + +```java +// before +String blob = f.reproduceBlob(); +Optional e = f.exception(); +if (f.flaky()) { ... } + +// after +Optional blob = f.reproduceBlob(); // empty for an unconfirmed failure or a blob replay +Throwable e = f.exception(); // always present +Optional caveat = f.caveat(); // present for a nondeterministic failure +if (f.nondeterministic()) { ... } +``` + +`Failure.flaky()` is gone: the engine no longer hands back a failure whose replay passed. A +`Failure.reproduceBlob()` is also empty for a failure reproduced from a `reproduceFailure` blob, since +the caller already holds it. + +In `dev.hegel:hegel-lowlevel`, `Libhegel` gains `runStartBlob` (`hegel_run_start_blob`), +`testCaseShouldCapture` (`hegel_test_case_should_capture`), `failureCaveat` (`hegel_failure_caveat`) +and the `settingsNondeterminismStrictness` / `settingsGetNondeterminismStrictness` pair, and +`Abi.RUN_STATUS_FAILED_NONDETERMINISTIC` is removed along with the ABI value it mirrored; a failing +nondeterministic run reports plain `RUN_STATUS_FAILED`. Bindings implementing `Libhegel` must add +the new methods. + +Between 0.43.4 and 0.44.0 the engine also improved float generation, so tests on floats reach +overflow, underflow, cancellation and special-value bugs far more often (each test case draws its own +mixture of float categories, and bounded ranges cover every power-of-two scale they touch); roughly +halved the per-case cost of text generators built inside the test body by sharing built alphabets +between generators with the same constraints; shrinks a size drawn twice — the rows and columns of a +square matrix — while the values it governs still vary, reaching the smallest failing square; and +picked up correctness fixes in its arbitrary-precision integer dependency. diff --git a/hegel-jna/src/main/java/dev/hegel/JnaLibhegel.java b/hegel-jna/src/main/java/dev/hegel/JnaLibhegel.java index aa52f8c..4c1a2f5 100644 --- a/hegel-jna/src/main/java/dev/hegel/JnaLibhegel.java +++ b/hegel-jna/src/main/java/dev/hegel/JnaLibhegel.java @@ -87,9 +87,21 @@ interface HegelNative extends Library { int hegel_settings_get_print_blob(Pointer ctx, Pointer s, ByteByReference out); + int hegel_settings_set_nondeterminism_strictness(Pointer ctx, Pointer s, int strictness); + + int hegel_settings_get_nondeterminism_strictness(Pointer ctx, Pointer s, IntByReference out); + int hegel_run_start( Pointer ctx, Pointer settings, LineCallback callback, Pointer userData, PointerByReference out); + int hegel_run_start_blob( + Pointer ctx, + Pointer settings, + String blob, + LineCallback callback, + Pointer userData, + PointerByReference out); + int hegel_next_test_case(Pointer ctx, Pointer run, PointerByReference out); int hegel_run_result(Pointer ctx, Pointer run, PointerByReference out); @@ -108,6 +120,8 @@ int hegel_test_case_from_blob( int hegel_test_case_free(Pointer ctx, Pointer tc); + int hegel_test_case_should_capture(Pointer ctx, Pointer tc, ByteByReference out); + int hegel_generate_boolean(Pointer ctx, Pointer tc, double p, byte hasForced, byte forced, ByteByReference out); int hegel_generate_integer(Pointer ctx, Pointer tc, long min, long max, LongByReference out); @@ -237,6 +251,8 @@ int hegel_state_machine_should_check_invariant( int hegel_failure_origin(Pointer ctx, Pointer failure, PointerByReference out); + int hegel_failure_caveat(Pointer ctx, Pointer failure, PointerByReference out); + int hegel_version(Pointer ctx, PointerByReference out); } @@ -430,6 +446,22 @@ public boolean settingsGetPrintBlob(long s) { return out.getValue() != 0; } + @Override + public void settingsNondeterminismStrictness(long s, int strictness) { + check( + "hegel_settings_set_nondeterminism_strictness", + lib.hegel_settings_set_nondeterminism_strictness(ctx(), pointer(s), strictness)); + } + + @Override + public int settingsGetNondeterminismStrictness(long s) { + IntByReference out = new IntByReference(); + check( + "hegel_settings_get_nondeterminism_strictness", + lib.hegel_settings_get_nondeterminism_strictness(ctx(), pointer(s), out)); + return out.getValue(); + } + // --- run lifecycle --- @Override @@ -448,6 +480,24 @@ public long runStart(long settings, Consumer output) { return run; } + @Override + public long runStartBlob(long settings, String blob, Consumer output) { + LineCallback callback = output == null ? null : new LineCallback(output); + PointerByReference out = new PointerByReference(); + int code = lib.hegel_run_start_blob(ctx(), pointer(settings), blob, callback, null, out); + if (code != Abi.OK) { + throw new HegelException("hegel_run_start_blob failed (rc=" + + code + + "): " + + java.util.Objects.toString(lastErrorMessage(), "")); + } + long run = address(out.getValue()); + if (callback != null) { + runCallbacks.put(run, callback); + } + return run; + } + @Override public long nextTestCase(long run) { PointerByReference out = new PointerByReference(); @@ -491,6 +541,13 @@ public void testCaseFree(long tc) { check("hegel_test_case_free", lib.hegel_test_case_free(ctx(), pointer(tc))); } + @Override + public boolean testCaseShouldCapture(long tc) { + ByteByReference out = new ByteByReference(); + check("hegel_test_case_should_capture", lib.hegel_test_case_should_capture(ctx(), pointer(tc), out)); + return out.getValue() != 0; + } + // --- draws --- @Override @@ -980,6 +1037,18 @@ public String failureOrigin(long result, long index) { return origin; } + @Override + public String failureCaveat(long result, long index) { + PointerByReference failureOut = new PointerByReference(); + check("hegel_run_result_failure", lib.hegel_run_result_failure(ctx(), pointer(result), index, failureOut)); + Pointer failure = failureOut.getValue(); + PointerByReference caveatOut = new PointerByReference(); + check("hegel_failure_caveat", lib.hegel_failure_caveat(ctx(), failure, caveatOut)); + String caveat = readCString(caveatOut.getValue()); + check("hegel_failure_free", lib.hegel_failure_free(ctx(), failure)); + return caveat; + } + // --- diagnostics --- @Override diff --git a/hegel-jna/src/test/java/dev/hegel/JnaLibhegelTest.java b/hegel-jna/src/test/java/dev/hegel/JnaLibhegelTest.java index 7fe4811..86a6f7b 100644 --- a/hegel-jna/src/test/java/dev/hegel/JnaLibhegelTest.java +++ b/hegel-jna/src/test/java/dev/hegel/JnaLibhegelTest.java @@ -72,6 +72,8 @@ void infrastructureCallsReportNullHandles() { // diagnostic rather than undefined behaviour. assertThrows(HegelException.class, () -> lib.runResultStatus(0)); assertThrows(HegelException.class, () -> lib.runStart(0, null)); + assertThrows(HegelException.class, () -> lib.runStartBlob(0, "blob", null)); + assertThrows(HegelException.class, () -> lib.testCaseShouldCapture(0)); } @Test @@ -197,6 +199,10 @@ void settingsGettersReadTheHandleBack() { assertFalse(lib.settingsGetPrintBlob(s)); lib.settingsPrintBlob(s, true); assertTrue(lib.settingsGetPrintBlob(s)); + lib.settingsNondeterminismStrictness(s, Abi.NONDETERMINISM_ERROR); + assertEquals(Abi.NONDETERMINISM_ERROR, lib.settingsGetNondeterminismStrictness(s)); + lib.settingsNondeterminismStrictness(s, Abi.NONDETERMINISM_WARN); + assertEquals(Abi.NONDETERMINISM_WARN, lib.settingsGetNondeterminismStrictness(s)); lib.settingsFree(s); } @@ -216,6 +222,23 @@ void undecodableBlobWithDefaultOutputIsRejected() { lib.settingsFree(s); } + @Test + void blobRunWithDefaultOutputStartsAndReportsTheBadBlob() { + JnaLibhegel lib = real(); + long s = newSettings(lib); + // A null output callback leaves the run's output on stderr; the garbage blob surfaces as + // the run's error once it is pumped dry. + long run = lib.runStartBlob(s, "not-a-blob!!!", null); + assertNotEquals(0, run); + assertEquals(0, lib.nextTestCase(run)); + long result = lib.runResult(run); + assertEquals(Abi.RUN_STATUS_ERROR, lib.runResultStatus(result)); + assertNotNull(lib.runResultError(result)); + lib.runResultFree(result); + lib.runFree(run); + lib.settingsFree(s); + } + @Test void regexGeneratorAcceptsATextAlphabet() { JnaLibhegel lib = real(); diff --git a/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Abi.java b/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Abi.java index dc97813..9a376e3 100644 --- a/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Abi.java +++ b/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Abi.java @@ -21,11 +21,17 @@ private Abi() {} public static final int E_CONCURRENT_USE = -9; public static final int E_RETRY = -10; - // Aggregate run outcome (hegel_run_status_t). + // Aggregate run outcome (hegel_run_status_t). Value 3 (FAILED_NONDETERMINISTIC, retired in the + // 0.44 ABI) is never reported and must not be reused: a failing nondeterministic run reports + // plain FAILED, with a caveat on each failure. public static final int RUN_STATUS_PASSED = 0; public static final int RUN_STATUS_FAILED = 1; public static final int RUN_STATUS_ERROR = 2; - public static final int RUN_STATUS_FAILED_NONDETERMINISTIC = 3; + + // hegel_nondeterminism_strictness_t: how a run reacts on detecting a nondeterministic test. + public static final int NONDETERMINISM_QUIET = 0; + public static final int NONDETERMINISM_WARN = 1; + public static final int NONDETERMINISM_ERROR = 2; // Phases (bitmask for hegel_settings_set_phases). public static final int PHASE_EXPLICIT = 1 << 0; diff --git a/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Libhegel.java b/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Libhegel.java index c5492e7..0bf169b 100644 --- a/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Libhegel.java +++ b/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Libhegel.java @@ -97,7 +97,8 @@ static Libhegel load(Path library, Iterable backends) * (a {@code hegel.toml} in the working directory or an ancestor, {@code HEGEL_DEFAULT_PROFILE}, * and the shipped {@code development}/{@code ci}/{@code workload} profiles) with the {@code * HEGEL_TEST_CASES}, {@code HEGEL_DATABASE}, {@code HEGEL_STATISTICS}, {@code HEGEL_SEED}, - * {@code HEGEL_DERANDOMIZE} and {@code HEGEL_PRINT_BLOB} environment variables applied over it. + * {@code HEGEL_DERANDOMIZE}, {@code HEGEL_PRINT_BLOB} and {@code HEGEL_NONDETERMINISM_STRICTNESS} + * environment variables applied over it. * Returns {@link Abi#E_INVALID_ARG} (with the message in {@link #lastErrorMessage()}) when a * {@code hegel.toml} or one of those variables is malformed; {@code out[0]} receives the handle * on {@link Abi#OK} and {@code 0} otherwise. @@ -112,6 +113,13 @@ static Libhegel load(Path library, Iterable backends) /** {@code hegel_settings_get_print_blob}: whether the resolved settings print reproduce blobs. */ boolean settingsGetPrintBlob(long s); + /** + * {@code hegel_settings_get_nondeterminism_strictness}: the resolved reaction to a + * nondeterministic test ({@link Abi#NONDETERMINISM_QUIET}, {@link Abi#NONDETERMINISM_WARN} or + * {@link Abi#NONDETERMINISM_ERROR}). + */ + int settingsGetNondeterminismStrictness(long s); + void settingsBackend(long s, int backend); void settingsTestCases(long s, long n); @@ -138,6 +146,13 @@ static Libhegel load(Path library, Iterable backends) /** {@code hegel_settings_set_print_blob}: whether to print a reproduce blob per failure. */ void settingsPrintBlob(long s, boolean yes); + /** + * {@code hegel_settings_set_nondeterminism_strictness}: how the run reacts when it detects a + * test whose structure or outcome changes when the same choices are replayed — one of {@link + * Abi#NONDETERMINISM_QUIET}, {@link Abi#NONDETERMINISM_WARN} or {@link Abi#NONDETERMINISM_ERROR}. + */ + void settingsNondeterminismStrictness(long s, int strictness); + // Run lifecycle. /** @@ -147,6 +162,15 @@ static Libhegel load(Path library, Iterable backends) */ long runStart(long settings, Consumer output); + /** + * {@code hegel_run_start_blob}: start a run that replays a reproduce blob until a replay fails + * (under the engine's bounded budget) instead of exploring. Driven exactly like a run from + * {@link #runStart}: a reproducing replay is the run's failure, a run with no failures means + * the blob is stale, and an undecodable blob surfaces as the run's error from {@link + * #runResultError}. {@code output} has the same contract as in {@link #runStart}. + */ + long runStartBlob(long settings, String blob, Consumer output); + /** The next test-case handle, or {@code 0} once the run is finished. */ long nextTestCase(long run); @@ -158,15 +182,24 @@ static Libhegel load(Path library, Iterable backends) void runFree(long run); /** - * Replay a base64 reproduce blob as a standalone test case. Returns the raw rc ({@link - * Abi#E_INVALID_ARG} for a corrupt or incompatible blob); on OK, {@code out[0]} receives the - * caller-owned handle. {@code output} has the same contract as in {@link #runStart} but need not - * outlive the call. + * Replay a base64 reproduce blob as a standalone test case, in a single attempt. Returns the raw + * rc ({@link Abi#E_INVALID_ARG} for a corrupt or incompatible blob); on OK, {@code out[0]} + * receives the caller-owned handle. {@code output} has the same contract as in {@link #runStart} + * but need not outlive the call. A nondeterministic blob may need several attempts to fail, so + * reproduce-failure features should prefer {@link #runStartBlob}. */ int testCaseFromBlob(long settings, String blob, Consumer output, long[] out); void testCaseFree(long tc); + /** + * {@code hegel_test_case_should_capture}: whether the engine stamped this case for capture — the + * caller should keep the case's output and, if it fails, its exception, keyed by the failure's + * origin, because a stamped failing execution is the material for that origin's failure report. + * Read once at case start. + */ + boolean testCaseShouldCapture(long tc); + // Per-test-case draws. Each returns the raw rc. int generateBoolean(long tc, double p, boolean[] out); @@ -303,6 +336,13 @@ int newStateMachine( /** The origin string the shrinker grouped the {@code index}-th distinct failure under. */ String failureOrigin(long result, long index); + /** + * {@code hegel_failure_caveat}: the {@code index}-th distinct failure's confirmation caveat — its + * standing under the run's nondeterministic handling, quoting the run's replay evidence — or + * {@code null} for a deterministic failure. + */ + String failureCaveat(long result, long index); + // Diagnostics. String lastErrorMessage(); diff --git a/hegel/src/main/java/dev/hegel/RealLibhegel.java b/hegel/src/main/java/dev/hegel/RealLibhegel.java index 34c9722..c965c9f 100644 --- a/hegel/src/main/java/dev/hegel/RealLibhegel.java +++ b/hegel/src/main/java/dev/hegel/RealLibhegel.java @@ -112,13 +112,17 @@ private static MethodHandle findEmitLine() { private final MethodHandle settingsSetPrintBlob; private final MethodHandle settingsGetTestCases; private final MethodHandle settingsGetPrintBlob; + private final MethodHandle settingsSetNondeterminismStrictness; + private final MethodHandle settingsGetNondeterminismStrictness; private final MethodHandle runStart; + private final MethodHandle runStartBlob; private final MethodHandle nextTestCase; private final MethodHandle runResult; private final MethodHandle runResultFree; private final MethodHandle runFree; private final MethodHandle testCaseFromBlob; private final MethodHandle testCaseFree; + private final MethodHandle testCaseShouldCapture; private final MethodHandle generateBoolean; private final MethodHandle generateInteger; private final MethodHandle generateFloat; @@ -161,6 +165,7 @@ private static MethodHandle findEmitLine() { private final MethodHandle failureFree; private final MethodHandle failureReproductionBlob; private final MethodHandle failureOrigin; + private final MethodHandle failureCaveat; private final MethodHandle version; RealLibhegel(Path libraryPath) { @@ -244,11 +249,26 @@ private static MethodHandle findEmitLine() { lookup, "hegel_settings_get_print_blob", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); + this.settingsSetNondeterminismStrictness = h( + linker, + lookup, + "hegel_settings_set_nondeterminism_strictness", + FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, JAVA_INT)); + this.settingsGetNondeterminismStrictness = h( + linker, + lookup, + "hegel_settings_get_nondeterminism_strictness", + FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.runStart = h( linker, lookup, "hegel_run_start", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS)); + this.runStartBlob = h( + linker, + lookup, + "hegel_run_start_blob", + FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS)); this.nextTestCase = h(linker, lookup, "hegel_next_test_case", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.runResult = @@ -263,6 +283,11 @@ private static MethodHandle findEmitLine() { FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS)); this.testCaseFree = h(linker, lookup, "hegel_test_case_free", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS)); + this.testCaseShouldCapture = h( + linker, + lookup, + "hegel_test_case_should_capture", + FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.generateBoolean = h( linker, lookup, @@ -443,6 +468,8 @@ private static MethodHandle findEmitLine() { FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.failureOrigin = h(linker, lookup, "hegel_failure_origin", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); + this.failureCaveat = + h(linker, lookup, "hegel_failure_caveat", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.version = h(linker, lookup, "hegel_version", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS)); this.context = ThreadLocal.withInitial(() -> (MemorySegment) invoke(contextNew)); } @@ -606,6 +633,20 @@ public boolean settingsGetPrintBlob(long s) { return out.get(JAVA_BOOLEAN, 0); } + @Override + public void settingsNondeterminismStrictness(long s, int strictness) { + check( + "hegel_settings_set_nondeterminism_strictness", + rc(settingsSetNondeterminismStrictness, segment(s), strictness)); + } + + @Override + public int settingsGetNondeterminismStrictness(long s) { + MemorySegment out = Arena.ofAuto().allocate(JAVA_INT); + check("hegel_settings_get_nondeterminism_strictness", rc(settingsGetNondeterminismStrictness, segment(s), out)); + return out.get(JAVA_INT, 0); + } + // --- run lifecycle --- @Override @@ -624,6 +665,24 @@ public long runStart(long settings, Consumer output) { return run; } + @Override + public long runStartBlob(long settings, String blob, Consumer output) { + Arena arena = Arena.ofConfined(); + MemorySegment callback = output == null ? MemorySegment.NULL : upcallStub(arena, output); + MemorySegment out = arena.allocate(ADDRESS); + int code = rc(runStartBlob, segment(settings), cstr(arena, blob), callback, MemorySegment.NULL, out); + if (code != Abi.OK) { + arena.close(); + throw new HegelException("hegel_run_start_blob failed (rc=" + + code + + "): " + + java.util.Objects.toString(lastErrorMessage(), "")); + } + long run = out.get(ADDRESS, 0).address(); + runArenas.put(run, arena); + return run; + } + @Override public long nextTestCase(long run) { MemorySegment out = Arena.ofAuto().allocate(ADDRESS); @@ -670,6 +729,13 @@ public void testCaseFree(long tc) { check("hegel_test_case_free", rc(testCaseFree, segment(tc))); } + @Override + public boolean testCaseShouldCapture(long tc) { + MemorySegment out = Arena.ofAuto().allocate(JAVA_BOOLEAN); + check("hegel_test_case_should_capture", rc(testCaseShouldCapture, segment(tc), out)); + return out.get(JAVA_BOOLEAN, 0); + } + // --- draws --- @Override @@ -1178,6 +1244,11 @@ public String failureOrigin(long result, long index) { return readFailureString(result, index, failureOrigin, "hegel_failure_origin"); } + @Override + public String failureCaveat(long result, long index) { + return readFailureString(result, index, failureCaveat, "hegel_failure_caveat"); + } + /** Fetch the {@code index}-th failure, read one of its strings with {@code reader}, free it. */ private String readFailureString(long result, long index, MethodHandle reader, String op) { MemorySegment failureOut = Arena.ofAuto().allocate(ADDRESS); diff --git a/hegel/src/test/java/dev/hegel/RealLibhegelTest.java b/hegel/src/test/java/dev/hegel/RealLibhegelTest.java index f6f4bc6..5a07383 100644 --- a/hegel/src/test/java/dev/hegel/RealLibhegelTest.java +++ b/hegel/src/test/java/dev/hegel/RealLibhegelTest.java @@ -3,6 +3,7 @@ import static org.junit.jupiter.api.Assertions.assertArrayEquals; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotEquals; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertThrows; @@ -80,6 +81,8 @@ void infrastructureCallsReportNullHandles() { // diagnostic rather than undefined behaviour. assertThrows(HegelException.class, () -> lib.runResultStatus(0)); assertThrows(HegelException.class, () -> lib.runStart(0, null)); + assertThrows(HegelException.class, () -> lib.runStartBlob(0, "blob", null)); + assertThrows(HegelException.class, () -> lib.testCaseShouldCapture(0)); } @Test @@ -193,6 +196,10 @@ void settingsGettersReadTheHandleBack() { assertFalse(lib.settingsGetPrintBlob(s)); lib.settingsPrintBlob(s, true); assertTrue(lib.settingsGetPrintBlob(s)); + lib.settingsNondeterminismStrictness(s, Abi.NONDETERMINISM_ERROR); + assertEquals(Abi.NONDETERMINISM_ERROR, lib.settingsGetNondeterminismStrictness(s)); + lib.settingsNondeterminismStrictness(s, Abi.NONDETERMINISM_WARN); + assertEquals(Abi.NONDETERMINISM_WARN, lib.settingsGetNondeterminismStrictness(s)); lib.settingsFree(s); } @@ -212,6 +219,23 @@ void undecodableBlobWithDefaultOutputIsRejected() { lib.settingsFree(s); } + @Test + void blobRunWithDefaultOutputStartsAndReportsTheBadBlob() { + RealLibhegel lib = real(); + long s = newSettings(lib); + // A null output callback leaves the run's output on stderr; the garbage blob surfaces as + // the run's error once it is pumped dry. + long run = lib.runStartBlob(s, "not-a-blob!!!", null); + assertNotEquals(0, run); + assertEquals(0, lib.nextTestCase(run)); + long result = lib.runResult(run); + assertEquals(Abi.RUN_STATUS_ERROR, lib.runResultStatus(result)); + assertNotNull(lib.runResultError(result)); + lib.runResultFree(result); + lib.runFree(run); + lib.settingsFree(s); + } + @Test void regexGeneratorAcceptsATextAlphabet() { RealLibhegel lib = real(); diff --git a/pom.xml b/pom.xml index ae0a165..dad1f7d 100644 --- a/pom.xml +++ b/pom.xml @@ -56,7 +56,7 @@ - 0.43.4 + 0.44.0 false diff --git a/shared/src/main/java/dev/hegel/Failure.java b/shared/src/main/java/dev/hegel/Failure.java index 9324c49..2b60326 100644 --- a/shared/src/main/java/dev/hegel/Failure.java +++ b/shared/src/main/java/dev/hegel/Failure.java @@ -8,24 +8,34 @@ import java.util.Optional; /** - * One distinct counterexample of a failed run, as observed on its final replay. + * One distinct counterexample of a failed run, as observed on the freshest failing execution the + * engine stamped for capture. * - *

After the engine shrinks a failure it hands back a reproduce blob; the runner replays that blob - * once more with reporting on, capturing the exception the body threw and every top-level draw and - * note along the way. That capture is what this class carries. A replay that no longer fails is - * {@linkplain #flaky() flaky}: the test's outcome depends on something other than its generated - * data. + *

The engine runs every failure it is about to report one final time, stamped for capture ({@link + * TestCase#isFinal()}); the runner keeps that execution's exception and every top-level draw and + * note, and that capture is what this class carries. A {@linkplain #nondeterministic() + * nondeterministic} failure — one whose test does not fail every time the same choices are + * replayed — additionally carries the engine's {@linkplain #caveat() caveat} quoting how reliably it + * reproduced, and has a reproduce blob only if the engine confirmed it. */ public final class Failure { private final String origin; private final String reproduceBlob; + private final String caveat; private final Throwable exception; private final Map draws; private final List notes; - Failure(String origin, String reproduceBlob, Throwable exception, Map draws, List notes) { + Failure( + String origin, + String reproduceBlob, + String caveat, + Throwable exception, + Map draws, + List notes) { this.origin = origin; this.reproduceBlob = reproduceBlob; + this.caveat = caveat; this.exception = exception; this.draws = Collections.unmodifiableMap(new LinkedHashMap<>(draws)); this.notes = Collections.unmodifiableList(new ArrayList<>(notes)); @@ -43,41 +53,58 @@ public String origin() { } /** - * The base64 blob that replays this counterexample exactly, via {@link + * The base64 blob that replays this counterexample, via {@link * Settings#reproduceFailure(String)}. Only guaranteed to reproduce under the Hegel version that * produced it. * - * @return the reproduce blob + * @return the reproduce blob, or empty when the engine produced none: for an unconfirmed + * {@linkplain #nondeterministic() nondeterministic} failure, and for a failure reproduced + * from a {@link Settings#reproduceFailure(String)} blob (the caller already holds it) */ - public String reproduceBlob() { - return reproduceBlob; + public Optional reproduceBlob() { + return Optional.ofNullable(reproduceBlob); } /** - * The exception the test body threw on the final replay. + * The engine's confirmation caveat for a nondeterministic failure: its standing under the run's + * nondeterministic handling, quoting the run's own replay evidence (for example {@code + * nondeterministic failure, confirmed: failed 7 of 20 replays at confirmation and 1 of 2 at + * report time}). Print it alongside the failure so the reader sees how reliably it reproduced. * - * @return the exception, or empty if the replay unexpectedly passed (see {@link #flaky()}) + * @return the caveat, or empty for a deterministic failure */ - public Optional exception() { - return Optional.ofNullable(exception); + public Optional caveat() { + return Optional.ofNullable(caveat); } /** - * Whether the final replay failed to reproduce the failure. The engine only reports a - * counterexample after re-running it, so a passing replay means the body's outcome depends on - * state outside its generated data (globals, time, an external RNG). + * Whether the test's outcome depended on something other than its generated data: the same + * choices did not fail every time the engine replayed them (hidden global state, time, an + * external RNG, thread scheduling). The engine confirms such a failure by repeated replay + * before shrinking it; {@link #caveat()} says how that went. * - * @return {@code true} if the replay passed + * @return {@code true} if the engine handled this failure as nondeterministic */ - public boolean flaky() { - return exception == null; + public boolean nondeterministic() { + return caveat != null; } /** - * Every top-level draw of the final replay, in draw order, keyed by the label passed to {@link - * TestCase#draw(Generator, String)} — numbered from its second use in the case ({@code x}, - * {@code x_2}, ...) so repeated draws are all kept — or {@code draw_N} for the N-th unlabelled - * draw. + * The exception the test body threw on the captured execution. + * + * @return the exception + */ + public Throwable exception() { + return exception; + } + + /** + * Every top-level draw of the captured execution, in draw order, keyed by the label passed to + * {@link TestCase#draw(Generator, String)} — numbered from its second use in the case ({@code + * x}, {@code x_2}, ...) so repeated draws are all kept — or {@code draw_N} for the N-th + * unlabelled draw. Empty when the engine reported the failure without a stamped failing + * execution to capture — an unconfirmed nondeterministic failure that never failed again after + * its discovery. * * @return an unmodifiable, insertion-ordered map of label to generated value */ @@ -86,7 +113,7 @@ public Map draws() { } /** - * Every {@link TestCase#note(String) note} recorded during the final replay, in order. + * Every {@link TestCase#note(String) note} recorded during the captured execution, in order. * * @return an unmodifiable list of notes */ diff --git a/shared/src/main/java/dev/hegel/HegelTest.java b/shared/src/main/java/dev/hegel/HegelTest.java index ad873ab..7513f00 100644 --- a/shared/src/main/java/dev/hegel/HegelTest.java +++ b/shared/src/main/java/dev/hegel/HegelTest.java @@ -133,6 +133,15 @@ */ OptBoolean printBlob() default OptBoolean.DEFAULT; + /** + * How the run reacts when it detects a nondeterministic test. The default leaves it to the + * engine's settings profile and the {@code HEGEL_NONDETERMINISM_STRICTNESS} environment variable + * ({@link NondeterminismStrictness#QUIET} unless configured otherwise). + * + * @return the reaction to nondeterminism + */ + NondeterminismStrictness nondeterminismStrictness() default NondeterminismStrictness.DEFAULT; + /** * Replay a stored failure blob (printed by {@link #printBlob}) instead of running the property: * the test body is re-run against exactly the choices the blob encodes, bypassing generation diff --git a/shared/src/main/java/dev/hegel/HegelTestExtension.java b/shared/src/main/java/dev/hegel/HegelTestExtension.java index 664e347..9b5bd56 100644 --- a/shared/src/main/java/dev/hegel/HegelTestExtension.java +++ b/shared/src/main/java/dev/hegel/HegelTestExtension.java @@ -85,6 +85,7 @@ static Settings settingsFrom(HegelTest ann, String methodName) { Settings s = new Settings() .verbosity(ann.verbosity()) .backend(ann.backend()) + .nondeterminismStrictness(ann.nondeterminismStrictness()) .reportMultipleFailures(ann.reportMultipleFailures()) .suppressHealthCheck(ann.suppressHealthCheck()) .name(name); diff --git a/shared/src/main/java/dev/hegel/NondeterminismStrictness.java b/shared/src/main/java/dev/hegel/NondeterminismStrictness.java new file mode 100644 index 0000000..0a8723a --- /dev/null +++ b/shared/src/main/java/dev/hegel/NondeterminismStrictness.java @@ -0,0 +1,45 @@ +package dev.hegel; + +import dev.hegel.lowlevel.Abi; + +/** + * How a run reacts when it detects a nondeterministic test: one whose structure or outcome changes + * when the same generated choices are replayed, because it depends on something other than its + * generated data (hidden global state, time, an outside service, thread scheduling). + * + *

Under {@link #QUIET} and {@link #WARN} the engine switches to nondeterministic handling: a + * failure is confirmed by repeated replay before it is shrunk or persisted, and reported with a + * {@linkplain Failure#caveat() caveat} quoting the run's replay evidence. An unconfirmed failure + * still fails the run, with a caveat instead of a reproduce blob. {@link #ERROR} aborts the run + * with a flaky-test error instead, for suites that use determinism as a lint. + */ +public enum NondeterminismStrictness { + /** + * Leave the choice to the engine's settings profile and the {@code HEGEL_NONDETERMINISM_STRICTNESS} + * environment variable ({@code quiet} unless configured otherwise). + */ + DEFAULT(null), + /** Switch to nondeterministic handling silently. The engine's default. */ + QUIET(Abi.NONDETERMINISM_QUIET), + /** Switch to nondeterministic handling, printing a one-line notice once per run. */ + WARN(Abi.NONDETERMINISM_WARN), + /** Abort the run with a flaky-test error ({@link HegelException}). */ + ERROR(Abi.NONDETERMINISM_ERROR); + + /** The {@code hegel_nondeterminism_strictness_t} value to send, or {@code null} to leave the profile's choice. */ + final Integer code; + + NondeterminismStrictness(Integer code) { + this.code = code; + } + + /** The strictness the engine reports back for {@code code}. */ + static NondeterminismStrictness fromCode(int code) { + for (NondeterminismStrictness s : values()) { + if (s.code != null && s.code == code) { + return s; + } + } + throw new HegelException("unknown hegel_nondeterminism_strictness_t value " + code); + } +} diff --git a/shared/src/main/java/dev/hegel/PrintingReporter.java b/shared/src/main/java/dev/hegel/PrintingReporter.java index 3d4cda7..53f1a70 100644 --- a/shared/src/main/java/dev/hegel/PrintingReporter.java +++ b/shared/src/main/java/dev/hegel/PrintingReporter.java @@ -59,10 +59,15 @@ public void failuresFound(int count) { @Override public void failure(Failure failure) { - if (printBlob && !failure.flaky()) { - out.println(); - out.println("To reproduce this failure, replay it with:"); - out.println(" @HegelTest(reproduceFailure = \"" + failure.reproduceBlob() + "\")"); + if (!quiet) { + failure.caveat().ifPresent(caveat -> out.println("note: " + caveat)); + } + if (printBlob) { + failure.reproduceBlob().ifPresent(blob -> { + out.println(); + out.println("To reproduce this failure, replay it with:"); + out.println(" @HegelTest(reproduceFailure = \"" + blob + "\")"); + }); } } } diff --git a/shared/src/main/java/dev/hegel/Reporter.java b/shared/src/main/java/dev/hegel/Reporter.java index b1e0793..d542c7d 100644 --- a/shared/src/main/java/dev/hegel/Reporter.java +++ b/shared/src/main/java/dev/hegel/Reporter.java @@ -15,13 +15,14 @@ * *

Every method has a no-op default, so an implementation overrides only what it needs. Callbacks * are invoked on the thread that drives the run, in this order: {@link #runStarted}, then for each - * generated case {@link #caseStarted} / {@link #caseFinished} (with {@link #engineOutput} lines - * interleaved as the engine emits them, and the case's {@link #draw}s and {@link #note}s in between - * when the run is {@linkplain Verbosity#VERBOSE verbose}); on a failed run {@link #failuresFound}, then for each - * distinct counterexample a final replay ({@link #caseStarted} with {@code finalReplay = true}, its - * {@link #draw}s and {@link #note}s, {@link #caseFinished}) followed by {@link #failure}; and - * finally {@link #runFinished}. A reporter is per run: two concurrent runs never share one unless - * the caller passes the same instance to both. + * case the engine runs {@link #caseStarted} / {@link #caseFinished} (with {@link #engineOutput} + * lines interleaved as the engine emits them, and the case's {@link #draw}s and {@link #note}s in + * between when the run is {@linkplain Verbosity#VERBOSE verbose}); on a failed run {@link + * #failuresFound}, then for each distinct counterexample the replay of its captured report ({@link + * #caseStarted} with {@code finalReplay = true}, the {@link #draw}s and {@link #note}s recorded on + * the failing execution the engine stamped for capture, {@link #caseFinished}) followed by {@link + * #failure}; and finally {@link #runFinished}. A reporter is per run: two concurrent runs never + * share one unless the caller passes the same instance to both. */ public interface Reporter { /** @@ -40,17 +41,19 @@ default void runStarted(Settings settings) {} default void engineOutput(String line) {} /** - * The test body is about to run against a case. + * The test body is about to run against a case ({@code finalReplay = false}), or the runner is + * about to replay the captured draws and notes of a counterexample it reports ({@code + * finalReplay = true}; no body runs). * - * @param finalReplay {@code true} for the replay of a minimal counterexample (or of a {@link - * Settings#reproduceFailure} blob), {@code false} during generation and shrinking + * @param finalReplay {@code true} for the replay of a reported counterexample's capture, {@code + * false} for a live case */ default void caseStarted(boolean finalReplay) {} /** - * A top-level {@link TestCase#draw(Generator, String) draw} completed. Called on final replays, - * and on every case when the run's {@link Verbosity} is {@code VERBOSE} or higher; never for - * draws nested inside another generator. + * A top-level {@link TestCase#draw(Generator, String) draw} completed. Called when a reported + * counterexample's capture is replayed, and live on every case when the run's {@link Verbosity} + * is {@code VERBOSE} or higher; never for draws nested inside another generator. * * @param label the label passed to {@code draw} (numbered from its second use in a case: {@code * x}, {@code x_2}, ...), or {@code draw_N} for the N-th unlabelled draw @@ -69,7 +72,9 @@ default void draw(String label, Object value, boolean finalReplay) {} default void note(String message, boolean finalReplay) {} /** - * The test body finished against a case and the outcome was reported to the engine. + * The test body finished against a case and the outcome was reported to the engine, or the + * replay of a reported counterexample's capture is complete (always {@link + * CaseOutcome#INTERESTING}). * * @param outcome how the case concluded * @param finalReplay as in {@link #caseStarted} @@ -77,16 +82,17 @@ default void note(String message, boolean finalReplay) {} default void caseFinished(CaseOutcome outcome, boolean finalReplay) {} /** - * The run's verdict is FAILED. Called once, before the counterexamples are replayed. + * The run's verdict is FAILED. Called once, before the counterexamples are reported. * - * @param count the number of distinct failures (by origin) that will be replayed + * @param count the number of distinct failures (by origin) that will be reported */ default void failuresFound(int count) {} /** - * A counterexample's final replay finished. + * A counterexample's captured report was replayed. * - * @param failure the failure, carrying the replay's exception, draws, notes, and blob + * @param failure the failure, carrying the captured exception, draws and notes, the engine's + * caveat for a nondeterministic failure, and the reproduce blob if there is one */ default void failure(Failure failure) {} @@ -99,10 +105,11 @@ default void runFinished(RunReport report) {} /** * The classic printed report: engine output, each reported draw as {@code label = value;}, - * notes, a header when several distinct failures are reported, and a copy-pasteable reproducer - * when {@link Settings#printBlob(boolean)} is on. Honours {@link Settings#verbosity}: {@link - * Verbosity#QUIET} prints no draws or notes at all, {@link Verbosity#VERBOSE} prints them for - * every case. + * notes, a header when several distinct failures are reported, a {@code note:} line with the + * engine's caveat for a nondeterministic failure, and a copy-pasteable reproducer when {@link + * Settings#printBlob(boolean)} is on and the failure has a blob. Honours {@link + * Settings#verbosity}: {@link Verbosity#QUIET} prints no draws, notes or caveats at all, {@link + * Verbosity#VERBOSE} prints draws and notes for every case. * * @param out where to print * @return a printing reporter diff --git a/shared/src/main/java/dev/hegel/RunReport.java b/shared/src/main/java/dev/hegel/RunReport.java index f3d04d4..8f53985 100644 --- a/shared/src/main/java/dev/hegel/RunReport.java +++ b/shared/src/main/java/dev/hegel/RunReport.java @@ -85,8 +85,7 @@ public List failures() { /** * Turn a non-passing report into the exception {@link Hegel#test} throws: nothing for a passed * run; {@link HealthCheckFailure} or {@link HegelException} (with the engine's message) for an - * errored run; for a failed run, a {@link HegelException} if any replay was {@linkplain - * Failure#flaky() flaky}, otherwise the single failure's own exception rethrown as-is, or an + * errored run; for a failed run, the single failure's own exception rethrown as-is, or an * {@link AssertionError} aggregating several distinct failures (carrying the originals as * suppressed exceptions). */ @@ -100,22 +99,17 @@ public void throwIfFailed() { } throw new HegelException(error); } - for (Failure failure : failures) { - if (failure.flaky()) { - throw new HegelException(Runner.FLAKY_DIAGNOSTIC); - } - } if (failures.size() == 1) { - throw unchecked(failures.get(0).exception().get()); + throw unchecked(failures.get(0).exception()); } StringBuilder sb = new StringBuilder(); sb.append("Hegel found ").append(failures.size()).append(" distinct failing examples:"); for (Failure failure : failures) { - sb.append("\n\n").append(describe(failure.exception().get())); + sb.append("\n\n").append(describe(failure.exception())); } AssertionError aggregate = new AssertionError(sb.toString()); for (Failure failure : failures) { - aggregate.addSuppressed(failure.exception().get()); + aggregate.addSuppressed(failure.exception()); } throw aggregate; } diff --git a/shared/src/main/java/dev/hegel/Runner.java b/shared/src/main/java/dev/hegel/Runner.java index 3dbd7ca..4dba0f8 100644 --- a/shared/src/main/java/dev/hegel/Runner.java +++ b/shared/src/main/java/dev/hegel/Runner.java @@ -4,19 +4,26 @@ import dev.hegel.lowlevel.Libhegel; import dev.hegel.lowlevel.LibhegelException; import java.util.ArrayList; +import java.util.HashMap; import java.util.List; +import java.util.Map; import java.util.function.Consumer; /** - * Drives a single property test: builds the settings handle, pumps the engine's exploration loop, - * and turns the aggregated result into a {@link RunReport}. + * Drives a single property test: builds the settings handle, pumps the engine's run loop, and turns + * the aggregated result into a {@link RunReport}. * - *

The engine only explores — generation and shrinking — so every pumped case is non-final. The - * client owns the final replays: once the loop drains, each discovered counterexample's reproduce - * blob is read off the run result and replayed via {@code hegel_test_case_from_blob} with reporting - * enabled, which surfaces the minimal example's draws and re-raises the test body's own exception. - * A counterexample whose replay does not fail again is a flaky test. Run-level errors (a failed - * health check, nondeterminism, an engine panic) surface with the engine's own message. + *

The engine owns the whole run — generation, shrinking, the replays that confirm a failure, and + * the final replay of every failure it is about to report — so the loop only pumps cases and keeps + * the report material as they run. The engine stamps the executions a failure report can be built + * from ({@code hegel_test_case_should_capture}); a stamped case records its draws and notes, and + * every interesting case's exception is kept per origin, a stamped capture winning over an unstamped + * one and the newest winning at equal rank. Once the loop drains, each reported failure is built + * from its origin's capture: the draws and notes are replayed to the reporter, the exception is the + * body's own, and the engine's caveat and reproduce blob are attached. A {@link + * Settings#reproduceFailure} run is the same loop over {@code hegel_run_start_blob}. Run-level errors + * (a failed health check, a nondeterminism abort under {@link NondeterminismStrictness#ERROR}, an + * engine panic) surface with the engine's own message. * *

Everything the run has to say goes through its {@link Reporter}; the runner itself never * prints. Binding and engine errors ({@link HegelException}) are thrown, not reported: they are bugs @@ -34,24 +41,31 @@ private Runner() {} "dev.hegel.", "org.junit.", "org.opentest4j.", "jdk.", "java.", "sun.", "com.sun." }; + /** Message for a {@link Settings#reproduceFailure} blob none of whose replays failed. */ + static final String STALE_BLOB = "reproduceFailure: the supplied failure blob did not reproduce a failure." + + " The failure may have been fixed — or, for a nondeterministic blob, it may not have" + + " recurred within the replay budget. Re-run to retry, or remove the blob once the failure" + + " is fixed."; + /** - * Message for a test whose outcome changed when re-run with the same generated data. After the - * engine shrinks and verifies a counterexample, the runner replays its blob one final time; if - * that replay does not fail, the test is non-deterministic. + * The body's outcome against one case: the case (holding its draws and notes when the engine + * stamped it for capture), the exception that made it interesting ({@code null} otherwise), and + * the origin the case was marked complete with. */ - static final String FLAKY_DIAGNOSTIC = "Flaky test detected: Your test produced different outcomes" - + " when run with the same generated data — it failed when it previously succeeded, or" - + " succeeded when it previously failed. This usually means your test depends on external" - + " state such as global variables, system time, or external random number generators."; - - /** The body's outcome against one case: the case (holding its draws and notes) and its failure. */ private static final class CaseRun { final TestCase testCase; final Throwable interesting; + final String origin; - CaseRun(TestCase testCase, Throwable interesting) { + CaseRun(TestCase testCase, Throwable interesting, String origin) { this.testCase = testCase; this.interesting = interesting; + this.origin = origin; + } + + /** A stamped capture outranks an unstamped one, whose only material is the exception. */ + int rank() { + return testCase.isFinal() ? 1 : 0; } } @@ -63,12 +77,18 @@ static RunReport run(Libhegel lib, Settings settings, Consumer body, R applySettings(lib, s, settings); // The engine resolved whatever the caller left unset (its profile, hegel.toml, the // HEGEL_* variables); the run and its reporters see the effective configuration. - Settings effective = settings.resolved(lib.settingsGetTestCases(s), lib.settingsGetPrintBlob(s)); + Settings effective = settings.resolved( + lib.settingsGetTestCases(s), + lib.settingsGetPrintBlob(s), + NondeterminismStrictness.fromCode(lib.settingsGetNondeterminismStrictness(s))); reporter.runStarted(effective); - if (effective.reproduceFailure != null) { - report = replayBlob(lib, s, effective, body, reporter, counts); - } else { - report = explore(lib, s, effective, body, reporter, counts); + long run = effective.reproduceFailure == null + ? lib.runStart(s, reporter::engineOutput) + : lib.runStartBlob(s, effective.reproduceFailure, reporter::engineOutput); + try { + report = drive(lib, run, effective, body, reporter, counts); + } finally { + lib.runFree(run); } } finally { lib.settingsFree(s); @@ -96,149 +116,115 @@ private static long newSettings(Libhegel lib) { return out[0]; } - /** Pump the engine's exploration loop to completion and translate its verdict. */ - private static RunReport explore( + /** + * Pump the run to completion, keeping every interesting case's capture per origin, and + * translate its verdict. + */ + private static RunReport drive( Libhegel lib, - long s, + long run, Settings settings, Consumer body, Reporter reporter, RunStatistics.Counter counts) { - long run = lib.runStart(s, reporter::engineOutput); - try { - while (true) { - long tc = lib.nextTestCase(run); - if (isNull(tc)) { - break; - } - driveOneCase(lib, tc, false, body, settings, reporter, counts); + Map captures = new HashMap<>(); + while (true) { + long tc = lib.nextTestCase(run); + if (isNull(tc)) { + break; } - long result = lib.runResult(run); - try { - return finish(lib, s, result, settings, body, reporter, counts); - } finally { - lib.runResultFree(result); + CaseRun caseRun = driveOneCase(lib, tc, body, settings, reporter, counts); + if (caseRun.interesting != null) { + CaseRun stored = captures.get(caseRun.origin); + if (stored == null || caseRun.rank() >= stored.rank()) { + captures.put(caseRun.origin, caseRun); + } } + } + long result = lib.runResult(run); + try { + return finish(lib, result, settings, reporter, counts, captures); } finally { - lib.runFree(run); + lib.runResultFree(result); } } /** Translate a drained run's result into a report. */ private static RunReport finish( Libhegel lib, - long s, long result, Settings settings, - Consumer body, Reporter reporter, - RunStatistics.Counter counts) { + RunStatistics.Counter counts, + Map captures) { + boolean replayingBlob = settings.reproduceFailure != null; switch (lib.runResultStatus(result)) { case Abi.RUN_STATUS_PASSED: + if (replayingBlob) { + throw new HegelException(STALE_BLOB); + } return new RunReport(RunStatus.PASSED, counts.snapshot(), null, List.of()); case Abi.RUN_STATUS_ERROR: // The run produced no verdict on the property: a failed health check, a - // nondeterminism mismatch, or an engine panic. - return new RunReport( - RunStatus.ERROR, counts.snapshot(), nullToEmpty(lib.runResultError(result)), List.of()); - case Abi.RUN_STATUS_FAILED_NONDETERMINISTIC: - // Only a state machine created with max_concurrency > 1 declares a run - // nondeterministic, and this binding drives every machine sequentially, so the - // engine should never report this: its failures carry no reproduce blob to replay. - throw new HegelException("internal error: the engine reported a failure on a nondeterministic run," - + " but this binding never declares a run nondeterministic"); + // nondeterminism abort under ERROR strictness, an engine panic — or, on a blob + // replay, a blob the engine could not decode. + String error = nullToEmpty(lib.runResultError(result)); + if (replayingBlob) { + throw new HegelException("reproduceFailure: the supplied blob is not valid: " + error); + } + return new RunReport(RunStatus.ERROR, counts.snapshot(), error, List.of()); default: - return replayFailures(lib, s, result, settings, body, reporter, counts); + return reportFailures(lib, result, reporter, counts, captures); } } - /** Replay every distinct counterexample's blob, capturing its draws, notes, and exception. */ - private static RunReport replayFailures( - Libhegel lib, - long s, - long result, - Settings settings, - Consumer body, - Reporter reporter, - RunStatistics.Counter counts) { + /** Build every distinct failure from its origin's capture and hand it to the reporter. */ + private static RunReport reportFailures( + Libhegel lib, long result, Reporter reporter, RunStatistics.Counter counts, Map captures) { long count = lib.runResultFailureCount(result); reporter.failuresFound((int) count); List failures = new ArrayList<>(); for (int i = 0; i < count; i++) { - String blob = lib.failureBlob(result, i); - if (blob == null) { - throw new HegelException("internal error: failure " + i + " carries no reproduce blob"); - } String origin = lib.failureOrigin(result, i); - long[] tcOut = new long[1]; - int rc = lib.testCaseFromBlob(s, blob, reporter::engineOutput, tcOut); - if (rc != Abi.OK) { + CaseRun capture = captures.remove(origin); + if (capture == null) { throw new HegelException( - "hegel_test_case_from_blob failed (rc=" + rc + "): " + nullToEmpty(lib.lastErrorMessage())); + "internal error: failure " + i + " (" + origin + ") has no captured failing execution"); } - CaseRun replay = driveOneCase(lib, tcOut[0], true, body, settings, reporter, counts); - Failure failure = - new Failure(origin, blob, replay.interesting, replay.testCase.draws(), replay.testCase.notes()); + reporter.caseStarted(true); + capture.testCase.replayTo(reporter); + reporter.caseFinished(CaseOutcome.INTERESTING, true); + Failure failure = new Failure( + origin, + lib.failureBlob(result, i), + lib.failureCaveat(result, i), + capture.interesting, + capture.testCase.draws(), + capture.testCase.notes()); reporter.failure(failure); failures.add(failure); } return new RunReport(RunStatus.FAILED, counts.snapshot(), null, failures); } - /** - * Replay a single stored blob ({@link Settings#reproduceFailure}), bypassing generation and - * shrinking: a reproduced failure is reported as the run's one failure, and a blob that no - * longer fails is a configuration error (the failure was fixed, or the blob is stale). - */ - private static RunReport replayBlob( - Libhegel lib, - long s, - Settings settings, - Consumer body, - Reporter reporter, - RunStatistics.Counter counts) { - String blob = settings.reproduceFailure; - long[] tcOut = new long[1]; - int rc = lib.testCaseFromBlob(s, blob, reporter::engineOutput, tcOut); - if (rc != Abi.OK) { - throw new HegelException("reproduceFailure: the supplied blob is not valid (rc=" - + rc - + "): " - + nullToEmpty(lib.lastErrorMessage())); - } - CaseRun replay = driveOneCase(lib, tcOut[0], true, body, settings, reporter, counts); - if (replay.interesting == null) { - throw new HegelException("reproduceFailure: the supplied failure blob no longer reproduces a" - + " failure. The failure may have been fixed, or the blob is stale."); - } - Failure failure = new Failure( - originOf(replay.interesting, settings.infrastructurePackages), - blob, - replay.interesting, - replay.testCase.draws(), - replay.testCase.notes()); - reporter.failure(failure); - return new RunReport(RunStatus.FAILED, counts.snapshot(), null, List.of(failure)); - } - /** * Run the body once against {@code tc}, report the outcome to the engine and the reporter, and * free the handle. The returned {@link CaseRun} carries the exception that made the case * interesting ({@code null} for any other outcome) and the case itself, whose draws and notes - * were recorded when {@code finalReplay} is set. + * were recorded if the engine stamped it for capture. */ private static CaseRun driveOneCase( Libhegel lib, long tc, - boolean finalReplay, Consumer body, Settings settings, Reporter reporter, RunStatistics.Counter counts) { try { boolean verbose = settings.verbosity.code >= Verbosity.VERBOSE.code; - TestCase testCase = new TestCase(new LiveDataSource(lib, tc), finalReplay, verbose, reporter); - reporter.caseStarted(finalReplay); + TestCase testCase = + new TestCase(new LiveDataSource(lib, tc), lib.testCaseShouldCapture(tc), verbose, reporter); + reporter.caseStarted(false); CaseOutcome outcome; String origin = null; Throwable interesting = null; @@ -263,8 +249,8 @@ private static CaseRun driveOneCase( "hegel_mark_complete failed (rc=" + rc + "): " + nullToEmpty(lib.lastErrorMessage())); } counts.record(outcome); - reporter.caseFinished(outcome, finalReplay); - return new CaseRun(testCase, interesting); + reporter.caseFinished(outcome, false); + return new CaseRun(testCase, interesting, origin); } finally { // The handle is caller-owned. On the error paths above the case may be incomplete; // the run still holds its own reference and completes it when freed. @@ -293,6 +279,9 @@ static void applySettings(Libhegel lib, long s, Settings st) { if (st.printBlob != null) { lib.settingsPrintBlob(s, st.printBlob); } + if (st.nondeterminismStrictness.code != null) { + lib.settingsNondeterminismStrictness(s, st.nondeterminismStrictness.code); + } if (st.backend.code != null) { lib.settingsBackend(s, st.backend.code); } diff --git a/shared/src/main/java/dev/hegel/Settings.java b/shared/src/main/java/dev/hegel/Settings.java index d8a141c..4c908e7 100644 --- a/shared/src/main/java/dev/hegel/Settings.java +++ b/shared/src/main/java/dev/hegel/Settings.java @@ -16,9 +16,9 @@ *

Whatever is not set here is resolved by the engine: from its settings profile — a {@code * hegel.toml} in the working directory or one of its parents (or the file named by {@code * HEGEL_CONFIG}), with {@code HEGEL_DEFAULT_PROFILE} selecting the profile in effect — and from the - * {@code HEGEL_TEST_CASES}, {@code HEGEL_DATABASE}, {@code HEGEL_SEED}, {@code HEGEL_DERANDOMIZE} and - * {@code HEGEL_PRINT_BLOB} environment variables, which win over the profile. Settings given here win - * over both. The shipped profiles run 100 test cases, and in CI (detected via {@code CI}/{@code + * {@code HEGEL_TEST_CASES}, {@code HEGEL_DATABASE}, {@code HEGEL_SEED}, {@code HEGEL_DERANDOMIZE}, + * {@code HEGEL_PRINT_BLOB} and {@code HEGEL_NONDETERMINISM_STRICTNESS} environment variables, which + * win over the profile. Settings given here win over both. The shipped profiles run 100 test cases, and in CI (detected via {@code CI}/{@code * GITHUB_ACTIONS}/... environment variables) the {@code ci} profile applies: runs are deterministic * ({@code derandomize}) and the example database is disabled. */ @@ -36,6 +36,7 @@ public final class Settings { // traces than an aggregated report — and that matters more in Java than elsewhere. final boolean reportMultipleFailures; final Boolean printBlob; // null = leave the engine's profile / HEGEL_PRINT_BLOB value + final NondeterminismStrictness nondeterminismStrictness; final String reproduceFailure; // null = run normally instead of replaying a blob final String name; final List infrastructurePackages; @@ -61,6 +62,7 @@ private Settings(Builder b) { this.backend = b.backend; this.reportMultipleFailures = b.reportMultipleFailures; this.printBlob = b.printBlob; + this.nondeterminismStrictness = b.nondeterminismStrictness; this.reproduceFailure = b.reproduceFailure; this.name = b.name; this.infrastructurePackages = b.infrastructurePackages; @@ -80,6 +82,7 @@ private Settings with(Consumer mutator) { b.backend = backend; b.reportMultipleFailures = reportMultipleFailures; b.printBlob = printBlob; + b.nondeterminismStrictness = nondeterminismStrictness; b.reproduceFailure = reproduceFailure; b.name = name; b.infrastructurePackages = infrastructurePackages; @@ -100,6 +103,7 @@ private static final class Builder { Backend backend = Backend.AUTO; boolean reportMultipleFailures = false; Boolean printBlob = null; + NondeterminismStrictness nondeterminismStrictness = NondeterminismStrictness.DEFAULT; String reproduceFailure = null; String name = null; List infrastructurePackages = List.of(); @@ -232,12 +236,27 @@ public Settings printBlob(boolean yes) { return with(b -> b.printBlob = yes); } + /** + * How the run reacts when it detects a nondeterministic test. Unset ({@link + * NondeterminismStrictness#DEFAULT}), the engine's profile or the {@code + * HEGEL_NONDETERMINISM_STRICTNESS} environment variable decides ({@link + * NondeterminismStrictness#QUIET} in the shipped profiles). + * + * @param strictness the reaction + * @return a new settings instance + */ + public Settings nondeterminismStrictness(NondeterminismStrictness strictness) { + return with(b -> b.nondeterminismStrictness = strictness); + } + /** * Replay a single stored failure instead of running the property: the blob (from {@link - * #printBlob(boolean)} output) is decoded and the test body re-run against exactly the choices - * it encodes, bypassing generation and shrinking. The run fails with the reproduced failure, or - * reports a stale blob if it no longer fails. A blob is only guaranteed to reproduce under the - * Hegel version that produced it. + * #printBlob(boolean)} output) is decoded and the test body re-run against the choices it + * encodes, bypassing generation and shrinking. The engine replays until a replay fails, under a + * bounded budget: a deterministic blob a few times, a nondeterministic one's recorded runs until + * one of them fails again. The run fails with the reproduced failure, or reports a stale blob if + * none of the replays fails. A blob is only guaranteed to reproduce under the Hegel version that + * produced it. * * @param blob the base64 reproduce blob * @return a new settings instance @@ -275,10 +294,11 @@ public Settings infrastructurePackages(String... prefixes) { * These settings with the values the engine resolved for what was left unset, so a run and its * reporters see the effective configuration. */ - Settings resolved(long testCases, boolean printBlob) { + Settings resolved(long testCases, boolean printBlob, NondeterminismStrictness strictness) { return with(b -> { b.testCases = testCases; b.printBlob = printBlob; + b.nondeterminismStrictness = strictness; }); } } diff --git a/shared/src/main/java/dev/hegel/TestCase.java b/shared/src/main/java/dev/hegel/TestCase.java index edff627..674c32d 100644 --- a/shared/src/main/java/dev/hegel/TestCase.java +++ b/shared/src/main/java/dev/hegel/TestCase.java @@ -10,6 +10,7 @@ import java.util.List; import java.util.Map; import java.util.UUID; +import java.util.function.Consumer; import java.util.function.Supplier; /** @@ -19,25 +20,30 @@ * {@link #draw(Generator)}, reject uninteresting inputs with {@link #assume(boolean)}, attach debug * context with {@link #note(String)}, and guide the search with {@link #target(double)}. * - *

On the replay of a minimal failing example ({@link #isFinal()}), each top-level {@code draw} - * and each note is handed to the run's {@link Reporter} (the default prints {@code x = 42;}) and - * recorded on the resulting {@link Failure}, so the counterexample is readable. Under {@link - * Verbosity#VERBOSE} or higher the reporter also sees every other case's draws and notes. A - * label drawn more than once in a case is numbered from its second use ({@code x}, {@code x_2}, - * {@code x_3}); unlabelled draws are {@code draw_1}, {@code draw_2}, ... + *

On an execution the engine stamped for capture ({@link #isFinal()}) — the final replay of a + * minimal failing example among them — each top-level {@code draw} and each note is recorded, and + * if the case fails and the engine reports that failure, they are handed to the run's {@link + * Reporter} (the default prints {@code x = 42;}) and carried by the resulting {@link Failure}, so + * the counterexample is readable. Under {@link Verbosity#VERBOSE} or higher the reporter sees every + * case's draws and notes live. A label drawn more than once in a case is numbered from its second + * use ({@code x}, {@code x_2}, {@code x_3}); unlabelled draws are {@code draw_1}, {@code draw_2}, + * ... * *

Frontends implementing their own composite generators enclose their draws in a labelled * {@link #span(long, Supplier) span} so the engine can shrink the structure they build. */ public final class TestCase { private final DataSource source; - private final boolean finalReplay; - /** Whether draws and notes reach the reporter: on a final replay, or on every case when verbose. */ + /** Whether the engine stamped this execution for capture: its draws and notes are recorded. */ + private final boolean captured; + /** Whether draws and notes reach the reporter live (a verbose run). */ private final boolean reporting; private final Reporter reporter; private final Map draws = new LinkedHashMap<>(); private final List notes = new ArrayList<>(); + /** The recorded draws and notes in report order, each as the reporter call that replays it. */ + private final List> events = new ArrayList<>(); /** Uses per draw name, for numbering repeats ({@code x}, {@code x_2}, ...; {@code draw_N}). */ private final Map nameUses = new HashMap<>(); /** Notes made while a top-level draw is in progress; flushed after that draw's line. */ @@ -45,27 +51,29 @@ public final class TestCase { private int drawDepth; - TestCase(DataSource source, boolean finalReplay, Reporter reporter) { - this(source, finalReplay, false, reporter); + TestCase(DataSource source, boolean captured, Reporter reporter) { + this(source, captured, false, reporter); } - TestCase(DataSource source, boolean finalReplay, boolean verbose, Reporter reporter) { + TestCase(DataSource source, boolean captured, boolean verbose, Reporter reporter) { this.source = source; - this.finalReplay = finalReplay; - this.reporting = finalReplay || verbose; + this.captured = captured; + this.reporting = verbose; this.reporter = reporter; } /** - * Whether this case is the final replay of a minimal counterexample (or of a {@link - * Settings#reproduceFailure(String)} blob), as opposed to one of the many cases the engine runs - * while generating and shrinking. Draws and notes are reported only on a final replay; a test - * body can use this to do its own expensive diagnostics only where they will be seen. + * Whether the engine stamped this execution as one a failure report can be built from — the + * final replay of a minimal counterexample (or of a {@link Settings#reproduceFailure(String)} + * blob), the replays that confirm a discovered failure — as opposed to one of the many cases it + * runs while generating and shrinking. Draws and notes are recorded only on a stamped + * execution, and reported only if it fails and the engine reports that failure; a test body can + * use this to do its own expensive diagnostics only where they may be seen. * - * @return {@code true} on a final replay + * @return {@code true} on an execution stamped for capture */ public boolean isFinal() { - return finalReplay; + return captured; } /** @@ -103,12 +111,15 @@ public T draw(Generator generator, String label) { } } if (top) { - if (reporting) { + if (captured || reporting) { String name = displayName(label); - if (finalReplay) { + if (captured) { draws.put(name, value); + events.add(r -> r.draw(name, value, true)); + } + if (reporting) { + reporter.draw(name, value, false); } - reporter.draw(name, value, finalReplay); } flushPendingNotes(); } @@ -137,10 +148,13 @@ private void flushPendingNotes() { } private void emitNote(String message) { - if (finalReplay) { + if (captured) { notes.add(message); + events.add(r -> r.note(message, true)); + } + if (reporting) { + reporter.note(message, false); } - reporter.note(message, finalReplay); } /** @@ -156,14 +170,15 @@ public void assume(boolean condition) { } /** - * Record a debug message, reported on the final replay of a failing case (and on every case - * under {@link Verbosity#VERBOSE}). A note made while a top-level draw is in progress — from - * inside a composite generator — is reported after that draw's value, never before it. + * Record a debug message, reported with the counterexample when this case's failure is + * reported (and live on every case under {@link Verbosity#VERBOSE}). A note made while a + * top-level draw is in progress — from inside a composite generator — is reported after that + * draw's value, never before it. * * @param message the message to record */ public void note(String message) { - if (!reporting) { + if (!(captured || reporting)) { return; } if (drawDepth > 0) { @@ -413,16 +428,23 @@ boolean isAborted() { return source.isAborted(); } - /** The top-level draws recorded on a final replay, in draw order. */ + /** The top-level draws recorded on a captured execution, in draw order. */ Map draws() { return draws; } - /** The notes recorded on a final replay, in order. */ + /** The notes recorded on a captured execution, in order. */ List notes() { return notes; } + /** Hand the recorded draws and notes to {@code reporter}, in report order, flagged as a replay. */ + void replayTo(Reporter reporter) { + for (Consumer event : events) { + event.accept(reporter); + } + } + static String repr(Object value) { if (value == null) { return "null"; diff --git a/shared/src/test/java/dev/hegel/CoverageTest.java b/shared/src/test/java/dev/hegel/CoverageTest.java index 342b344..ca9137d 100644 --- a/shared/src/test/java/dev/hegel/CoverageTest.java +++ b/shared/src/test/java/dev/hegel/CoverageTest.java @@ -63,6 +63,7 @@ void plain(TestCase tc) {} phases = {Phase.GENERATE}, suppressHealthCheck = {HealthCheck.TOO_SLOW}, backend = Backend.URANDOM, + nondeterminismStrictness = NondeterminismStrictness.ERROR, reportMultipleFailures = true, printBlob = OptBoolean.TRUE, reproduceFailure = "blob-xyz", @@ -106,6 +107,7 @@ void hegelTestHelpers() throws Exception { assertEquals(Database.Kind.UNSET, noSeed.database.kind); assertEquals(0, noSeed.suppressMask); assertEquals(Backend.AUTO, noSeed.backend); + assertEquals(NondeterminismStrictness.DEFAULT, noSeed.nondeterminismStrictness); assertFalse(noSeed.reportMultipleFailures); assertNull(noSeed.printBlob); assertNull(noSeed.testCases); @@ -120,6 +122,7 @@ void hegelTestHelpers() throws Exception { assertEquals(Integer.valueOf(Phase.GENERATE.bit), c.phasesMask); assertEquals(HealthCheck.TOO_SLOW.bit, c.suppressMask); assertEquals(Backend.URANDOM, c.backend); + assertEquals(NondeterminismStrictness.ERROR, c.nondeterminismStrictness); assertTrue(c.reportMultipleFailures); assertEquals(Boolean.TRUE, c.printBlob); assertEquals("blob-xyz", c.reproduceFailure); diff --git a/shared/src/test/java/dev/hegel/EnvironmentFixture.java b/shared/src/test/java/dev/hegel/EnvironmentFixture.java index adf0107..6666a96 100644 --- a/shared/src/test/java/dev/hegel/EnvironmentFixture.java +++ b/shared/src/test/java/dev/hegel/EnvironmentFixture.java @@ -11,6 +11,8 @@ *

  • {@code count}: {@code valid=N}, the number of cases a passing property ran under settings * that leave the budget to the engine; *
  • {@code count-explicit}: the same with an explicit {@code testCases(4)}; + *
  • {@code strictness}: {@code strictness=}, the nondeterminism strictness the engine + * resolved for settings that leave it unset; *
  • {@code fail}: the printing reporter's output for a property that always fails, followed by * {@code done}; *
  • {@code error=} when constructing the settings failed on a malformed variable. @@ -22,7 +24,7 @@ private EnvironmentFixture() {} /** * Entry point. * - * @param args the mode: {@code count}, {@code count-explicit} or {@code fail} + * @param args the mode: {@code count}, {@code count-explicit}, {@code strictness} or {@code fail} */ public static void main(String[] args) { try { @@ -33,6 +35,16 @@ public static void main(String[] args) { case "count-explicit": System.out.println("valid=" + count(new Settings().testCases(4))); break; + case "strictness": + Settings[] effective = new Settings[1]; + Hegel.run(tc -> {}, new Settings().testCases(1), new Reporter() { + @Override + public void runStarted(Settings settings) { + effective[0] = settings; + } + }); + System.out.println("strictness=" + effective[0].nondeterminismStrictness); + break; default: Hegel.run( tc -> { diff --git a/shared/src/test/java/dev/hegel/EnvironmentTest.java b/shared/src/test/java/dev/hegel/EnvironmentTest.java index f0c97fd..a0e2b07 100644 --- a/shared/src/test/java/dev/hegel/EnvironmentTest.java +++ b/shared/src/test/java/dev/hegel/EnvironmentTest.java @@ -66,6 +66,16 @@ void hegelTomlProfileSetsTheBudget() throws Exception { assertTrue(output.contains("valid=9"), output); } + @Test + void hegelNondeterminismStrictnessIsResolvedByTheEngine() throws Exception { + String unset = fixture("strictness", Map.of()); + assertTrue(unset.contains("strictness=QUIET"), unset); + String error = fixture("strictness", Map.of("HEGEL_NONDETERMINISM_STRICTNESS", "error")); + assertTrue(error.contains("strictness=ERROR"), error); + String malformed = fixture("strictness", Map.of("HEGEL_NONDETERMINISM_STRICTNESS", "loud")); + assertTrue(malformed.contains("error=") && malformed.contains("HEGEL_NONDETERMINISM_STRICTNESS"), malformed); + } + @Test void hegelPrintBlobDecidesWhetherReproducersArePrinted() throws Exception { // The shipped profiles print a reproducer for each failure; the variable turns that off. diff --git a/shared/src/test/java/dev/hegel/FakeLibhegel.java b/shared/src/test/java/dev/hegel/FakeLibhegel.java index 1e43c7d..1a78aeb 100644 --- a/shared/src/test/java/dev/hegel/FakeLibhegel.java +++ b/shared/src/test/java/dev/hegel/FakeLibhegel.java @@ -34,9 +34,18 @@ final class FakeLibhegel implements Libhegel { int runStatus = Abi.RUN_STATUS_PASSED; String runError = "run error"; final List failureBlobs = new ArrayList<>(); // one entry per distinct failure + final List failureCaveats = new ArrayList<>(); // parallel to failureBlobs; missing = null + // The origin of each distinct failure. Like the engine, the fake echoes back the origins marked + // interesting, distinct and in first-seen order; an explicit entry here overrides that. + final List failureOrigins = new ArrayList<>(); int fromBlobRc = Abi.OK; final List replayedBlobs = new ArrayList<>(); - Consumer output; // the callback runStart registered + String startedBlob; // what runStartBlob received; null = the run explored + Consumer output; // the callback runStart / runStartBlob registered + // The engine's capture stamp per served case: the i-th case is stamped when captureSequence + // has an i-th entry and it is true; past the end of the sequence every case is stamped, so + // the default stamps everything. + boolean[] captureSequence = {}; // Recorded outcomes and teardown. final List markedStatuses = new ArrayList<>(); @@ -149,7 +158,9 @@ final class FakeLibhegel implements Libhegel { int settingsNewRc = Abi.OK; long resolvedTestCases = 100; boolean resolvedPrintBlob = false; + int resolvedStrictness = Abi.NONDETERMINISM_QUIET; Boolean printBlob; // what the setter received; null = never set + Integer strictness; // what the setter received; null = never set @Override public int settingsNew(long[] out) { @@ -174,6 +185,16 @@ public boolean settingsGetPrintBlob(long s) { return printBlob == null ? resolvedPrintBlob : printBlob; } + @Override + public void settingsNondeterminismStrictness(long s, int strictness) { + this.strictness = strictness; + } + + @Override + public int settingsGetNondeterminismStrictness(long s) { + return strictness == null ? resolvedStrictness : strictness; + } + @Override public void settingsFree(long s) { settingsFreed = true; @@ -232,6 +253,16 @@ public long runStart(long settings, Consumer output) { return RUN; } + @Override + public long runStartBlob(long settings, String blob, Consumer output) { + if (runStartFails) { + throw new HegelException("hegel_run_start_blob failed: " + lastError); + } + startedBlob = blob; + this.output = output; + return RUN; + } + @Override public long nextTestCase(long run) { if (nextTestCaseFails) { @@ -244,6 +275,12 @@ public long nextTestCase(long run) { return TC; } + @Override + public boolean testCaseShouldCapture(long tc) { + int index = casesServed - 1; + return index >= captureSequence.length || captureSequence[index]; + } + @Override public long runResult(long run) { return RESULT; @@ -603,7 +640,21 @@ public String failureBlob(long result, long index) { @Override public String failureOrigin(long result, long index) { - return "fake-origin-" + index; + if (index < failureOrigins.size()) { + return failureOrigins.get((int) index); + } + List distinct = new ArrayList<>(); + for (String origin : markedOrigins) { + if (origin != null && !distinct.contains(origin)) { + distinct.add(origin); + } + } + return distinct.get((int) index); + } + + @Override + public String failureCaveat(long result, long index) { + return index < failureCaveats.size() ? failureCaveats.get((int) index) : null; } @Override diff --git a/shared/src/test/java/dev/hegel/LibraryApiTest.java b/shared/src/test/java/dev/hegel/LibraryApiTest.java index b9fae9f..8fa6441 100644 --- a/shared/src/test/java/dev/hegel/LibraryApiTest.java +++ b/shared/src/test/java/dev/hegel/LibraryApiTest.java @@ -3,11 +3,11 @@ import static dev.hegel.Generators.integers; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.concurrent.atomic.AtomicInteger; import org.junit.jupiter.api.Test; @@ -38,13 +38,15 @@ void runReportsTheShrunkCounterexample() { Failure f = report.failures().get(0); assertEquals(Map.of("x", 11), f.draws()); assertEquals(List.of("observed x=11"), f.notes()); - assertTrue(f.exception().get() instanceof AssertionError); - assertTrue(f.exception().get().getMessage().contains("x was too big: 11")); + assertTrue(f.exception() instanceof AssertionError); + assertTrue(f.exception().getMessage().contains("x was too big: 11")); assertTrue(f.origin().startsWith("AssertionFailedError at "), f.origin()); - assertNotNull(f.reproduceBlob()); - assertFalse(f.flaky()); - // Exactly one case was the final replay, and the run counted more than just it. - assertEquals(1, finals.get()); + assertTrue(f.reproduceBlob().isPresent()); + assertFalse(f.nondeterministic()); + assertEquals(Optional.empty(), f.caveat()); + // The engine stamped at least the final replay for capture, and the run counted more than + // the stamped cases. + assertTrue(finals.get() >= 1, "stamped cases: " + finals.get()); assertTrue(report.statistics().interesting() >= 2, report.statistics().toString()); assertTrue(report.statistics().total() > report.statistics().interesting()); } diff --git a/shared/src/test/java/dev/hegel/ReporterTest.java b/shared/src/test/java/dev/hegel/ReporterTest.java index b3d91f7..186799e 100644 --- a/shared/src/test/java/dev/hegel/ReporterTest.java +++ b/shared/src/test/java/dev/hegel/ReporterTest.java @@ -20,7 +20,7 @@ private static String text(ByteArrayOutputStream buf) { void silentReporterIgnoresEveryCallback() { Reporter silent = Reporter.silent(); RunReport report = new RunReport(RunStatus.PASSED, new RunStatistics.Counter().snapshot(), null, List.of()); - Failure failure = new Failure("origin", "blob", new AssertionError("x"), Map.of(), List.of()); + Failure failure = new Failure("origin", "blob", null, new AssertionError("x"), Map.of(), List.of()); silent.runStarted(new Settings()); silent.engineOutput("line"); silent.caseStarted(false); @@ -42,10 +42,23 @@ void printingReporterFormatsDrawsNotesAndEngineOutput() { printing.draw("xs", List.of(1, 2), true); printing.note("a note", true); printing.caseFinished(CaseOutcome.INTERESTING, true); - printing.failure(new Failure("o", "b64", new AssertionError("x"), Map.of(), List.of())); // printBlob off + printing.failure(new Failure("o", "b64", null, new AssertionError("x"), Map.of(), List.of())); // printBlob off assertEquals("engine line\nxs = [1, 2];\na note\n", text(buf)); } + @Test + void printingReporterPrintsTheCaveatThenTheReproducer() { + ByteArrayOutputStream buf = new ByteArrayOutputStream(); + Reporter printing = Reporter.printing(new PrintStream(buf, true, StandardCharsets.UTF_8)); + printing.runStarted(new Settings().printBlob(true)); + printing.failure(new Failure( + "o", "b64", "nondeterministic failure, confirmed", new AssertionError("x"), Map.of(), List.of())); + assertEquals( + "note: nondeterministic failure, confirmed\n\nTo reproduce this failure, replay it with:\n" + + " @HegelTest(reproduceFailure = \"b64\")\n", + text(buf)); + } + @Test void printingReporterHonoursVerbosity() { ByteArrayOutputStream quietBuf = new ByteArrayOutputStream(); diff --git a/shared/src/test/java/dev/hegel/ReproduceFailureTest.java b/shared/src/test/java/dev/hegel/ReproduceFailureTest.java index 1fab64b..9b9549e 100644 --- a/shared/src/test/java/dev/hegel/ReproduceFailureTest.java +++ b/shared/src/test/java/dev/hegel/ReproduceFailureTest.java @@ -1,12 +1,19 @@ package dev.hegel; import static dev.hegel.Generators.integers; +import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; +import dev.hegel.lowlevel.Abi; +import dev.hegel.lowlevel.Libhegel; import java.io.ByteArrayOutputStream; import java.io.PrintStream; import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Consumer; import org.junit.jupiter.api.Test; @@ -46,7 +53,27 @@ void printedBlobReplaysTheExactFailure() { () -> Hegel.test( tc -> tc.draw(integers().min(0).max(1000), "x"), new Settings().database(Database.disabled()).reproduceFailure(blob))); - assertTrue(stale.getMessage().contains("no longer reproduces"), stale.getMessage()); + assertTrue(stale.getMessage().contains("did not reproduce"), stale.getMessage()); + } + + @Test + void standaloneSingleAttemptReplayRemainsAvailable() { + // The runner drives blob replays through the engine's run loop, but the binding still + // exposes the one-shot hegel_test_case_from_blob for frontends built on hegel-lowlevel. + Libhegel lib = Engine.get(); + RunReport report = + Hegel.run(FAILING, new Settings().database(Database.disabled()).seed(1), Reporter.silent()); + String blob = report.failures().get(0).reproduceBlob().orElseThrow(); + long[] settings = new long[1]; + assertEquals(Abi.OK, lib.settingsNew(settings)); + lib.settingsDatabase(settings[0], ""); + List lines = new ArrayList<>(); + long[] tc = new long[1]; + assertEquals(Abi.OK, lib.testCaseFromBlob(settings[0], blob, lines::add, tc)); + assertTrue(tc[0] != 0); + assertEquals(Abi.OK, lib.markComplete(tc[0], Abi.STATUS_VALID, null)); + lib.testCaseFree(tc[0]); + lib.settingsFree(settings[0]); } @Test @@ -59,8 +86,37 @@ void corruptBlobsAreRejected() { } @Test - void flakyTestsAreDetected() { - java.util.concurrent.atomic.AtomicInteger calls = new java.util.concurrent.atomic.AtomicInteger(); + void aTestThatFailsOnceIsReportedUnconfirmedWithACaveat() { + AtomicInteger calls = new AtomicInteger(); + Consumer onceOnly = tc -> { + tc.draw(integers(), "x"); + if (calls.incrementAndGet() == 1) { + throw new AssertionError("only the first time"); + } + }; + Settings settings = new Settings().database(Database.disabled()).seed(3); + RunReport report = Hegel.run(onceOnly, settings, Reporter.silent()); + assertEquals(RunStatus.FAILED, report.status()); + Failure f = report.failures().get(0); + assertTrue(f.nondeterministic(), f.toString()); + String caveat = f.caveat().orElseThrow(); + assertTrue(caveat.startsWith("unconfirmed failure"), caveat); + // No replay ever failed again, so there is no blob, and the only failing execution was the + // unstamped discovery: the exception is the body's own, the draws were not recorded. + assertEquals(Optional.empty(), f.reproduceBlob()); + assertTrue(f.exception() instanceof AssertionError, String.valueOf(f.exception())); + assertTrue(f.draws().isEmpty(), f.draws().toString()); + + // Hegel.test rethrows the body's own failure, and the printed report carries the caveat. + calls.set(0); + String output = runCapturing(settings, onceOnly, AssertionError.class); + assertTrue(output.contains("note: unconfirmed failure"), output); + assertTrue(!output.contains("reproduceFailure = \""), output); + } + + @Test + void errorStrictnessAbortsOnANondeterministicTest() { + AtomicInteger calls = new AtomicInteger(); HegelException e = assertThrows( HegelException.class, () -> Hegel.test( @@ -70,8 +126,38 @@ void flakyTestsAreDetected() { throw new AssertionError("only the first time"); } }, - new Settings().database(Database.disabled()).seed(3))); - assertTrue(e.getMessage().contains("Flaky"), e.getMessage()); + new Settings() + .database(Database.disabled()) + .seed(3) + .nondeterminismStrictness(NondeterminismStrictness.ERROR))); + assertTrue(e.getMessage().toLowerCase().contains("flaky"), e.getMessage()); + } + + @Test + void confirmedNondeterministicFailureCarriesABlobThatReplays() { + // Fails every other time a large value is drawn: nondeterministic, but reproducible often + // enough for the engine to confirm it, shrink it, and hand back a blob. + AtomicInteger bigDraws = new AtomicInteger(); + Consumer intermittent = tc -> { + int x = tc.draw(integers().min(0).max(1000), "x"); + if (x > 10 && bigDraws.incrementAndGet() % 2 == 0) { + throw new AssertionError("x was too big (this time): " + x); + } + }; + Settings settings = new Settings().database(Database.disabled()).seed(7).printBlob(true); + String output = runCapturing(settings, intermittent, AssertionError.class); + assertTrue(output.contains("note: nondeterministic failure, confirmed"), output); + assertTrue(output.contains("reproduceFailure = \""), output); + String tail = output.substring(output.indexOf("reproduceFailure = \"") + "reproduceFailure = \"".length()); + String blob = tail.substring(0, tail.indexOf('"')); + + // The engine replays a nondeterministic blob until one of its recorded runs fails again. + RunReport replay = Hegel.run( + intermittent, new Settings().database(Database.disabled()).reproduceFailure(blob), Reporter.silent()); + assertEquals(RunStatus.FAILED, replay.status()); + assertTrue(replay.failures().get(0).nondeterministic()); + assertTrue(replay.failures().get(0).exception() instanceof AssertionError); + assertEquals(Optional.empty(), replay.failures().get(0).reproduceBlob()); } @Test diff --git a/shared/src/test/java/dev/hegel/RunReportTest.java b/shared/src/test/java/dev/hegel/RunReportTest.java index 925a27c..21a9d95 100644 --- a/shared/src/test/java/dev/hegel/RunReportTest.java +++ b/shared/src/test/java/dev/hegel/RunReportTest.java @@ -141,41 +141,87 @@ void failedRunCarriesTheReplayedFailure() { assertFalse(r.passed()); assertEquals(1, r.failures().size()); Failure f = r.failures().get(0); - assertEquals("fake-origin-0", f.origin()); - assertEquals("blob-1", f.reproduceBlob()); - assertSame(err, f.exception().get()); - assertFalse(f.flaky()); - // Only the final replay records draws and notes, so one exploration case plus one replay - // yields a single set of each. + // The engine echoes back the origin the runner marked the case with. + assertEquals(Runner.originOf(err, List.of()), f.origin()); + assertEquals(Optional.of("blob-1"), f.reproduceBlob()); + assertSame(err, f.exception()); + assertFalse(f.nondeterministic()); + assertEquals(Optional.empty(), f.caveat()); + // The fake stamps the case for capture, so its draws and notes are the report. assertEquals(List.of("x", "draw_1"), List.copyOf(f.draws().keySet())); assertEquals(7, f.draws().get("x")); assertEquals(List.of("saw 7"), f.notes()); assertThrows(UnsupportedOperationException.class, () -> f.draws().put("y", 1)); assertThrows(UnsupportedOperationException.class, () -> f.notes().add("more")); assertThrows(UnsupportedOperationException.class, () -> r.failures().clear()); - // Two cases ran: the exploration case and the replay. - assertEquals(2, r.statistics().total()); - assertEquals(2, r.statistics().interesting()); + // The engine ran the one case; nothing is replayed client-side. + assertEquals(1, r.statistics().total()); + assertEquals(1, r.statistics().interesting()); assertSame(err, assertThrows(AssertionError.class, r::throwIfFailed)); } @Test - void flakyReplayIsReportedRatherThanThrown() { + void nondeterministicFailureCarriesItsCaveat() { FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; - fake.failureBlobs.add("blob-1"); + fake.failureBlobs.add("nd-blob"); + fake.failureCaveats.add("nondeterministic failure, confirmed: failed 3 of 20 replays"); + AssertionError err = new AssertionError("sometimes"); + RunReport r = report(fake, new Settings().database(Database.disabled()), tc -> { + throw err; + }); + assertEquals(RunStatus.FAILED, r.status()); + Failure f = r.failures().get(0); + assertTrue(f.nondeterministic()); + assertEquals(Optional.of("nondeterministic failure, confirmed: failed 3 of 20 replays"), f.caveat()); + assertEquals(Optional.of("nd-blob"), f.reproduceBlob()); + // The failure is the test's own, caveat or not. + assertSame(err, assertThrows(AssertionError.class, r::throwIfFailed)); + } + + @Test + void unconfirmedFailureIsReportedFromItsDiscovery() { + // A test that failed once and never again: the engine reports the origin unconfirmed, with a + // caveat and no blob, and the only failing execution was the unstamped discovery, whose + // exception is all the runner kept. + FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 3; + fake.captureSequence = new boolean[] {false, true, true}; + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add(null); + fake.failureCaveats.add("nondeterministic failure, unconfirmed: failed 0 of 20 replays"); AtomicInteger calls = new AtomicInteger(); + AssertionError err = new AssertionError("only once"); RunReport r = report(fake, new Settings().database(Database.disabled()), tc -> { + tc.draw(integers(), "x"); if (calls.incrementAndGet() == 1) { - throw new AssertionError("only once"); + throw err; } }); - assertEquals(RunStatus.FAILED, r.status()); Failure f = r.failures().get(0); - assertTrue(f.flaky()); - assertEquals(Optional.empty(), f.exception()); - HegelException e = assertThrows(HegelException.class, r::throwIfFailed); - assertEquals(Runner.FLAKY_DIAGNOSTIC, e.getMessage()); + assertTrue(f.nondeterministic()); + assertEquals(Optional.empty(), f.reproduceBlob()); + assertSame(err, f.exception()); + assertTrue(f.draws().isEmpty()); + assertSame(err, assertThrows(AssertionError.class, r::throwIfFailed)); + } + + @Test + void stampedCaptureOutranksALaterUnstampedOne() { + // Shrink probes (unstamped) keep failing after the stamped replay: the report still comes + // from the stamped execution, and among stamped ones the newest wins. + FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 4; + fake.captureSequence = new boolean[] {false, true, true, false}; + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add("blob-1"); + AtomicInteger calls = new AtomicInteger(); + RunReport r = report(fake, new Settings().database(Database.disabled()), tc -> { + int n = calls.incrementAndGet(); + tc.note("case " + n); + throw new AssertionError("always"); + }); + assertEquals(List.of("case 3"), r.failures().get(0).notes()); } @Test @@ -187,7 +233,7 @@ void checkedExceptionsAreRethrownAsIs() { RunReport r = report(fake, new Settings().database(Database.disabled()), tc -> { throw sneaky(io); }); - assertSame(io, r.failures().get(0).exception().get()); + assertSame(io, r.failures().get(0).exception()); assertSame(io, assertThrows(IOException.class, r::throwIfFailed)); } @@ -226,7 +272,7 @@ void reporterCallbacksArriveInOrder() { fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); RecordingReporter reporter = new RecordingReporter(); - report( + RunReport r = report( fake, new Settings().database(Database.disabled()).testCases(5), tc -> { @@ -237,6 +283,8 @@ void reporterCallbacksArriveInOrder() { reporter); // Engine output goes through the run's callback to the reporter, whenever it arrives. fake.output.accept("engine says hi"); + // The stamped case's draws and notes are not reported live; they are replayed, flagged as + // such, once the engine reports the failure. assertEquals( List.of( "runStarted:5", @@ -247,14 +295,14 @@ void reporterCallbacksArriveInOrder() { "draw:x=3:true", "note:hi:true", "caseFinished:INTERESTING:true", - "failure:fake-origin-0", + "failure:" + r.failures().get(0).origin(), "runFinished:FAILED", "engineOutput:engine says hi"), reporter.events); } @Test - void verboseRunsReportEveryCaseButRecordOnlyTheReplay() { + void verboseRunsReportEveryCaseLiveAndReplayTheCapture() { FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); @@ -268,11 +316,11 @@ void verboseRunsReportEveryCaseButRecordOnlyTheReplay() { throw new AssertionError("boom"); }, reporter); - // The exploration case's draws and notes are reported (flagged non-final) ... + // The case's draws and notes are reported live (flagged as such) ... assertTrue(reporter.events.contains("draw:x=3:false"), reporter.events.toString()); assertTrue(reporter.events.contains("note:hi:false"), reporter.events.toString()); + // ... and replayed with the failure report. assertTrue(reporter.events.contains("draw:x=3:true"), reporter.events.toString()); - // ... but the failure captures the final replay only. assertEquals(Map.of("x", 3), r.failures().get(0).draws()); assertEquals(List.of("hi"), r.failures().get(0).notes()); } @@ -280,6 +328,7 @@ void verboseRunsReportEveryCaseButRecordOnlyTheReplay() { @Test void multipleFailuresAreReportedInEngineOrder() { FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 2; // one case per distinct bug fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); fake.failureBlobs.add("blob-2"); @@ -296,8 +345,9 @@ void multipleFailuresAreReportedInEngineOrder() { }, reporter); assertEquals(2, r.failures().size()); - assertEquals("blob-1", r.failures().get(0).reproduceBlob()); - assertEquals("fake-origin-1", r.failures().get(1).origin()); + assertEquals(Optional.of("blob-1"), r.failures().get(0).reproduceBlob()); + assertTrue(r.failures().get(0).origin().startsWith("AssertionError at ")); + assertTrue(r.failures().get(1).origin().startsWith("IllegalStateException at ")); assertTrue(reporter.events.contains("failuresFound:2")); AssertionError e = assertThrows(AssertionError.class, r::throwIfFailed); assertEquals(2, e.getSuppressed().length); @@ -306,6 +356,8 @@ void multipleFailuresAreReportedInEngineOrder() { @Test void reproduceFailureReportsTheReplayedFailure() { FakeLibhegel fake = new FakeLibhegel(); + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add(null); // a blob replay's failure carries no fresh blob RecordingReporter reporter = new RecordingReporter(); IllegalStateException err = new IllegalStateException("reproduced"); RunReport r = report( @@ -316,20 +368,28 @@ void reproduceFailureReportsTheReplayedFailure() { throw err; }, reporter); + assertEquals("stored-blob", fake.startedBlob); assertEquals(RunStatus.FAILED, r.status()); Failure f = r.failures().get(0); - assertEquals("stored-blob", f.reproduceBlob()); - assertSame(err, f.exception().get()); + assertEquals(Optional.empty(), f.reproduceBlob()); + assertSame(err, f.exception()); // The test suite itself lives in dev.hegel, which counts as infrastructure, so the origin // names the exception and whatever frame sits below the suite. assertTrue(f.origin().startsWith("IllegalStateException at "), f.origin()); assertEquals(Map.of("x", 1), f.draws()); assertEquals(1, r.statistics().total()); assertEquals(1, r.statistics().interesting()); - // No exploration loop: straight to the replay. + // The blob run is pumped like any other, then its failure is reported from the capture. assertEquals( - List.of("runStarted:100", "caseStarted:true", "draw:x=1:true", "caseFinished:INTERESTING:true"), - reporter.events.subList(0, 4)); + List.of( + "runStarted:100", + "caseStarted:false", + "caseFinished:INTERESTING:false", + "failuresFound:1", + "caseStarted:true", + "draw:x=1:true", + "caseFinished:INTERESTING:true"), + reporter.events.subList(0, 7)); assertEquals("runFinished:FAILED", reporter.events.get(reporter.events.size() - 1)); } diff --git a/shared/src/test/java/dev/hegel/RunnerTest.java b/shared/src/test/java/dev/hegel/RunnerTest.java index 6cf0eb5..bc85704 100644 --- a/shared/src/test/java/dev/hegel/RunnerTest.java +++ b/shared/src/test/java/dev/hegel/RunnerTest.java @@ -139,7 +139,7 @@ private static FakeLibhegel runFailing(Consumer body) { } @Test - void replayPassesTheBlobToTheEngine() { + void failuresAreReportedFromTheCaptureWithoutAClientReplay() { FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-xyz"); @@ -148,49 +148,52 @@ void replayPassesTheBlobToTheEngine() { () -> run(fake, new Settings().database(Database.disabled()), tc -> { throw new AssertionError("always"); })); - assertEquals(List.of("blob-xyz"), fake.replayedBlobs); - // One exploration case plus one replay case, all freed. - assertEquals(2, fake.freedTestCases); - } - - @Test - void replayThatPassesIsFlaky() { - AtomicInteger calls = new AtomicInteger(); - HegelException e = assertThrows( - HegelException.class, - () -> runFailing(tc -> { - if (calls.incrementAndGet() == 1) { - throw new AssertionError("only once"); - } - })); - assertTrue(e.getMessage().contains("Flaky"), e.getMessage()); + // The engine owns every replay: nothing is replayed from the blob client-side. + assertTrue(fake.replayedBlobs.isEmpty()); + assertNull(fake.startedBlob); + assertEquals(1, fake.freedTestCases); } @Test - void missingBlobIsAnInternalError() { + void failureWithoutACapturedExecutionIsAnInternalError() { + // The engine reports a failure whose origin never failed in this process: a plumbing bug. FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; - fake.failureBlobs.add(null); + fake.failureBlobs.add("blob-1"); + fake.failureOrigins.add("AssertionError at Elsewhere.java:1"); HegelException e = assertThrows( HegelException.class, () -> run(fake, new Settings().database(Database.disabled()), tc -> {})); - assertTrue(e.getMessage().contains("no reproduce blob"), e.getMessage()); + assertTrue(e.getMessage().contains("no captured failing execution"), e.getMessage()); + assertTrue(e.getMessage().contains("Elsewhere.java:1"), e.getMessage()); } @Test - void undecodableBlobFailsTheRun() { + void unstampedCasesKeepOnlyTheirException() { FakeLibhegel fake = new FakeLibhegel(); + fake.captureSequence = new boolean[] {false}; fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); - fake.fromBlobRc = Abi.E_INVALID_ARG; - fake.lastError = "bad blob"; - HegelException e = assertThrows( - HegelException.class, () -> run(fake, new Settings().database(Database.disabled()), tc -> {})); - assertTrue(e.getMessage().contains("bad blob"), e.getMessage()); + ByteArrayOutputStream buf = new ByteArrayOutputStream(); + AssertionError err = new AssertionError("always"); + RunReport report = Runner.run( + fake, + new Settings().database(Database.disabled()).printBlob(false), + tc -> { + tc.draw(integers(), "x"); + tc.note("unseen"); + throw err; + }, + Reporter.printing(capture(buf))); + assertSame(err, report.failures().get(0).exception()); + assertTrue(report.failures().get(0).draws().isEmpty()); + assertTrue(report.failures().get(0).notes().isEmpty()); + assertEquals("", buf.toString(StandardCharsets.UTF_8)); } @Test void multipleFailuresAggregateWithSuppressedOriginals() { FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 2; // one case per distinct bug fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); fake.failureBlobs.add("blob-2"); @@ -219,15 +222,21 @@ void multipleFailuresAggregateWithSuppressedOriginals() { @Test void aggregateMessageHandlesNullExceptionMessages() { FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 2; fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); fake.failureBlobs.add("blob-2"); + AtomicInteger calls = new AtomicInteger(); AssertionError e = assertThrows( AssertionError.class, () -> run(fake, new Settings(), tc -> { - throw new IllegalStateException(); // null message + if (calls.incrementAndGet() == 1) { + throw new IllegalStateException(); // null message + } + throw new UnsupportedOperationException(); // null message })); assertTrue(e.getMessage().contains(IllegalStateException.class.getName()), e.getMessage()); + assertTrue(e.getMessage().contains(UnsupportedOperationException.class.getName()), e.getMessage()); } @Test @@ -282,8 +291,10 @@ void nullRunErrorBecomesEmptyMessage() { } @Test - void reproduceFailureReplaysWithoutARun() { + void reproduceFailureStartsABlobRun() { FakeLibhegel fake = new FakeLibhegel(); + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add(null); IllegalStateException err = new IllegalStateException("reproduced"); assertSame( err, @@ -292,38 +303,65 @@ void reproduceFailureReplaysWithoutARun() { () -> run(fake, new Settings().reproduceFailure("stored-blob"), tc -> { throw err; }))); - assertEquals(List.of("stored-blob"), fake.replayedBlobs); - assertNull(fake.output); // runStart was never called + assertEquals("stored-blob", fake.startedBlob); + assertTrue(fake.replayedBlobs.isEmpty()); + assertTrue(fake.runFreed); assertTrue(fake.settingsFreed); } @Test void reproduceFailureReportsAStaleBlob() { + // The engine's blob run passed: none of its replays failed. FakeLibhegel fake = new FakeLibhegel(); HegelException e = assertThrows(HegelException.class, () -> run(fake, new Settings().reproduceFailure("stale"), tc -> {})); - assertTrue(e.getMessage().contains("no longer reproduces"), e.getMessage()); + assertEquals(Runner.STALE_BLOB, e.getMessage()); + assertTrue(e.getMessage().contains("did not reproduce"), e.getMessage()); } @Test void reproduceFailureRejectsAnInvalidBlob() { + // An undecodable blob surfaces as the blob run's error. FakeLibhegel fake = new FakeLibhegel(); - fake.fromBlobRc = Abi.E_INVALID_ARG; - fake.lastError = "corrupt"; + fake.runStatus = Abi.RUN_STATUS_ERROR; + fake.runError = "corrupt"; HegelException e = assertThrows(HegelException.class, () -> run(fake, new Settings().reproduceFailure("???"), tc -> {})); + assertTrue(e.getMessage().contains("not valid"), e.getMessage()); assertTrue(e.getMessage().contains("corrupt"), e.getMessage()); } @Test - void nondeterministicRunFailureIsAnInternalError() { - // Only a concurrent state machine declares a run nondeterministic, and this binding never - // creates one, so the engine reporting that status (whose failures carry no blob) is a bug. + void nondeterminismStrictnessIsSentOnlyWhenSet() { FakeLibhegel fake = new FakeLibhegel(); - fake.runStatus = Abi.RUN_STATUS_FAILED_NONDETERMINISTIC; - HegelException e = assertThrows(HegelException.class, () -> run(fake, new Settings(), tc -> {})); - assertTrue(e.getMessage().contains("nondeterministic"), e.getMessage()); - assertTrue(fake.replayedBlobs.isEmpty()); + run(fake, new Settings().database(Database.disabled()), tc -> {}); + assertNull(fake.strictness); + run( + fake, + new Settings().database(Database.disabled()).nondeterminismStrictness(NondeterminismStrictness.WARN), + tc -> {}); + assertEquals(Integer.valueOf(Abi.NONDETERMINISM_WARN), fake.strictness); + } + + @Test + void effectiveStrictnessIsReadBackFromTheEngine() { + FakeLibhegel fake = new FakeLibhegel(); + fake.resolvedStrictness = Abi.NONDETERMINISM_ERROR; + Settings[] seen = new Settings[1]; + Reporter reporter = new Reporter() { + @Override + public void runStarted(Settings settings) { + seen[0] = settings; + } + }; + Runner.run(fake, new Settings().database(Database.disabled()), tc -> {}, reporter); + assertEquals(NondeterminismStrictness.ERROR, seen[0].nondeterminismStrictness); + assertEquals(NondeterminismStrictness.DEFAULT, new Settings().nondeterminismStrictness); + + FakeLibhegel unknown = new FakeLibhegel(); + unknown.resolvedStrictness = 99; + HegelException e = assertThrows(HegelException.class, () -> run(unknown, new Settings(), tc -> {})); + assertTrue(e.getMessage().contains("99"), e.getMessage()); } @Test @@ -448,22 +486,41 @@ void infrastructurePackagesAreSkippedWhenLocatingTheOrigin() { } @Test - void printBlobIsSkippedForAFlakyReplay() { + void caveatIsPrintedAndABloblessFailurePrintsNoReproducer() { FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; - fake.failureBlobs.add("blob-b64"); - AtomicInteger calls = new AtomicInteger(); + fake.failureBlobs.add(null); + fake.failureCaveats.add("nondeterministic failure, unconfirmed: failed 0 of 20 replays"); ByteArrayOutputStream buf = new ByteArrayOutputStream(); RunReport report = Runner.run( fake, new Settings().database(Database.disabled()).printBlob(true), tc -> { - if (calls.incrementAndGet() == 1) { - throw new AssertionError("only once"); - } + tc.draw(integers().min(0), "x"); + throw new AssertionError("sometimes"); }, Reporter.printing(capture(buf))); - assertTrue(report.failures().get(0).flaky()); - assertEquals("", buf.toString(StandardCharsets.UTF_8)); + assertTrue(report.failures().get(0).nondeterministic()); + String out = buf.toString(StandardCharsets.UTF_8).replace("\r\n", "\n"); + assertEquals("x = 0;\nnote: nondeterministic failure, unconfirmed: failed 0 of 20 replays\n", out); + } + + @Test + void quietRunsPrintNoCaveat() { + FakeLibhegel fake = new FakeLibhegel(); + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add("nd-blob"); + fake.failureCaveats.add("nondeterministic failure, confirmed: failed 9 of 20 replays"); + ByteArrayOutputStream buf = new ByteArrayOutputStream(); + Runner.run( + fake, + new Settings().database(Database.disabled()).printBlob(true).verbosity(Verbosity.QUIET), + tc -> { + throw new AssertionError("sometimes"); + }, + Reporter.printing(capture(buf))); + String out = buf.toString(StandardCharsets.UTF_8); + assertTrue(!out.contains("note:"), out); + assertTrue(out.contains("reproduceFailure = \"nd-blob\""), out); } } diff --git a/shared/src/test/java/dev/hegel/TestCaseTest.java b/shared/src/test/java/dev/hegel/TestCaseTest.java index cecaab1..dc7988a 100644 --- a/shared/src/test/java/dev/hegel/TestCaseTest.java +++ b/shared/src/test/java/dev/hegel/TestCaseTest.java @@ -18,11 +18,15 @@ private static String text(ByteArrayOutputStream buf) { return buf.toString(StandardCharsets.UTF_8).replace("\r\n", "\n"); } - /** A TestCase over a fake binding; only the reporting/target plumbing is under test here. */ + /** + * A TestCase over a fake binding, both stamped for capture and reporting live (verbose) when + * {@code reporting}; only the reporting/target plumbing is under test here. + */ private TestCase newCase(FakeLibhegel fake, boolean reporting, ByteArrayOutputStream buf) { return new TestCase( new LiveDataSource(fake, FakeLibhegel.TC), reporting, + reporting, Reporter.printing(new PrintStream(buf, true, StandardCharsets.UTF_8))); } @@ -75,7 +79,7 @@ void noteRespectsReporting() { } @Test - void finalReplayRecordsDrawsAndNotes() { + void capturedCaseRecordsDrawsAndNotesAndReplaysThem() { TestCase tc = newCase(new FakeLibhegel(), true, new ByteArrayOutputStream()); assertTrue(tc.isFinal()); tc.draw(constant(1), "x"); @@ -87,6 +91,24 @@ void finalReplayRecordsDrawsAndNotes() { assertEquals(want, tc.draws()); assertEquals(List.of("x", "draw_1"), List.copyOf(tc.draws().keySet())); assertEquals(List.of("first"), tc.notes()); + // The recording replays in report order, flagged as a replay. + ByteArrayOutputStream replay = new ByteArrayOutputStream(); + Reporter reporter = Reporter.printing(new PrintStream(replay, true, StandardCharsets.UTF_8)); + reporter.runStarted(new Settings()); + tc.replayTo(reporter); + assertEquals("x = 1;\nfirst\ndraw_1 = 2;\n", text(replay)); + + // A case stamped for capture but not verbose records without reporting live. + ByteArrayOutputStream silent = new ByteArrayOutputStream(); + TestCase captured = new TestCase( + new LiveDataSource(new FakeLibhegel(), FakeLibhegel.TC), + true, + Reporter.printing(new PrintStream(silent, true, StandardCharsets.UTF_8))); + captured.draw(constant(3), "y"); + captured.note("kept"); + assertEquals("", text(silent)); + assertEquals(Map.of("y", 3), captured.draws()); + assertEquals(List.of("kept"), captured.notes()); TestCase exploring = newCase(new FakeLibhegel(), false, new ByteArrayOutputStream()); assertTrue(!exploring.isFinal());