diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 575f615..df5d39c 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -114,12 +114,26 @@ public `Generator`/`TestCase`/`Generators`/`Hegel`/`Stateful` surface stays in ` `AssumeRejected` → INVALID, `INVALID_ARG` → `IllegalArgumentException` with the engine's diagnostic, other negatives → `HegelException`) and short-circuiting once a case is aborted. - **Run loop** — `Runner` builds the settings handle and drives `hegel_run_start` → - `hegel_next_test_case` → `hegel_mark_complete`. The engine only *explores* (generation and - shrinking), so every pumped case is non-final; after the loop drains, `hegel_run_result` yields - PASSED, FAILED, or ERROR. On FAILED the runner replays each distinct counterexample's - **reproduce blob** (`hegel_test_case_from_blob`) with reporting enabled — capturing the shrunk - draws, notes, and the body's own exception into a `Failure` (a replay that no longer fails is a - *flaky* failure). The runner returns a `RunReport` (status, client-side `RunStatistics` counted + `hegel_next_test_case` → `hegel_mark_complete`. The engine owns the *whole* run — generation, + shrinking, the replays that confirm a failure, and (since 0.44) the final replay of every failure + it is about to report — so the runner never replays anything itself. It reads + `hegel_test_case_should_capture` once per case: a *stamped* case's `TestCase` records its + top-level draws and notes (`isFinal()` exposes the stamp), and every interesting case's exception + is kept per origin in a capture map (a stamped capture outranks an unstamped one, newest wins at + equal rank — the unstamped discovery of a failure that never fails again is all there is for an + *unconfirmed* nondeterministic failure). After the loop drains, `hegel_run_result` yields PASSED, + FAILED, or ERROR. On FAILED each distinct failure is built from its origin's capture: the recorded + draws/notes are replayed to the reporter (`caseStarted(true)` … `caseFinished(true)`), and the + `Failure` carries the body's own exception, the reproduce blob (`hegel_failure_reproduction_blob`, + null for an unconfirmed failure and for a blob replay) and the **caveat** + (`hegel_failure_caveat`, non-null for a nondeterministic failure — `Failure.nondeterministic()`; + the printing reporter prints it as `note: …`). A failure whose origin has no capture is an + internal error. `Settings.reproduceFailure` drives the same loop over `hegel_run_start_blob` + (the engine replays until a replay fails, under its budget): PASSED → `Runner.STALE_BLOB` + `HegelException`, ERROR → "blob is not valid" `HegelException`. There is no Java-side flaky + detection any more: the engine handles nondeterminism per `nondeterminismStrictness` + (`NondeterminismStrictness.QUIET`/`WARN` confirm-by-replay and report with a caveat; `ERROR` + aborts the run, surfacing as the engine's own `Flaky test detected` message). The runner returns a `RunReport` (status, client-side `RunStatistics` counted per `mark_complete` — the C ABI has no counter accessor — engine error, failures) and never prints: everything goes through the run's `Reporter` (`Reporter.printing(System.err)` is the default and reproduces the classic output; `Reporter.silent()` for library use). `Hegel.run` @@ -127,17 +141,17 @@ public `Generator`/`TestCase`/`Generators`/`Hegel`/`Stateful` surface stays in ` failure as-is (checked exceptions included), aggregates several into an `AssertionError`, and maps ERROR to `HealthCheckFailure`/`HegelException`. `HegelException` (binding/engine errors) is always thrown, never reported. `Settings` is the immutable config; its closed-state setting types live - alongside it — `Backend` (auto = leave it to the engine's profile, which picks urandom inside Antithesis / default / urandom), `Database`, `OptBoolean` — + alongside it — `Backend` (auto = leave it to the engine's profile, which picks urandom inside Antithesis / default / urandom), `Database`, `OptBoolean`, `NondeterminismStrictness` (`DEFAULT` = leave it to the engine) — plus `printBlob` (print a copy-pasteable reproducer per failure) and `reproduceFailure` (replay - a stored blob instead of running the property). `testCases`, `printBlob`, `derandomize`, `seed` - and `database` are nullable ("unset"): `Runner.applySettings` sends only what the user set, so + a stored blob instead of running the property). `testCases`, `printBlob`, `derandomize`, `seed`, + `nondeterminismStrictness` and `database` are nullable ("unset"): `Runner.applySettings` sends only what the user set, so the engine's resolution — its profile (`hegel.toml`, the shipped `development`/`ci`/`workload` profiles) with the `HEGEL_TEST_CASES`/`HEGEL_DATABASE`/`HEGEL_SEED`/`HEGEL_DERANDOMIZE`/ - `HEGEL_PRINT_BLOB` variables applied over it in `hegel_settings_new` — stands for the rest, and + `HEGEL_PRINT_BLOB`/`HEGEL_NONDETERMINISM_STRICTNESS` variables applied over it in `hegel_settings_new` — stands for the rest, and explicit Java settings win over both. There is no Java-side CI detection (the engine's `ci` profile covers it). `Libhegel.settingsNew` returns a raw result code because a malformed variable or `hegel.toml` fails it with `E_INVALID_ARG` (→ `IllegalArgumentException`); after - applying, the runner reads `settingsGetTestCases`/`settingsGetPrintBlob` back and hands the + applying, the runner reads `settingsGetTestCases`/`settingsGetPrintBlob`/`settingsGetNondeterminismStrictness` back and hands the *effective* `Settings` to the reporter and the run. Verbosity and `reportMultipleFailures` keep Java-side defaults and are always sent. `EnvironmentTest` covers the variables and `hegel.toml` through a child JVM (`EnvironmentFixture`), since the engine reads its own process environment. @@ -173,7 +187,9 @@ public `Generator`/`TestCase`/`Generators`/`Hegel`/`Stateful` surface stays in ` for `@Invariant(alwaysRun = true)`, sampled otherwise; the initial and final checks run every invariant unconditionally. The machine handle is freed in a `finally`. An engine-level `E_ASSUME` inside a rule (the data source is aborted) unwinds the whole case as invalid instead - of being reported as the rule's rejection. `Pool` tracks previously generated values over + of being reported as the rule's rejection. (Since engine 0.44 concurrency alone no longer marks + a run nondeterministic, and `HEGEL_RUN_STATUS_FAILED_NONDETERMINISTIC` is retired; this binding + drives machines sequentially anyway.) `Pool` tracks previously generated values over the engine's pool primitives so rules can reuse or consume them. - **Derivation** — `dev.hegel.generators.Derive` + `RecordGenerator` build generators from records, enums, scalars, and generic `List`/`Set`/`Optional`/`Map` by reflection. diff --git a/README.md b/README.md index a5bdd1d..87167d9 100644 --- a/README.md +++ b/README.md @@ -109,13 +109,15 @@ report.statistics(); // valid / invalid / overrun / interesti for (Failure f : report.failures()) { // one per distinct counterexample f.draws(); // labelled draws of the minimal example, as Java values f.exception(); // the body's own throwable - f.reproduceBlob(); // replay it later with Settings.reproduceFailure + f.reproduceBlob(); // Optional: replay it later with Settings.reproduceFailure + f.caveat(); // Optional: how reliably a nondeterministic failure reproduced } ``` -`TestCase.isFinal()` identifies the final replay of a counterexample, `TestCase.span` and `Label` -let custom composite generators tell the engine about their structure, and -`Settings.infrastructurePackages` keeps a frontend's own stack frames out of failure origins. +`TestCase.isFinal()` identifies the executions the engine stamped for the failure report (the +final replay of a counterexample among them), `TestCase.span` and `Label` let custom composite +generators tell the engine about their structure, and `Settings.infrastructurePackages` keeps a +frontend's own stack frames out of failure origins. ## Binding Hegel yourself diff --git a/RELEASE.md b/RELEASE.md new file mode 100644 index 0000000..271edb3 --- /dev/null +++ b/RELEASE.md @@ -0,0 +1,79 @@ +RELEASE_TYPE: minor + +This release upgrades the bundled libhegel engine from 0.43.4 to 0.44.0, which handles +nondeterministic tests instead of refusing them, and changes the `Failure` API accordingly. + +A test whose outcome changes when the same generated data is replayed — because it depends on hidden +global state, time, an outside service, or thread scheduling — previously aborted the run with a +`HegelException` reading "Flaky test detected". The engine now switches such a run to +nondeterministic handling: a discovered failure is confirmed by repeated replay before it is shrunk +or saved to the database, shrunk under a guard on how much reproduction reliability a shrink step may +trade away, and reported with a *caveat* quoting the run's own replay evidence. `Hegel.test` then +throws the test's own exception as for any other failure, and the printed report carries the caveat: + +``` +x = 11; +note: nondeterministic failure, confirmed: failed 7 of 20 replays at confirmation and 1 of 2 at report time + +To reproduce this failure, replay it with: + @HegelTest(reproduceFailure = "...") +``` + +A failure that never reproduces after its discovery still fails the run, reported as +`unconfirmed failure: failed 0 of 10 replays after the observed failure` and without a reproduce +blob. A reproduce blob from a nondeterministic failure records the failing runs the engine saw, and +`Settings.reproduceFailure` / `@HegelTest(reproduceFailure = ...)` now replays a blob until one of +its replays fails, under a bounded budget, instead of judging it stale after a single attempt; a +blob none of whose replays fail reports that "the supplied failure blob did not reproduce a failure". + +The new `Settings.nondeterminismStrictness` (and `@HegelTest(nondeterminismStrictness = ...)`) +controls the reaction: `NondeterminismStrictness.QUIET` (the engine's default) switches silently, +`WARN` prints a one-line notice once per run, and `ERROR` aborts the run with the previous +flaky-test error, for suites that use determinism as a lint. It is a profile setting like any other +(`nondeterminism_strictness = "error"` in `hegel.toml`), the `HEGEL_NONDETERMINISM_STRICTNESS` +environment variable (`quiet`, `warn` or `error`) overrides it for one run, and a value set in Java +wins over both. + +The engine now also runs every failure it is about to report one final time itself, so hegel-java no +longer replays reproduce blobs after a run: the reported draws, notes and exception come from the +engine's own final replay, and a failing test's body runs once less per reported failure. Two +visible consequences. `TestCase.isFinal()` is now true on every execution the engine stamps for the +failure report — the final replay of a counterexample, but also the short replays that confirm a +discovered failure — so a body that uses it to gate expensive diagnostics may run them a few times +per failure rather than once. And a `Reporter` receives the counterexample's `draw` and `note` +callbacks (flagged `finalReplay = true`) when the failure is reported, after the run loop, rather +than live while a replay executes; the order of callbacks is unchanged. + +`Failure` changes shape to carry the caveat and to admit failures without a blob: + +```java +// before +String blob = f.reproduceBlob(); +Optional e = f.exception(); +if (f.flaky()) { ... } + +// after +Optional blob = f.reproduceBlob(); // empty for an unconfirmed failure or a blob replay +Throwable e = f.exception(); // always present +Optional caveat = f.caveat(); // present for a nondeterministic failure +if (f.nondeterministic()) { ... } +``` + +`Failure.flaky()` is gone: the engine no longer hands back a failure whose replay passed. A +`Failure.reproduceBlob()` is also empty for a failure reproduced from a `reproduceFailure` blob, since +the caller already holds it. + +In `dev.hegel:hegel-lowlevel`, `Libhegel` gains `runStartBlob` (`hegel_run_start_blob`), +`testCaseShouldCapture` (`hegel_test_case_should_capture`), `failureCaveat` (`hegel_failure_caveat`) +and the `settingsNondeterminismStrictness` / `settingsGetNondeterminismStrictness` pair, and +`Abi.RUN_STATUS_FAILED_NONDETERMINISTIC` is removed along with the ABI value it mirrored; a failing +nondeterministic run reports plain `RUN_STATUS_FAILED`. Bindings implementing `Libhegel` must add +the new methods. + +Between 0.43.4 and 0.44.0 the engine also improved float generation, so tests on floats reach +overflow, underflow, cancellation and special-value bugs far more often (each test case draws its own +mixture of float categories, and bounded ranges cover every power-of-two scale they touch); roughly +halved the per-case cost of text generators built inside the test body by sharing built alphabets +between generators with the same constraints; shrinks a size drawn twice — the rows and columns of a +square matrix — while the values it governs still vary, reaching the smallest failing square; and +picked up correctness fixes in its arbitrary-precision integer dependency. diff --git a/hegel-jna/src/main/java/dev/hegel/JnaLibhegel.java b/hegel-jna/src/main/java/dev/hegel/JnaLibhegel.java index aa52f8c..4c1a2f5 100644 --- a/hegel-jna/src/main/java/dev/hegel/JnaLibhegel.java +++ b/hegel-jna/src/main/java/dev/hegel/JnaLibhegel.java @@ -87,9 +87,21 @@ interface HegelNative extends Library { int hegel_settings_get_print_blob(Pointer ctx, Pointer s, ByteByReference out); + int hegel_settings_set_nondeterminism_strictness(Pointer ctx, Pointer s, int strictness); + + int hegel_settings_get_nondeterminism_strictness(Pointer ctx, Pointer s, IntByReference out); + int hegel_run_start( Pointer ctx, Pointer settings, LineCallback callback, Pointer userData, PointerByReference out); + int hegel_run_start_blob( + Pointer ctx, + Pointer settings, + String blob, + LineCallback callback, + Pointer userData, + PointerByReference out); + int hegel_next_test_case(Pointer ctx, Pointer run, PointerByReference out); int hegel_run_result(Pointer ctx, Pointer run, PointerByReference out); @@ -108,6 +120,8 @@ int hegel_test_case_from_blob( int hegel_test_case_free(Pointer ctx, Pointer tc); + int hegel_test_case_should_capture(Pointer ctx, Pointer tc, ByteByReference out); + int hegel_generate_boolean(Pointer ctx, Pointer tc, double p, byte hasForced, byte forced, ByteByReference out); int hegel_generate_integer(Pointer ctx, Pointer tc, long min, long max, LongByReference out); @@ -237,6 +251,8 @@ int hegel_state_machine_should_check_invariant( int hegel_failure_origin(Pointer ctx, Pointer failure, PointerByReference out); + int hegel_failure_caveat(Pointer ctx, Pointer failure, PointerByReference out); + int hegel_version(Pointer ctx, PointerByReference out); } @@ -430,6 +446,22 @@ public boolean settingsGetPrintBlob(long s) { return out.getValue() != 0; } + @Override + public void settingsNondeterminismStrictness(long s, int strictness) { + check( + "hegel_settings_set_nondeterminism_strictness", + lib.hegel_settings_set_nondeterminism_strictness(ctx(), pointer(s), strictness)); + } + + @Override + public int settingsGetNondeterminismStrictness(long s) { + IntByReference out = new IntByReference(); + check( + "hegel_settings_get_nondeterminism_strictness", + lib.hegel_settings_get_nondeterminism_strictness(ctx(), pointer(s), out)); + return out.getValue(); + } + // --- run lifecycle --- @Override @@ -448,6 +480,24 @@ public long runStart(long settings, Consumer output) { return run; } + @Override + public long runStartBlob(long settings, String blob, Consumer output) { + LineCallback callback = output == null ? null : new LineCallback(output); + PointerByReference out = new PointerByReference(); + int code = lib.hegel_run_start_blob(ctx(), pointer(settings), blob, callback, null, out); + if (code != Abi.OK) { + throw new HegelException("hegel_run_start_blob failed (rc=" + + code + + "): " + + java.util.Objects.toString(lastErrorMessage(), "")); + } + long run = address(out.getValue()); + if (callback != null) { + runCallbacks.put(run, callback); + } + return run; + } + @Override public long nextTestCase(long run) { PointerByReference out = new PointerByReference(); @@ -491,6 +541,13 @@ public void testCaseFree(long tc) { check("hegel_test_case_free", lib.hegel_test_case_free(ctx(), pointer(tc))); } + @Override + public boolean testCaseShouldCapture(long tc) { + ByteByReference out = new ByteByReference(); + check("hegel_test_case_should_capture", lib.hegel_test_case_should_capture(ctx(), pointer(tc), out)); + return out.getValue() != 0; + } + // --- draws --- @Override @@ -980,6 +1037,18 @@ public String failureOrigin(long result, long index) { return origin; } + @Override + public String failureCaveat(long result, long index) { + PointerByReference failureOut = new PointerByReference(); + check("hegel_run_result_failure", lib.hegel_run_result_failure(ctx(), pointer(result), index, failureOut)); + Pointer failure = failureOut.getValue(); + PointerByReference caveatOut = new PointerByReference(); + check("hegel_failure_caveat", lib.hegel_failure_caveat(ctx(), failure, caveatOut)); + String caveat = readCString(caveatOut.getValue()); + check("hegel_failure_free", lib.hegel_failure_free(ctx(), failure)); + return caveat; + } + // --- diagnostics --- @Override diff --git a/hegel-jna/src/test/java/dev/hegel/JnaLibhegelTest.java b/hegel-jna/src/test/java/dev/hegel/JnaLibhegelTest.java index 7fe4811..86a6f7b 100644 --- a/hegel-jna/src/test/java/dev/hegel/JnaLibhegelTest.java +++ b/hegel-jna/src/test/java/dev/hegel/JnaLibhegelTest.java @@ -72,6 +72,8 @@ void infrastructureCallsReportNullHandles() { // diagnostic rather than undefined behaviour. assertThrows(HegelException.class, () -> lib.runResultStatus(0)); assertThrows(HegelException.class, () -> lib.runStart(0, null)); + assertThrows(HegelException.class, () -> lib.runStartBlob(0, "blob", null)); + assertThrows(HegelException.class, () -> lib.testCaseShouldCapture(0)); } @Test @@ -197,6 +199,10 @@ void settingsGettersReadTheHandleBack() { assertFalse(lib.settingsGetPrintBlob(s)); lib.settingsPrintBlob(s, true); assertTrue(lib.settingsGetPrintBlob(s)); + lib.settingsNondeterminismStrictness(s, Abi.NONDETERMINISM_ERROR); + assertEquals(Abi.NONDETERMINISM_ERROR, lib.settingsGetNondeterminismStrictness(s)); + lib.settingsNondeterminismStrictness(s, Abi.NONDETERMINISM_WARN); + assertEquals(Abi.NONDETERMINISM_WARN, lib.settingsGetNondeterminismStrictness(s)); lib.settingsFree(s); } @@ -216,6 +222,23 @@ void undecodableBlobWithDefaultOutputIsRejected() { lib.settingsFree(s); } + @Test + void blobRunWithDefaultOutputStartsAndReportsTheBadBlob() { + JnaLibhegel lib = real(); + long s = newSettings(lib); + // A null output callback leaves the run's output on stderr; the garbage blob surfaces as + // the run's error once it is pumped dry. + long run = lib.runStartBlob(s, "not-a-blob!!!", null); + assertNotEquals(0, run); + assertEquals(0, lib.nextTestCase(run)); + long result = lib.runResult(run); + assertEquals(Abi.RUN_STATUS_ERROR, lib.runResultStatus(result)); + assertNotNull(lib.runResultError(result)); + lib.runResultFree(result); + lib.runFree(run); + lib.settingsFree(s); + } + @Test void regexGeneratorAcceptsATextAlphabet() { JnaLibhegel lib = real(); diff --git a/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Abi.java b/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Abi.java index dc97813..9a376e3 100644 --- a/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Abi.java +++ b/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Abi.java @@ -21,11 +21,17 @@ private Abi() {} public static final int E_CONCURRENT_USE = -9; public static final int E_RETRY = -10; - // Aggregate run outcome (hegel_run_status_t). + // Aggregate run outcome (hegel_run_status_t). Value 3 (FAILED_NONDETERMINISTIC, retired in the + // 0.44 ABI) is never reported and must not be reused: a failing nondeterministic run reports + // plain FAILED, with a caveat on each failure. public static final int RUN_STATUS_PASSED = 0; public static final int RUN_STATUS_FAILED = 1; public static final int RUN_STATUS_ERROR = 2; - public static final int RUN_STATUS_FAILED_NONDETERMINISTIC = 3; + + // hegel_nondeterminism_strictness_t: how a run reacts on detecting a nondeterministic test. + public static final int NONDETERMINISM_QUIET = 0; + public static final int NONDETERMINISM_WARN = 1; + public static final int NONDETERMINISM_ERROR = 2; // Phases (bitmask for hegel_settings_set_phases). public static final int PHASE_EXPLICIT = 1 << 0; diff --git a/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Libhegel.java b/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Libhegel.java index c5492e7..0bf169b 100644 --- a/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Libhegel.java +++ b/hegel-lowlevel/src/main/java/dev/hegel/lowlevel/Libhegel.java @@ -97,7 +97,8 @@ static Libhegel load(Path library, Iterable backends) * (a {@code hegel.toml} in the working directory or an ancestor, {@code HEGEL_DEFAULT_PROFILE}, * and the shipped {@code development}/{@code ci}/{@code workload} profiles) with the {@code * HEGEL_TEST_CASES}, {@code HEGEL_DATABASE}, {@code HEGEL_STATISTICS}, {@code HEGEL_SEED}, - * {@code HEGEL_DERANDOMIZE} and {@code HEGEL_PRINT_BLOB} environment variables applied over it. + * {@code HEGEL_DERANDOMIZE}, {@code HEGEL_PRINT_BLOB} and {@code HEGEL_NONDETERMINISM_STRICTNESS} + * environment variables applied over it. * Returns {@link Abi#E_INVALID_ARG} (with the message in {@link #lastErrorMessage()}) when a * {@code hegel.toml} or one of those variables is malformed; {@code out[0]} receives the handle * on {@link Abi#OK} and {@code 0} otherwise. @@ -112,6 +113,13 @@ static Libhegel load(Path library, Iterable backends) /** {@code hegel_settings_get_print_blob}: whether the resolved settings print reproduce blobs. */ boolean settingsGetPrintBlob(long s); + /** + * {@code hegel_settings_get_nondeterminism_strictness}: the resolved reaction to a + * nondeterministic test ({@link Abi#NONDETERMINISM_QUIET}, {@link Abi#NONDETERMINISM_WARN} or + * {@link Abi#NONDETERMINISM_ERROR}). + */ + int settingsGetNondeterminismStrictness(long s); + void settingsBackend(long s, int backend); void settingsTestCases(long s, long n); @@ -138,6 +146,13 @@ static Libhegel load(Path library, Iterable backends) /** {@code hegel_settings_set_print_blob}: whether to print a reproduce blob per failure. */ void settingsPrintBlob(long s, boolean yes); + /** + * {@code hegel_settings_set_nondeterminism_strictness}: how the run reacts when it detects a + * test whose structure or outcome changes when the same choices are replayed — one of {@link + * Abi#NONDETERMINISM_QUIET}, {@link Abi#NONDETERMINISM_WARN} or {@link Abi#NONDETERMINISM_ERROR}. + */ + void settingsNondeterminismStrictness(long s, int strictness); + // Run lifecycle. /** @@ -147,6 +162,15 @@ static Libhegel load(Path library, Iterable backends) */ long runStart(long settings, Consumer output); + /** + * {@code hegel_run_start_blob}: start a run that replays a reproduce blob until a replay fails + * (under the engine's bounded budget) instead of exploring. Driven exactly like a run from + * {@link #runStart}: a reproducing replay is the run's failure, a run with no failures means + * the blob is stale, and an undecodable blob surfaces as the run's error from {@link + * #runResultError}. {@code output} has the same contract as in {@link #runStart}. + */ + long runStartBlob(long settings, String blob, Consumer output); + /** The next test-case handle, or {@code 0} once the run is finished. */ long nextTestCase(long run); @@ -158,15 +182,24 @@ static Libhegel load(Path library, Iterable backends) void runFree(long run); /** - * Replay a base64 reproduce blob as a standalone test case. Returns the raw rc ({@link - * Abi#E_INVALID_ARG} for a corrupt or incompatible blob); on OK, {@code out[0]} receives the - * caller-owned handle. {@code output} has the same contract as in {@link #runStart} but need not - * outlive the call. + * Replay a base64 reproduce blob as a standalone test case, in a single attempt. Returns the raw + * rc ({@link Abi#E_INVALID_ARG} for a corrupt or incompatible blob); on OK, {@code out[0]} + * receives the caller-owned handle. {@code output} has the same contract as in {@link #runStart} + * but need not outlive the call. A nondeterministic blob may need several attempts to fail, so + * reproduce-failure features should prefer {@link #runStartBlob}. */ int testCaseFromBlob(long settings, String blob, Consumer output, long[] out); void testCaseFree(long tc); + /** + * {@code hegel_test_case_should_capture}: whether the engine stamped this case for capture — the + * caller should keep the case's output and, if it fails, its exception, keyed by the failure's + * origin, because a stamped failing execution is the material for that origin's failure report. + * Read once at case start. + */ + boolean testCaseShouldCapture(long tc); + // Per-test-case draws. Each returns the raw rc. int generateBoolean(long tc, double p, boolean[] out); @@ -303,6 +336,13 @@ int newStateMachine( /** The origin string the shrinker grouped the {@code index}-th distinct failure under. */ String failureOrigin(long result, long index); + /** + * {@code hegel_failure_caveat}: the {@code index}-th distinct failure's confirmation caveat — its + * standing under the run's nondeterministic handling, quoting the run's replay evidence — or + * {@code null} for a deterministic failure. + */ + String failureCaveat(long result, long index); + // Diagnostics. String lastErrorMessage(); diff --git a/hegel/src/main/java/dev/hegel/RealLibhegel.java b/hegel/src/main/java/dev/hegel/RealLibhegel.java index 34c9722..c965c9f 100644 --- a/hegel/src/main/java/dev/hegel/RealLibhegel.java +++ b/hegel/src/main/java/dev/hegel/RealLibhegel.java @@ -112,13 +112,17 @@ private static MethodHandle findEmitLine() { private final MethodHandle settingsSetPrintBlob; private final MethodHandle settingsGetTestCases; private final MethodHandle settingsGetPrintBlob; + private final MethodHandle settingsSetNondeterminismStrictness; + private final MethodHandle settingsGetNondeterminismStrictness; private final MethodHandle runStart; + private final MethodHandle runStartBlob; private final MethodHandle nextTestCase; private final MethodHandle runResult; private final MethodHandle runResultFree; private final MethodHandle runFree; private final MethodHandle testCaseFromBlob; private final MethodHandle testCaseFree; + private final MethodHandle testCaseShouldCapture; private final MethodHandle generateBoolean; private final MethodHandle generateInteger; private final MethodHandle generateFloat; @@ -161,6 +165,7 @@ private static MethodHandle findEmitLine() { private final MethodHandle failureFree; private final MethodHandle failureReproductionBlob; private final MethodHandle failureOrigin; + private final MethodHandle failureCaveat; private final MethodHandle version; RealLibhegel(Path libraryPath) { @@ -244,11 +249,26 @@ private static MethodHandle findEmitLine() { lookup, "hegel_settings_get_print_blob", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); + this.settingsSetNondeterminismStrictness = h( + linker, + lookup, + "hegel_settings_set_nondeterminism_strictness", + FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, JAVA_INT)); + this.settingsGetNondeterminismStrictness = h( + linker, + lookup, + "hegel_settings_get_nondeterminism_strictness", + FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.runStart = h( linker, lookup, "hegel_run_start", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS)); + this.runStartBlob = h( + linker, + lookup, + "hegel_run_start_blob", + FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS)); this.nextTestCase = h(linker, lookup, "hegel_next_test_case", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.runResult = @@ -263,6 +283,11 @@ private static MethodHandle findEmitLine() { FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS, ADDRESS)); this.testCaseFree = h(linker, lookup, "hegel_test_case_free", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS)); + this.testCaseShouldCapture = h( + linker, + lookup, + "hegel_test_case_should_capture", + FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.generateBoolean = h( linker, lookup, @@ -443,6 +468,8 @@ private static MethodHandle findEmitLine() { FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.failureOrigin = h(linker, lookup, "hegel_failure_origin", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); + this.failureCaveat = + h(linker, lookup, "hegel_failure_caveat", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS, ADDRESS)); this.version = h(linker, lookup, "hegel_version", FunctionDescriptor.of(JAVA_INT, ADDRESS, ADDRESS)); this.context = ThreadLocal.withInitial(() -> (MemorySegment) invoke(contextNew)); } @@ -606,6 +633,20 @@ public boolean settingsGetPrintBlob(long s) { return out.get(JAVA_BOOLEAN, 0); } + @Override + public void settingsNondeterminismStrictness(long s, int strictness) { + check( + "hegel_settings_set_nondeterminism_strictness", + rc(settingsSetNondeterminismStrictness, segment(s), strictness)); + } + + @Override + public int settingsGetNondeterminismStrictness(long s) { + MemorySegment out = Arena.ofAuto().allocate(JAVA_INT); + check("hegel_settings_get_nondeterminism_strictness", rc(settingsGetNondeterminismStrictness, segment(s), out)); + return out.get(JAVA_INT, 0); + } + // --- run lifecycle --- @Override @@ -624,6 +665,24 @@ public long runStart(long settings, Consumer output) { return run; } + @Override + public long runStartBlob(long settings, String blob, Consumer output) { + Arena arena = Arena.ofConfined(); + MemorySegment callback = output == null ? MemorySegment.NULL : upcallStub(arena, output); + MemorySegment out = arena.allocate(ADDRESS); + int code = rc(runStartBlob, segment(settings), cstr(arena, blob), callback, MemorySegment.NULL, out); + if (code != Abi.OK) { + arena.close(); + throw new HegelException("hegel_run_start_blob failed (rc=" + + code + + "): " + + java.util.Objects.toString(lastErrorMessage(), "")); + } + long run = out.get(ADDRESS, 0).address(); + runArenas.put(run, arena); + return run; + } + @Override public long nextTestCase(long run) { MemorySegment out = Arena.ofAuto().allocate(ADDRESS); @@ -670,6 +729,13 @@ public void testCaseFree(long tc) { check("hegel_test_case_free", rc(testCaseFree, segment(tc))); } + @Override + public boolean testCaseShouldCapture(long tc) { + MemorySegment out = Arena.ofAuto().allocate(JAVA_BOOLEAN); + check("hegel_test_case_should_capture", rc(testCaseShouldCapture, segment(tc), out)); + return out.get(JAVA_BOOLEAN, 0); + } + // --- draws --- @Override @@ -1178,6 +1244,11 @@ public String failureOrigin(long result, long index) { return readFailureString(result, index, failureOrigin, "hegel_failure_origin"); } + @Override + public String failureCaveat(long result, long index) { + return readFailureString(result, index, failureCaveat, "hegel_failure_caveat"); + } + /** Fetch the {@code index}-th failure, read one of its strings with {@code reader}, free it. */ private String readFailureString(long result, long index, MethodHandle reader, String op) { MemorySegment failureOut = Arena.ofAuto().allocate(ADDRESS); diff --git a/hegel/src/test/java/dev/hegel/RealLibhegelTest.java b/hegel/src/test/java/dev/hegel/RealLibhegelTest.java index f6f4bc6..5a07383 100644 --- a/hegel/src/test/java/dev/hegel/RealLibhegelTest.java +++ b/hegel/src/test/java/dev/hegel/RealLibhegelTest.java @@ -3,6 +3,7 @@ import static org.junit.jupiter.api.Assertions.assertArrayEquals; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotEquals; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertThrows; @@ -80,6 +81,8 @@ void infrastructureCallsReportNullHandles() { // diagnostic rather than undefined behaviour. assertThrows(HegelException.class, () -> lib.runResultStatus(0)); assertThrows(HegelException.class, () -> lib.runStart(0, null)); + assertThrows(HegelException.class, () -> lib.runStartBlob(0, "blob", null)); + assertThrows(HegelException.class, () -> lib.testCaseShouldCapture(0)); } @Test @@ -193,6 +196,10 @@ void settingsGettersReadTheHandleBack() { assertFalse(lib.settingsGetPrintBlob(s)); lib.settingsPrintBlob(s, true); assertTrue(lib.settingsGetPrintBlob(s)); + lib.settingsNondeterminismStrictness(s, Abi.NONDETERMINISM_ERROR); + assertEquals(Abi.NONDETERMINISM_ERROR, lib.settingsGetNondeterminismStrictness(s)); + lib.settingsNondeterminismStrictness(s, Abi.NONDETERMINISM_WARN); + assertEquals(Abi.NONDETERMINISM_WARN, lib.settingsGetNondeterminismStrictness(s)); lib.settingsFree(s); } @@ -212,6 +219,23 @@ void undecodableBlobWithDefaultOutputIsRejected() { lib.settingsFree(s); } + @Test + void blobRunWithDefaultOutputStartsAndReportsTheBadBlob() { + RealLibhegel lib = real(); + long s = newSettings(lib); + // A null output callback leaves the run's output on stderr; the garbage blob surfaces as + // the run's error once it is pumped dry. + long run = lib.runStartBlob(s, "not-a-blob!!!", null); + assertNotEquals(0, run); + assertEquals(0, lib.nextTestCase(run)); + long result = lib.runResult(run); + assertEquals(Abi.RUN_STATUS_ERROR, lib.runResultStatus(result)); + assertNotNull(lib.runResultError(result)); + lib.runResultFree(result); + lib.runFree(run); + lib.settingsFree(s); + } + @Test void regexGeneratorAcceptsATextAlphabet() { RealLibhegel lib = real(); diff --git a/pom.xml b/pom.xml index ae0a165..dad1f7d 100644 --- a/pom.xml +++ b/pom.xml @@ -56,7 +56,7 @@ - 0.43.4 + 0.44.0 false diff --git a/shared/src/main/java/dev/hegel/Failure.java b/shared/src/main/java/dev/hegel/Failure.java index 9324c49..2b60326 100644 --- a/shared/src/main/java/dev/hegel/Failure.java +++ b/shared/src/main/java/dev/hegel/Failure.java @@ -8,24 +8,34 @@ import java.util.Optional; /** - * One distinct counterexample of a failed run, as observed on its final replay. + * One distinct counterexample of a failed run, as observed on the freshest failing execution the + * engine stamped for capture. * - *

After the engine shrinks a failure it hands back a reproduce blob; the runner replays that blob - * once more with reporting on, capturing the exception the body threw and every top-level draw and - * note along the way. That capture is what this class carries. A replay that no longer fails is - * {@linkplain #flaky() flaky}: the test's outcome depends on something other than its generated - * data. + *

The engine runs every failure it is about to report one final time, stamped for capture ({@link + * TestCase#isFinal()}); the runner keeps that execution's exception and every top-level draw and + * note, and that capture is what this class carries. A {@linkplain #nondeterministic() + * nondeterministic} failure — one whose test does not fail every time the same choices are + * replayed — additionally carries the engine's {@linkplain #caveat() caveat} quoting how reliably it + * reproduced, and has a reproduce blob only if the engine confirmed it. */ public final class Failure { private final String origin; private final String reproduceBlob; + private final String caveat; private final Throwable exception; private final Map draws; private final List notes; - Failure(String origin, String reproduceBlob, Throwable exception, Map draws, List notes) { + Failure( + String origin, + String reproduceBlob, + String caveat, + Throwable exception, + Map draws, + List notes) { this.origin = origin; this.reproduceBlob = reproduceBlob; + this.caveat = caveat; this.exception = exception; this.draws = Collections.unmodifiableMap(new LinkedHashMap<>(draws)); this.notes = Collections.unmodifiableList(new ArrayList<>(notes)); @@ -43,41 +53,58 @@ public String origin() { } /** - * The base64 blob that replays this counterexample exactly, via {@link + * The base64 blob that replays this counterexample, via {@link * Settings#reproduceFailure(String)}. Only guaranteed to reproduce under the Hegel version that * produced it. * - * @return the reproduce blob + * @return the reproduce blob, or empty when the engine produced none: for an unconfirmed + * {@linkplain #nondeterministic() nondeterministic} failure, and for a failure reproduced + * from a {@link Settings#reproduceFailure(String)} blob (the caller already holds it) */ - public String reproduceBlob() { - return reproduceBlob; + public Optional reproduceBlob() { + return Optional.ofNullable(reproduceBlob); } /** - * The exception the test body threw on the final replay. + * The engine's confirmation caveat for a nondeterministic failure: its standing under the run's + * nondeterministic handling, quoting the run's own replay evidence (for example {@code + * nondeterministic failure, confirmed: failed 7 of 20 replays at confirmation and 1 of 2 at + * report time}). Print it alongside the failure so the reader sees how reliably it reproduced. * - * @return the exception, or empty if the replay unexpectedly passed (see {@link #flaky()}) + * @return the caveat, or empty for a deterministic failure */ - public Optional exception() { - return Optional.ofNullable(exception); + public Optional caveat() { + return Optional.ofNullable(caveat); } /** - * Whether the final replay failed to reproduce the failure. The engine only reports a - * counterexample after re-running it, so a passing replay means the body's outcome depends on - * state outside its generated data (globals, time, an external RNG). + * Whether the test's outcome depended on something other than its generated data: the same + * choices did not fail every time the engine replayed them (hidden global state, time, an + * external RNG, thread scheduling). The engine confirms such a failure by repeated replay + * before shrinking it; {@link #caveat()} says how that went. * - * @return {@code true} if the replay passed + * @return {@code true} if the engine handled this failure as nondeterministic */ - public boolean flaky() { - return exception == null; + public boolean nondeterministic() { + return caveat != null; } /** - * Every top-level draw of the final replay, in draw order, keyed by the label passed to {@link - * TestCase#draw(Generator, String)} — numbered from its second use in the case ({@code x}, - * {@code x_2}, ...) so repeated draws are all kept — or {@code draw_N} for the N-th unlabelled - * draw. + * The exception the test body threw on the captured execution. + * + * @return the exception + */ + public Throwable exception() { + return exception; + } + + /** + * Every top-level draw of the captured execution, in draw order, keyed by the label passed to + * {@link TestCase#draw(Generator, String)} — numbered from its second use in the case ({@code + * x}, {@code x_2}, ...) so repeated draws are all kept — or {@code draw_N} for the N-th + * unlabelled draw. Empty when the engine reported the failure without a stamped failing + * execution to capture — an unconfirmed nondeterministic failure that never failed again after + * its discovery. * * @return an unmodifiable, insertion-ordered map of label to generated value */ @@ -86,7 +113,7 @@ public Map draws() { } /** - * Every {@link TestCase#note(String) note} recorded during the final replay, in order. + * Every {@link TestCase#note(String) note} recorded during the captured execution, in order. * * @return an unmodifiable list of notes */ diff --git a/shared/src/main/java/dev/hegel/HegelTest.java b/shared/src/main/java/dev/hegel/HegelTest.java index ad873ab..7513f00 100644 --- a/shared/src/main/java/dev/hegel/HegelTest.java +++ b/shared/src/main/java/dev/hegel/HegelTest.java @@ -133,6 +133,15 @@ */ OptBoolean printBlob() default OptBoolean.DEFAULT; + /** + * How the run reacts when it detects a nondeterministic test. The default leaves it to the + * engine's settings profile and the {@code HEGEL_NONDETERMINISM_STRICTNESS} environment variable + * ({@link NondeterminismStrictness#QUIET} unless configured otherwise). + * + * @return the reaction to nondeterminism + */ + NondeterminismStrictness nondeterminismStrictness() default NondeterminismStrictness.DEFAULT; + /** * Replay a stored failure blob (printed by {@link #printBlob}) instead of running the property: * the test body is re-run against exactly the choices the blob encodes, bypassing generation diff --git a/shared/src/main/java/dev/hegel/HegelTestExtension.java b/shared/src/main/java/dev/hegel/HegelTestExtension.java index 664e347..9b5bd56 100644 --- a/shared/src/main/java/dev/hegel/HegelTestExtension.java +++ b/shared/src/main/java/dev/hegel/HegelTestExtension.java @@ -85,6 +85,7 @@ static Settings settingsFrom(HegelTest ann, String methodName) { Settings s = new Settings() .verbosity(ann.verbosity()) .backend(ann.backend()) + .nondeterminismStrictness(ann.nondeterminismStrictness()) .reportMultipleFailures(ann.reportMultipleFailures()) .suppressHealthCheck(ann.suppressHealthCheck()) .name(name); diff --git a/shared/src/main/java/dev/hegel/NondeterminismStrictness.java b/shared/src/main/java/dev/hegel/NondeterminismStrictness.java new file mode 100644 index 0000000..0a8723a --- /dev/null +++ b/shared/src/main/java/dev/hegel/NondeterminismStrictness.java @@ -0,0 +1,45 @@ +package dev.hegel; + +import dev.hegel.lowlevel.Abi; + +/** + * How a run reacts when it detects a nondeterministic test: one whose structure or outcome changes + * when the same generated choices are replayed, because it depends on something other than its + * generated data (hidden global state, time, an outside service, thread scheduling). + * + *

Under {@link #QUIET} and {@link #WARN} the engine switches to nondeterministic handling: a + * failure is confirmed by repeated replay before it is shrunk or persisted, and reported with a + * {@linkplain Failure#caveat() caveat} quoting the run's replay evidence. An unconfirmed failure + * still fails the run, with a caveat instead of a reproduce blob. {@link #ERROR} aborts the run + * with a flaky-test error instead, for suites that use determinism as a lint. + */ +public enum NondeterminismStrictness { + /** + * Leave the choice to the engine's settings profile and the {@code HEGEL_NONDETERMINISM_STRICTNESS} + * environment variable ({@code quiet} unless configured otherwise). + */ + DEFAULT(null), + /** Switch to nondeterministic handling silently. The engine's default. */ + QUIET(Abi.NONDETERMINISM_QUIET), + /** Switch to nondeterministic handling, printing a one-line notice once per run. */ + WARN(Abi.NONDETERMINISM_WARN), + /** Abort the run with a flaky-test error ({@link HegelException}). */ + ERROR(Abi.NONDETERMINISM_ERROR); + + /** The {@code hegel_nondeterminism_strictness_t} value to send, or {@code null} to leave the profile's choice. */ + final Integer code; + + NondeterminismStrictness(Integer code) { + this.code = code; + } + + /** The strictness the engine reports back for {@code code}. */ + static NondeterminismStrictness fromCode(int code) { + for (NondeterminismStrictness s : values()) { + if (s.code != null && s.code == code) { + return s; + } + } + throw new HegelException("unknown hegel_nondeterminism_strictness_t value " + code); + } +} diff --git a/shared/src/main/java/dev/hegel/PrintingReporter.java b/shared/src/main/java/dev/hegel/PrintingReporter.java index 3d4cda7..53f1a70 100644 --- a/shared/src/main/java/dev/hegel/PrintingReporter.java +++ b/shared/src/main/java/dev/hegel/PrintingReporter.java @@ -59,10 +59,15 @@ public void failuresFound(int count) { @Override public void failure(Failure failure) { - if (printBlob && !failure.flaky()) { - out.println(); - out.println("To reproduce this failure, replay it with:"); - out.println(" @HegelTest(reproduceFailure = \"" + failure.reproduceBlob() + "\")"); + if (!quiet) { + failure.caveat().ifPresent(caveat -> out.println("note: " + caveat)); + } + if (printBlob) { + failure.reproduceBlob().ifPresent(blob -> { + out.println(); + out.println("To reproduce this failure, replay it with:"); + out.println(" @HegelTest(reproduceFailure = \"" + blob + "\")"); + }); } } } diff --git a/shared/src/main/java/dev/hegel/Reporter.java b/shared/src/main/java/dev/hegel/Reporter.java index b1e0793..d542c7d 100644 --- a/shared/src/main/java/dev/hegel/Reporter.java +++ b/shared/src/main/java/dev/hegel/Reporter.java @@ -15,13 +15,14 @@ * *

Every method has a no-op default, so an implementation overrides only what it needs. Callbacks * are invoked on the thread that drives the run, in this order: {@link #runStarted}, then for each - * generated case {@link #caseStarted} / {@link #caseFinished} (with {@link #engineOutput} lines - * interleaved as the engine emits them, and the case's {@link #draw}s and {@link #note}s in between - * when the run is {@linkplain Verbosity#VERBOSE verbose}); on a failed run {@link #failuresFound}, then for each - * distinct counterexample a final replay ({@link #caseStarted} with {@code finalReplay = true}, its - * {@link #draw}s and {@link #note}s, {@link #caseFinished}) followed by {@link #failure}; and - * finally {@link #runFinished}. A reporter is per run: two concurrent runs never share one unless - * the caller passes the same instance to both. + * case the engine runs {@link #caseStarted} / {@link #caseFinished} (with {@link #engineOutput} + * lines interleaved as the engine emits them, and the case's {@link #draw}s and {@link #note}s in + * between when the run is {@linkplain Verbosity#VERBOSE verbose}); on a failed run {@link + * #failuresFound}, then for each distinct counterexample the replay of its captured report ({@link + * #caseStarted} with {@code finalReplay = true}, the {@link #draw}s and {@link #note}s recorded on + * the failing execution the engine stamped for capture, {@link #caseFinished}) followed by {@link + * #failure}; and finally {@link #runFinished}. A reporter is per run: two concurrent runs never + * share one unless the caller passes the same instance to both. */ public interface Reporter { /** @@ -40,17 +41,19 @@ default void runStarted(Settings settings) {} default void engineOutput(String line) {} /** - * The test body is about to run against a case. + * The test body is about to run against a case ({@code finalReplay = false}), or the runner is + * about to replay the captured draws and notes of a counterexample it reports ({@code + * finalReplay = true}; no body runs). * - * @param finalReplay {@code true} for the replay of a minimal counterexample (or of a {@link - * Settings#reproduceFailure} blob), {@code false} during generation and shrinking + * @param finalReplay {@code true} for the replay of a reported counterexample's capture, {@code + * false} for a live case */ default void caseStarted(boolean finalReplay) {} /** - * A top-level {@link TestCase#draw(Generator, String) draw} completed. Called on final replays, - * and on every case when the run's {@link Verbosity} is {@code VERBOSE} or higher; never for - * draws nested inside another generator. + * A top-level {@link TestCase#draw(Generator, String) draw} completed. Called when a reported + * counterexample's capture is replayed, and live on every case when the run's {@link Verbosity} + * is {@code VERBOSE} or higher; never for draws nested inside another generator. * * @param label the label passed to {@code draw} (numbered from its second use in a case: {@code * x}, {@code x_2}, ...), or {@code draw_N} for the N-th unlabelled draw @@ -69,7 +72,9 @@ default void draw(String label, Object value, boolean finalReplay) {} default void note(String message, boolean finalReplay) {} /** - * The test body finished against a case and the outcome was reported to the engine. + * The test body finished against a case and the outcome was reported to the engine, or the + * replay of a reported counterexample's capture is complete (always {@link + * CaseOutcome#INTERESTING}). * * @param outcome how the case concluded * @param finalReplay as in {@link #caseStarted} @@ -77,16 +82,17 @@ default void note(String message, boolean finalReplay) {} default void caseFinished(CaseOutcome outcome, boolean finalReplay) {} /** - * The run's verdict is FAILED. Called once, before the counterexamples are replayed. + * The run's verdict is FAILED. Called once, before the counterexamples are reported. * - * @param count the number of distinct failures (by origin) that will be replayed + * @param count the number of distinct failures (by origin) that will be reported */ default void failuresFound(int count) {} /** - * A counterexample's final replay finished. + * A counterexample's captured report was replayed. * - * @param failure the failure, carrying the replay's exception, draws, notes, and blob + * @param failure the failure, carrying the captured exception, draws and notes, the engine's + * caveat for a nondeterministic failure, and the reproduce blob if there is one */ default void failure(Failure failure) {} @@ -99,10 +105,11 @@ default void runFinished(RunReport report) {} /** * The classic printed report: engine output, each reported draw as {@code label = value;}, - * notes, a header when several distinct failures are reported, and a copy-pasteable reproducer - * when {@link Settings#printBlob(boolean)} is on. Honours {@link Settings#verbosity}: {@link - * Verbosity#QUIET} prints no draws or notes at all, {@link Verbosity#VERBOSE} prints them for - * every case. + * notes, a header when several distinct failures are reported, a {@code note:} line with the + * engine's caveat for a nondeterministic failure, and a copy-pasteable reproducer when {@link + * Settings#printBlob(boolean)} is on and the failure has a blob. Honours {@link + * Settings#verbosity}: {@link Verbosity#QUIET} prints no draws, notes or caveats at all, {@link + * Verbosity#VERBOSE} prints draws and notes for every case. * * @param out where to print * @return a printing reporter diff --git a/shared/src/main/java/dev/hegel/RunReport.java b/shared/src/main/java/dev/hegel/RunReport.java index f3d04d4..8f53985 100644 --- a/shared/src/main/java/dev/hegel/RunReport.java +++ b/shared/src/main/java/dev/hegel/RunReport.java @@ -85,8 +85,7 @@ public List failures() { /** * Turn a non-passing report into the exception {@link Hegel#test} throws: nothing for a passed * run; {@link HealthCheckFailure} or {@link HegelException} (with the engine's message) for an - * errored run; for a failed run, a {@link HegelException} if any replay was {@linkplain - * Failure#flaky() flaky}, otherwise the single failure's own exception rethrown as-is, or an + * errored run; for a failed run, the single failure's own exception rethrown as-is, or an * {@link AssertionError} aggregating several distinct failures (carrying the originals as * suppressed exceptions). */ @@ -100,22 +99,17 @@ public void throwIfFailed() { } throw new HegelException(error); } - for (Failure failure : failures) { - if (failure.flaky()) { - throw new HegelException(Runner.FLAKY_DIAGNOSTIC); - } - } if (failures.size() == 1) { - throw unchecked(failures.get(0).exception().get()); + throw unchecked(failures.get(0).exception()); } StringBuilder sb = new StringBuilder(); sb.append("Hegel found ").append(failures.size()).append(" distinct failing examples:"); for (Failure failure : failures) { - sb.append("\n\n").append(describe(failure.exception().get())); + sb.append("\n\n").append(describe(failure.exception())); } AssertionError aggregate = new AssertionError(sb.toString()); for (Failure failure : failures) { - aggregate.addSuppressed(failure.exception().get()); + aggregate.addSuppressed(failure.exception()); } throw aggregate; } diff --git a/shared/src/main/java/dev/hegel/Runner.java b/shared/src/main/java/dev/hegel/Runner.java index 3dbd7ca..4dba0f8 100644 --- a/shared/src/main/java/dev/hegel/Runner.java +++ b/shared/src/main/java/dev/hegel/Runner.java @@ -4,19 +4,26 @@ import dev.hegel.lowlevel.Libhegel; import dev.hegel.lowlevel.LibhegelException; import java.util.ArrayList; +import java.util.HashMap; import java.util.List; +import java.util.Map; import java.util.function.Consumer; /** - * Drives a single property test: builds the settings handle, pumps the engine's exploration loop, - * and turns the aggregated result into a {@link RunReport}. + * Drives a single property test: builds the settings handle, pumps the engine's run loop, and turns + * the aggregated result into a {@link RunReport}. * - *

The engine only explores — generation and shrinking — so every pumped case is non-final. The - * client owns the final replays: once the loop drains, each discovered counterexample's reproduce - * blob is read off the run result and replayed via {@code hegel_test_case_from_blob} with reporting - * enabled, which surfaces the minimal example's draws and re-raises the test body's own exception. - * A counterexample whose replay does not fail again is a flaky test. Run-level errors (a failed - * health check, nondeterminism, an engine panic) surface with the engine's own message. + *

The engine owns the whole run — generation, shrinking, the replays that confirm a failure, and + * the final replay of every failure it is about to report — so the loop only pumps cases and keeps + * the report material as they run. The engine stamps the executions a failure report can be built + * from ({@code hegel_test_case_should_capture}); a stamped case records its draws and notes, and + * every interesting case's exception is kept per origin, a stamped capture winning over an unstamped + * one and the newest winning at equal rank. Once the loop drains, each reported failure is built + * from its origin's capture: the draws and notes are replayed to the reporter, the exception is the + * body's own, and the engine's caveat and reproduce blob are attached. A {@link + * Settings#reproduceFailure} run is the same loop over {@code hegel_run_start_blob}. Run-level errors + * (a failed health check, a nondeterminism abort under {@link NondeterminismStrictness#ERROR}, an + * engine panic) surface with the engine's own message. * *

Everything the run has to say goes through its {@link Reporter}; the runner itself never * prints. Binding and engine errors ({@link HegelException}) are thrown, not reported: they are bugs @@ -34,24 +41,31 @@ private Runner() {} "dev.hegel.", "org.junit.", "org.opentest4j.", "jdk.", "java.", "sun.", "com.sun." }; + /** Message for a {@link Settings#reproduceFailure} blob none of whose replays failed. */ + static final String STALE_BLOB = "reproduceFailure: the supplied failure blob did not reproduce a failure." + + " The failure may have been fixed — or, for a nondeterministic blob, it may not have" + + " recurred within the replay budget. Re-run to retry, or remove the blob once the failure" + + " is fixed."; + /** - * Message for a test whose outcome changed when re-run with the same generated data. After the - * engine shrinks and verifies a counterexample, the runner replays its blob one final time; if - * that replay does not fail, the test is non-deterministic. + * The body's outcome against one case: the case (holding its draws and notes when the engine + * stamped it for capture), the exception that made it interesting ({@code null} otherwise), and + * the origin the case was marked complete with. */ - static final String FLAKY_DIAGNOSTIC = "Flaky test detected: Your test produced different outcomes" - + " when run with the same generated data — it failed when it previously succeeded, or" - + " succeeded when it previously failed. This usually means your test depends on external" - + " state such as global variables, system time, or external random number generators."; - - /** The body's outcome against one case: the case (holding its draws and notes) and its failure. */ private static final class CaseRun { final TestCase testCase; final Throwable interesting; + final String origin; - CaseRun(TestCase testCase, Throwable interesting) { + CaseRun(TestCase testCase, Throwable interesting, String origin) { this.testCase = testCase; this.interesting = interesting; + this.origin = origin; + } + + /** A stamped capture outranks an unstamped one, whose only material is the exception. */ + int rank() { + return testCase.isFinal() ? 1 : 0; } } @@ -63,12 +77,18 @@ static RunReport run(Libhegel lib, Settings settings, Consumer body, R applySettings(lib, s, settings); // The engine resolved whatever the caller left unset (its profile, hegel.toml, the // HEGEL_* variables); the run and its reporters see the effective configuration. - Settings effective = settings.resolved(lib.settingsGetTestCases(s), lib.settingsGetPrintBlob(s)); + Settings effective = settings.resolved( + lib.settingsGetTestCases(s), + lib.settingsGetPrintBlob(s), + NondeterminismStrictness.fromCode(lib.settingsGetNondeterminismStrictness(s))); reporter.runStarted(effective); - if (effective.reproduceFailure != null) { - report = replayBlob(lib, s, effective, body, reporter, counts); - } else { - report = explore(lib, s, effective, body, reporter, counts); + long run = effective.reproduceFailure == null + ? lib.runStart(s, reporter::engineOutput) + : lib.runStartBlob(s, effective.reproduceFailure, reporter::engineOutput); + try { + report = drive(lib, run, effective, body, reporter, counts); + } finally { + lib.runFree(run); } } finally { lib.settingsFree(s); @@ -96,149 +116,115 @@ private static long newSettings(Libhegel lib) { return out[0]; } - /** Pump the engine's exploration loop to completion and translate its verdict. */ - private static RunReport explore( + /** + * Pump the run to completion, keeping every interesting case's capture per origin, and + * translate its verdict. + */ + private static RunReport drive( Libhegel lib, - long s, + long run, Settings settings, Consumer body, Reporter reporter, RunStatistics.Counter counts) { - long run = lib.runStart(s, reporter::engineOutput); - try { - while (true) { - long tc = lib.nextTestCase(run); - if (isNull(tc)) { - break; - } - driveOneCase(lib, tc, false, body, settings, reporter, counts); + Map captures = new HashMap<>(); + while (true) { + long tc = lib.nextTestCase(run); + if (isNull(tc)) { + break; } - long result = lib.runResult(run); - try { - return finish(lib, s, result, settings, body, reporter, counts); - } finally { - lib.runResultFree(result); + CaseRun caseRun = driveOneCase(lib, tc, body, settings, reporter, counts); + if (caseRun.interesting != null) { + CaseRun stored = captures.get(caseRun.origin); + if (stored == null || caseRun.rank() >= stored.rank()) { + captures.put(caseRun.origin, caseRun); + } } + } + long result = lib.runResult(run); + try { + return finish(lib, result, settings, reporter, counts, captures); } finally { - lib.runFree(run); + lib.runResultFree(result); } } /** Translate a drained run's result into a report. */ private static RunReport finish( Libhegel lib, - long s, long result, Settings settings, - Consumer body, Reporter reporter, - RunStatistics.Counter counts) { + RunStatistics.Counter counts, + Map captures) { + boolean replayingBlob = settings.reproduceFailure != null; switch (lib.runResultStatus(result)) { case Abi.RUN_STATUS_PASSED: + if (replayingBlob) { + throw new HegelException(STALE_BLOB); + } return new RunReport(RunStatus.PASSED, counts.snapshot(), null, List.of()); case Abi.RUN_STATUS_ERROR: // The run produced no verdict on the property: a failed health check, a - // nondeterminism mismatch, or an engine panic. - return new RunReport( - RunStatus.ERROR, counts.snapshot(), nullToEmpty(lib.runResultError(result)), List.of()); - case Abi.RUN_STATUS_FAILED_NONDETERMINISTIC: - // Only a state machine created with max_concurrency > 1 declares a run - // nondeterministic, and this binding drives every machine sequentially, so the - // engine should never report this: its failures carry no reproduce blob to replay. - throw new HegelException("internal error: the engine reported a failure on a nondeterministic run," - + " but this binding never declares a run nondeterministic"); + // nondeterminism abort under ERROR strictness, an engine panic — or, on a blob + // replay, a blob the engine could not decode. + String error = nullToEmpty(lib.runResultError(result)); + if (replayingBlob) { + throw new HegelException("reproduceFailure: the supplied blob is not valid: " + error); + } + return new RunReport(RunStatus.ERROR, counts.snapshot(), error, List.of()); default: - return replayFailures(lib, s, result, settings, body, reporter, counts); + return reportFailures(lib, result, reporter, counts, captures); } } - /** Replay every distinct counterexample's blob, capturing its draws, notes, and exception. */ - private static RunReport replayFailures( - Libhegel lib, - long s, - long result, - Settings settings, - Consumer body, - Reporter reporter, - RunStatistics.Counter counts) { + /** Build every distinct failure from its origin's capture and hand it to the reporter. */ + private static RunReport reportFailures( + Libhegel lib, long result, Reporter reporter, RunStatistics.Counter counts, Map captures) { long count = lib.runResultFailureCount(result); reporter.failuresFound((int) count); List failures = new ArrayList<>(); for (int i = 0; i < count; i++) { - String blob = lib.failureBlob(result, i); - if (blob == null) { - throw new HegelException("internal error: failure " + i + " carries no reproduce blob"); - } String origin = lib.failureOrigin(result, i); - long[] tcOut = new long[1]; - int rc = lib.testCaseFromBlob(s, blob, reporter::engineOutput, tcOut); - if (rc != Abi.OK) { + CaseRun capture = captures.remove(origin); + if (capture == null) { throw new HegelException( - "hegel_test_case_from_blob failed (rc=" + rc + "): " + nullToEmpty(lib.lastErrorMessage())); + "internal error: failure " + i + " (" + origin + ") has no captured failing execution"); } - CaseRun replay = driveOneCase(lib, tcOut[0], true, body, settings, reporter, counts); - Failure failure = - new Failure(origin, blob, replay.interesting, replay.testCase.draws(), replay.testCase.notes()); + reporter.caseStarted(true); + capture.testCase.replayTo(reporter); + reporter.caseFinished(CaseOutcome.INTERESTING, true); + Failure failure = new Failure( + origin, + lib.failureBlob(result, i), + lib.failureCaveat(result, i), + capture.interesting, + capture.testCase.draws(), + capture.testCase.notes()); reporter.failure(failure); failures.add(failure); } return new RunReport(RunStatus.FAILED, counts.snapshot(), null, failures); } - /** - * Replay a single stored blob ({@link Settings#reproduceFailure}), bypassing generation and - * shrinking: a reproduced failure is reported as the run's one failure, and a blob that no - * longer fails is a configuration error (the failure was fixed, or the blob is stale). - */ - private static RunReport replayBlob( - Libhegel lib, - long s, - Settings settings, - Consumer body, - Reporter reporter, - RunStatistics.Counter counts) { - String blob = settings.reproduceFailure; - long[] tcOut = new long[1]; - int rc = lib.testCaseFromBlob(s, blob, reporter::engineOutput, tcOut); - if (rc != Abi.OK) { - throw new HegelException("reproduceFailure: the supplied blob is not valid (rc=" - + rc - + "): " - + nullToEmpty(lib.lastErrorMessage())); - } - CaseRun replay = driveOneCase(lib, tcOut[0], true, body, settings, reporter, counts); - if (replay.interesting == null) { - throw new HegelException("reproduceFailure: the supplied failure blob no longer reproduces a" - + " failure. The failure may have been fixed, or the blob is stale."); - } - Failure failure = new Failure( - originOf(replay.interesting, settings.infrastructurePackages), - blob, - replay.interesting, - replay.testCase.draws(), - replay.testCase.notes()); - reporter.failure(failure); - return new RunReport(RunStatus.FAILED, counts.snapshot(), null, List.of(failure)); - } - /** * Run the body once against {@code tc}, report the outcome to the engine and the reporter, and * free the handle. The returned {@link CaseRun} carries the exception that made the case * interesting ({@code null} for any other outcome) and the case itself, whose draws and notes - * were recorded when {@code finalReplay} is set. + * were recorded if the engine stamped it for capture. */ private static CaseRun driveOneCase( Libhegel lib, long tc, - boolean finalReplay, Consumer body, Settings settings, Reporter reporter, RunStatistics.Counter counts) { try { boolean verbose = settings.verbosity.code >= Verbosity.VERBOSE.code; - TestCase testCase = new TestCase(new LiveDataSource(lib, tc), finalReplay, verbose, reporter); - reporter.caseStarted(finalReplay); + TestCase testCase = + new TestCase(new LiveDataSource(lib, tc), lib.testCaseShouldCapture(tc), verbose, reporter); + reporter.caseStarted(false); CaseOutcome outcome; String origin = null; Throwable interesting = null; @@ -263,8 +249,8 @@ private static CaseRun driveOneCase( "hegel_mark_complete failed (rc=" + rc + "): " + nullToEmpty(lib.lastErrorMessage())); } counts.record(outcome); - reporter.caseFinished(outcome, finalReplay); - return new CaseRun(testCase, interesting); + reporter.caseFinished(outcome, false); + return new CaseRun(testCase, interesting, origin); } finally { // The handle is caller-owned. On the error paths above the case may be incomplete; // the run still holds its own reference and completes it when freed. @@ -293,6 +279,9 @@ static void applySettings(Libhegel lib, long s, Settings st) { if (st.printBlob != null) { lib.settingsPrintBlob(s, st.printBlob); } + if (st.nondeterminismStrictness.code != null) { + lib.settingsNondeterminismStrictness(s, st.nondeterminismStrictness.code); + } if (st.backend.code != null) { lib.settingsBackend(s, st.backend.code); } diff --git a/shared/src/main/java/dev/hegel/Settings.java b/shared/src/main/java/dev/hegel/Settings.java index d8a141c..4c908e7 100644 --- a/shared/src/main/java/dev/hegel/Settings.java +++ b/shared/src/main/java/dev/hegel/Settings.java @@ -16,9 +16,9 @@ *

Whatever is not set here is resolved by the engine: from its settings profile — a {@code * hegel.toml} in the working directory or one of its parents (or the file named by {@code * HEGEL_CONFIG}), with {@code HEGEL_DEFAULT_PROFILE} selecting the profile in effect — and from the - * {@code HEGEL_TEST_CASES}, {@code HEGEL_DATABASE}, {@code HEGEL_SEED}, {@code HEGEL_DERANDOMIZE} and - * {@code HEGEL_PRINT_BLOB} environment variables, which win over the profile. Settings given here win - * over both. The shipped profiles run 100 test cases, and in CI (detected via {@code CI}/{@code + * {@code HEGEL_TEST_CASES}, {@code HEGEL_DATABASE}, {@code HEGEL_SEED}, {@code HEGEL_DERANDOMIZE}, + * {@code HEGEL_PRINT_BLOB} and {@code HEGEL_NONDETERMINISM_STRICTNESS} environment variables, which + * win over the profile. Settings given here win over both. The shipped profiles run 100 test cases, and in CI (detected via {@code CI}/{@code * GITHUB_ACTIONS}/... environment variables) the {@code ci} profile applies: runs are deterministic * ({@code derandomize}) and the example database is disabled. */ @@ -36,6 +36,7 @@ public final class Settings { // traces than an aggregated report — and that matters more in Java than elsewhere. final boolean reportMultipleFailures; final Boolean printBlob; // null = leave the engine's profile / HEGEL_PRINT_BLOB value + final NondeterminismStrictness nondeterminismStrictness; final String reproduceFailure; // null = run normally instead of replaying a blob final String name; final List infrastructurePackages; @@ -61,6 +62,7 @@ private Settings(Builder b) { this.backend = b.backend; this.reportMultipleFailures = b.reportMultipleFailures; this.printBlob = b.printBlob; + this.nondeterminismStrictness = b.nondeterminismStrictness; this.reproduceFailure = b.reproduceFailure; this.name = b.name; this.infrastructurePackages = b.infrastructurePackages; @@ -80,6 +82,7 @@ private Settings with(Consumer mutator) { b.backend = backend; b.reportMultipleFailures = reportMultipleFailures; b.printBlob = printBlob; + b.nondeterminismStrictness = nondeterminismStrictness; b.reproduceFailure = reproduceFailure; b.name = name; b.infrastructurePackages = infrastructurePackages; @@ -100,6 +103,7 @@ private static final class Builder { Backend backend = Backend.AUTO; boolean reportMultipleFailures = false; Boolean printBlob = null; + NondeterminismStrictness nondeterminismStrictness = NondeterminismStrictness.DEFAULT; String reproduceFailure = null; String name = null; List infrastructurePackages = List.of(); @@ -232,12 +236,27 @@ public Settings printBlob(boolean yes) { return with(b -> b.printBlob = yes); } + /** + * How the run reacts when it detects a nondeterministic test. Unset ({@link + * NondeterminismStrictness#DEFAULT}), the engine's profile or the {@code + * HEGEL_NONDETERMINISM_STRICTNESS} environment variable decides ({@link + * NondeterminismStrictness#QUIET} in the shipped profiles). + * + * @param strictness the reaction + * @return a new settings instance + */ + public Settings nondeterminismStrictness(NondeterminismStrictness strictness) { + return with(b -> b.nondeterminismStrictness = strictness); + } + /** * Replay a single stored failure instead of running the property: the blob (from {@link - * #printBlob(boolean)} output) is decoded and the test body re-run against exactly the choices - * it encodes, bypassing generation and shrinking. The run fails with the reproduced failure, or - * reports a stale blob if it no longer fails. A blob is only guaranteed to reproduce under the - * Hegel version that produced it. + * #printBlob(boolean)} output) is decoded and the test body re-run against the choices it + * encodes, bypassing generation and shrinking. The engine replays until a replay fails, under a + * bounded budget: a deterministic blob a few times, a nondeterministic one's recorded runs until + * one of them fails again. The run fails with the reproduced failure, or reports a stale blob if + * none of the replays fails. A blob is only guaranteed to reproduce under the Hegel version that + * produced it. * * @param blob the base64 reproduce blob * @return a new settings instance @@ -275,10 +294,11 @@ public Settings infrastructurePackages(String... prefixes) { * These settings with the values the engine resolved for what was left unset, so a run and its * reporters see the effective configuration. */ - Settings resolved(long testCases, boolean printBlob) { + Settings resolved(long testCases, boolean printBlob, NondeterminismStrictness strictness) { return with(b -> { b.testCases = testCases; b.printBlob = printBlob; + b.nondeterminismStrictness = strictness; }); } } diff --git a/shared/src/main/java/dev/hegel/TestCase.java b/shared/src/main/java/dev/hegel/TestCase.java index edff627..674c32d 100644 --- a/shared/src/main/java/dev/hegel/TestCase.java +++ b/shared/src/main/java/dev/hegel/TestCase.java @@ -10,6 +10,7 @@ import java.util.List; import java.util.Map; import java.util.UUID; +import java.util.function.Consumer; import java.util.function.Supplier; /** @@ -19,25 +20,30 @@ * {@link #draw(Generator)}, reject uninteresting inputs with {@link #assume(boolean)}, attach debug * context with {@link #note(String)}, and guide the search with {@link #target(double)}. * - *

On the replay of a minimal failing example ({@link #isFinal()}), each top-level {@code draw} - * and each note is handed to the run's {@link Reporter} (the default prints {@code x = 42;}) and - * recorded on the resulting {@link Failure}, so the counterexample is readable. Under {@link - * Verbosity#VERBOSE} or higher the reporter also sees every other case's draws and notes. A - * label drawn more than once in a case is numbered from its second use ({@code x}, {@code x_2}, - * {@code x_3}); unlabelled draws are {@code draw_1}, {@code draw_2}, ... + *

On an execution the engine stamped for capture ({@link #isFinal()}) — the final replay of a + * minimal failing example among them — each top-level {@code draw} and each note is recorded, and + * if the case fails and the engine reports that failure, they are handed to the run's {@link + * Reporter} (the default prints {@code x = 42;}) and carried by the resulting {@link Failure}, so + * the counterexample is readable. Under {@link Verbosity#VERBOSE} or higher the reporter sees every + * case's draws and notes live. A label drawn more than once in a case is numbered from its second + * use ({@code x}, {@code x_2}, {@code x_3}); unlabelled draws are {@code draw_1}, {@code draw_2}, + * ... * *

Frontends implementing their own composite generators enclose their draws in a labelled * {@link #span(long, Supplier) span} so the engine can shrink the structure they build. */ public final class TestCase { private final DataSource source; - private final boolean finalReplay; - /** Whether draws and notes reach the reporter: on a final replay, or on every case when verbose. */ + /** Whether the engine stamped this execution for capture: its draws and notes are recorded. */ + private final boolean captured; + /** Whether draws and notes reach the reporter live (a verbose run). */ private final boolean reporting; private final Reporter reporter; private final Map draws = new LinkedHashMap<>(); private final List notes = new ArrayList<>(); + /** The recorded draws and notes in report order, each as the reporter call that replays it. */ + private final List> events = new ArrayList<>(); /** Uses per draw name, for numbering repeats ({@code x}, {@code x_2}, ...; {@code draw_N}). */ private final Map nameUses = new HashMap<>(); /** Notes made while a top-level draw is in progress; flushed after that draw's line. */ @@ -45,27 +51,29 @@ public final class TestCase { private int drawDepth; - TestCase(DataSource source, boolean finalReplay, Reporter reporter) { - this(source, finalReplay, false, reporter); + TestCase(DataSource source, boolean captured, Reporter reporter) { + this(source, captured, false, reporter); } - TestCase(DataSource source, boolean finalReplay, boolean verbose, Reporter reporter) { + TestCase(DataSource source, boolean captured, boolean verbose, Reporter reporter) { this.source = source; - this.finalReplay = finalReplay; - this.reporting = finalReplay || verbose; + this.captured = captured; + this.reporting = verbose; this.reporter = reporter; } /** - * Whether this case is the final replay of a minimal counterexample (or of a {@link - * Settings#reproduceFailure(String)} blob), as opposed to one of the many cases the engine runs - * while generating and shrinking. Draws and notes are reported only on a final replay; a test - * body can use this to do its own expensive diagnostics only where they will be seen. + * Whether the engine stamped this execution as one a failure report can be built from — the + * final replay of a minimal counterexample (or of a {@link Settings#reproduceFailure(String)} + * blob), the replays that confirm a discovered failure — as opposed to one of the many cases it + * runs while generating and shrinking. Draws and notes are recorded only on a stamped + * execution, and reported only if it fails and the engine reports that failure; a test body can + * use this to do its own expensive diagnostics only where they may be seen. * - * @return {@code true} on a final replay + * @return {@code true} on an execution stamped for capture */ public boolean isFinal() { - return finalReplay; + return captured; } /** @@ -103,12 +111,15 @@ public T draw(Generator generator, String label) { } } if (top) { - if (reporting) { + if (captured || reporting) { String name = displayName(label); - if (finalReplay) { + if (captured) { draws.put(name, value); + events.add(r -> r.draw(name, value, true)); + } + if (reporting) { + reporter.draw(name, value, false); } - reporter.draw(name, value, finalReplay); } flushPendingNotes(); } @@ -137,10 +148,13 @@ private void flushPendingNotes() { } private void emitNote(String message) { - if (finalReplay) { + if (captured) { notes.add(message); + events.add(r -> r.note(message, true)); + } + if (reporting) { + reporter.note(message, false); } - reporter.note(message, finalReplay); } /** @@ -156,14 +170,15 @@ public void assume(boolean condition) { } /** - * Record a debug message, reported on the final replay of a failing case (and on every case - * under {@link Verbosity#VERBOSE}). A note made while a top-level draw is in progress — from - * inside a composite generator — is reported after that draw's value, never before it. + * Record a debug message, reported with the counterexample when this case's failure is + * reported (and live on every case under {@link Verbosity#VERBOSE}). A note made while a + * top-level draw is in progress — from inside a composite generator — is reported after that + * draw's value, never before it. * * @param message the message to record */ public void note(String message) { - if (!reporting) { + if (!(captured || reporting)) { return; } if (drawDepth > 0) { @@ -413,16 +428,23 @@ boolean isAborted() { return source.isAborted(); } - /** The top-level draws recorded on a final replay, in draw order. */ + /** The top-level draws recorded on a captured execution, in draw order. */ Map draws() { return draws; } - /** The notes recorded on a final replay, in order. */ + /** The notes recorded on a captured execution, in order. */ List notes() { return notes; } + /** Hand the recorded draws and notes to {@code reporter}, in report order, flagged as a replay. */ + void replayTo(Reporter reporter) { + for (Consumer event : events) { + event.accept(reporter); + } + } + static String repr(Object value) { if (value == null) { return "null"; diff --git a/shared/src/test/java/dev/hegel/CoverageTest.java b/shared/src/test/java/dev/hegel/CoverageTest.java index 342b344..ca9137d 100644 --- a/shared/src/test/java/dev/hegel/CoverageTest.java +++ b/shared/src/test/java/dev/hegel/CoverageTest.java @@ -63,6 +63,7 @@ void plain(TestCase tc) {} phases = {Phase.GENERATE}, suppressHealthCheck = {HealthCheck.TOO_SLOW}, backend = Backend.URANDOM, + nondeterminismStrictness = NondeterminismStrictness.ERROR, reportMultipleFailures = true, printBlob = OptBoolean.TRUE, reproduceFailure = "blob-xyz", @@ -106,6 +107,7 @@ void hegelTestHelpers() throws Exception { assertEquals(Database.Kind.UNSET, noSeed.database.kind); assertEquals(0, noSeed.suppressMask); assertEquals(Backend.AUTO, noSeed.backend); + assertEquals(NondeterminismStrictness.DEFAULT, noSeed.nondeterminismStrictness); assertFalse(noSeed.reportMultipleFailures); assertNull(noSeed.printBlob); assertNull(noSeed.testCases); @@ -120,6 +122,7 @@ void hegelTestHelpers() throws Exception { assertEquals(Integer.valueOf(Phase.GENERATE.bit), c.phasesMask); assertEquals(HealthCheck.TOO_SLOW.bit, c.suppressMask); assertEquals(Backend.URANDOM, c.backend); + assertEquals(NondeterminismStrictness.ERROR, c.nondeterminismStrictness); assertTrue(c.reportMultipleFailures); assertEquals(Boolean.TRUE, c.printBlob); assertEquals("blob-xyz", c.reproduceFailure); diff --git a/shared/src/test/java/dev/hegel/EnvironmentFixture.java b/shared/src/test/java/dev/hegel/EnvironmentFixture.java index adf0107..6666a96 100644 --- a/shared/src/test/java/dev/hegel/EnvironmentFixture.java +++ b/shared/src/test/java/dev/hegel/EnvironmentFixture.java @@ -11,6 +11,8 @@ *

  • {@code count}: {@code valid=N}, the number of cases a passing property ran under settings * that leave the budget to the engine; *
  • {@code count-explicit}: the same with an explicit {@code testCases(4)}; + *
  • {@code strictness}: {@code strictness=}, the nondeterminism strictness the engine + * resolved for settings that leave it unset; *
  • {@code fail}: the printing reporter's output for a property that always fails, followed by * {@code done}; *
  • {@code error=} when constructing the settings failed on a malformed variable. @@ -22,7 +24,7 @@ private EnvironmentFixture() {} /** * Entry point. * - * @param args the mode: {@code count}, {@code count-explicit} or {@code fail} + * @param args the mode: {@code count}, {@code count-explicit}, {@code strictness} or {@code fail} */ public static void main(String[] args) { try { @@ -33,6 +35,16 @@ public static void main(String[] args) { case "count-explicit": System.out.println("valid=" + count(new Settings().testCases(4))); break; + case "strictness": + Settings[] effective = new Settings[1]; + Hegel.run(tc -> {}, new Settings().testCases(1), new Reporter() { + @Override + public void runStarted(Settings settings) { + effective[0] = settings; + } + }); + System.out.println("strictness=" + effective[0].nondeterminismStrictness); + break; default: Hegel.run( tc -> { diff --git a/shared/src/test/java/dev/hegel/EnvironmentTest.java b/shared/src/test/java/dev/hegel/EnvironmentTest.java index f0c97fd..a0e2b07 100644 --- a/shared/src/test/java/dev/hegel/EnvironmentTest.java +++ b/shared/src/test/java/dev/hegel/EnvironmentTest.java @@ -66,6 +66,16 @@ void hegelTomlProfileSetsTheBudget() throws Exception { assertTrue(output.contains("valid=9"), output); } + @Test + void hegelNondeterminismStrictnessIsResolvedByTheEngine() throws Exception { + String unset = fixture("strictness", Map.of()); + assertTrue(unset.contains("strictness=QUIET"), unset); + String error = fixture("strictness", Map.of("HEGEL_NONDETERMINISM_STRICTNESS", "error")); + assertTrue(error.contains("strictness=ERROR"), error); + String malformed = fixture("strictness", Map.of("HEGEL_NONDETERMINISM_STRICTNESS", "loud")); + assertTrue(malformed.contains("error=") && malformed.contains("HEGEL_NONDETERMINISM_STRICTNESS"), malformed); + } + @Test void hegelPrintBlobDecidesWhetherReproducersArePrinted() throws Exception { // The shipped profiles print a reproducer for each failure; the variable turns that off. diff --git a/shared/src/test/java/dev/hegel/FakeLibhegel.java b/shared/src/test/java/dev/hegel/FakeLibhegel.java index 1e43c7d..1a78aeb 100644 --- a/shared/src/test/java/dev/hegel/FakeLibhegel.java +++ b/shared/src/test/java/dev/hegel/FakeLibhegel.java @@ -34,9 +34,18 @@ final class FakeLibhegel implements Libhegel { int runStatus = Abi.RUN_STATUS_PASSED; String runError = "run error"; final List failureBlobs = new ArrayList<>(); // one entry per distinct failure + final List failureCaveats = new ArrayList<>(); // parallel to failureBlobs; missing = null + // The origin of each distinct failure. Like the engine, the fake echoes back the origins marked + // interesting, distinct and in first-seen order; an explicit entry here overrides that. + final List failureOrigins = new ArrayList<>(); int fromBlobRc = Abi.OK; final List replayedBlobs = new ArrayList<>(); - Consumer output; // the callback runStart registered + String startedBlob; // what runStartBlob received; null = the run explored + Consumer output; // the callback runStart / runStartBlob registered + // The engine's capture stamp per served case: the i-th case is stamped when captureSequence + // has an i-th entry and it is true; past the end of the sequence every case is stamped, so + // the default stamps everything. + boolean[] captureSequence = {}; // Recorded outcomes and teardown. final List markedStatuses = new ArrayList<>(); @@ -149,7 +158,9 @@ final class FakeLibhegel implements Libhegel { int settingsNewRc = Abi.OK; long resolvedTestCases = 100; boolean resolvedPrintBlob = false; + int resolvedStrictness = Abi.NONDETERMINISM_QUIET; Boolean printBlob; // what the setter received; null = never set + Integer strictness; // what the setter received; null = never set @Override public int settingsNew(long[] out) { @@ -174,6 +185,16 @@ public boolean settingsGetPrintBlob(long s) { return printBlob == null ? resolvedPrintBlob : printBlob; } + @Override + public void settingsNondeterminismStrictness(long s, int strictness) { + this.strictness = strictness; + } + + @Override + public int settingsGetNondeterminismStrictness(long s) { + return strictness == null ? resolvedStrictness : strictness; + } + @Override public void settingsFree(long s) { settingsFreed = true; @@ -232,6 +253,16 @@ public long runStart(long settings, Consumer output) { return RUN; } + @Override + public long runStartBlob(long settings, String blob, Consumer output) { + if (runStartFails) { + throw new HegelException("hegel_run_start_blob failed: " + lastError); + } + startedBlob = blob; + this.output = output; + return RUN; + } + @Override public long nextTestCase(long run) { if (nextTestCaseFails) { @@ -244,6 +275,12 @@ public long nextTestCase(long run) { return TC; } + @Override + public boolean testCaseShouldCapture(long tc) { + int index = casesServed - 1; + return index >= captureSequence.length || captureSequence[index]; + } + @Override public long runResult(long run) { return RESULT; @@ -603,7 +640,21 @@ public String failureBlob(long result, long index) { @Override public String failureOrigin(long result, long index) { - return "fake-origin-" + index; + if (index < failureOrigins.size()) { + return failureOrigins.get((int) index); + } + List distinct = new ArrayList<>(); + for (String origin : markedOrigins) { + if (origin != null && !distinct.contains(origin)) { + distinct.add(origin); + } + } + return distinct.get((int) index); + } + + @Override + public String failureCaveat(long result, long index) { + return index < failureCaveats.size() ? failureCaveats.get((int) index) : null; } @Override diff --git a/shared/src/test/java/dev/hegel/LibraryApiTest.java b/shared/src/test/java/dev/hegel/LibraryApiTest.java index b9fae9f..8fa6441 100644 --- a/shared/src/test/java/dev/hegel/LibraryApiTest.java +++ b/shared/src/test/java/dev/hegel/LibraryApiTest.java @@ -3,11 +3,11 @@ import static dev.hegel.Generators.integers; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; -import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.junit.jupiter.api.Assertions.assertTrue; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.concurrent.atomic.AtomicInteger; import org.junit.jupiter.api.Test; @@ -38,13 +38,15 @@ void runReportsTheShrunkCounterexample() { Failure f = report.failures().get(0); assertEquals(Map.of("x", 11), f.draws()); assertEquals(List.of("observed x=11"), f.notes()); - assertTrue(f.exception().get() instanceof AssertionError); - assertTrue(f.exception().get().getMessage().contains("x was too big: 11")); + assertTrue(f.exception() instanceof AssertionError); + assertTrue(f.exception().getMessage().contains("x was too big: 11")); assertTrue(f.origin().startsWith("AssertionFailedError at "), f.origin()); - assertNotNull(f.reproduceBlob()); - assertFalse(f.flaky()); - // Exactly one case was the final replay, and the run counted more than just it. - assertEquals(1, finals.get()); + assertTrue(f.reproduceBlob().isPresent()); + assertFalse(f.nondeterministic()); + assertEquals(Optional.empty(), f.caveat()); + // The engine stamped at least the final replay for capture, and the run counted more than + // the stamped cases. + assertTrue(finals.get() >= 1, "stamped cases: " + finals.get()); assertTrue(report.statistics().interesting() >= 2, report.statistics().toString()); assertTrue(report.statistics().total() > report.statistics().interesting()); } diff --git a/shared/src/test/java/dev/hegel/ReporterTest.java b/shared/src/test/java/dev/hegel/ReporterTest.java index b3d91f7..186799e 100644 --- a/shared/src/test/java/dev/hegel/ReporterTest.java +++ b/shared/src/test/java/dev/hegel/ReporterTest.java @@ -20,7 +20,7 @@ private static String text(ByteArrayOutputStream buf) { void silentReporterIgnoresEveryCallback() { Reporter silent = Reporter.silent(); RunReport report = new RunReport(RunStatus.PASSED, new RunStatistics.Counter().snapshot(), null, List.of()); - Failure failure = new Failure("origin", "blob", new AssertionError("x"), Map.of(), List.of()); + Failure failure = new Failure("origin", "blob", null, new AssertionError("x"), Map.of(), List.of()); silent.runStarted(new Settings()); silent.engineOutput("line"); silent.caseStarted(false); @@ -42,10 +42,23 @@ void printingReporterFormatsDrawsNotesAndEngineOutput() { printing.draw("xs", List.of(1, 2), true); printing.note("a note", true); printing.caseFinished(CaseOutcome.INTERESTING, true); - printing.failure(new Failure("o", "b64", new AssertionError("x"), Map.of(), List.of())); // printBlob off + printing.failure(new Failure("o", "b64", null, new AssertionError("x"), Map.of(), List.of())); // printBlob off assertEquals("engine line\nxs = [1, 2];\na note\n", text(buf)); } + @Test + void printingReporterPrintsTheCaveatThenTheReproducer() { + ByteArrayOutputStream buf = new ByteArrayOutputStream(); + Reporter printing = Reporter.printing(new PrintStream(buf, true, StandardCharsets.UTF_8)); + printing.runStarted(new Settings().printBlob(true)); + printing.failure(new Failure( + "o", "b64", "nondeterministic failure, confirmed", new AssertionError("x"), Map.of(), List.of())); + assertEquals( + "note: nondeterministic failure, confirmed\n\nTo reproduce this failure, replay it with:\n" + + " @HegelTest(reproduceFailure = \"b64\")\n", + text(buf)); + } + @Test void printingReporterHonoursVerbosity() { ByteArrayOutputStream quietBuf = new ByteArrayOutputStream(); diff --git a/shared/src/test/java/dev/hegel/ReproduceFailureTest.java b/shared/src/test/java/dev/hegel/ReproduceFailureTest.java index 1fab64b..9b9549e 100644 --- a/shared/src/test/java/dev/hegel/ReproduceFailureTest.java +++ b/shared/src/test/java/dev/hegel/ReproduceFailureTest.java @@ -1,12 +1,19 @@ package dev.hegel; import static dev.hegel.Generators.integers; +import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertThrows; import static org.junit.jupiter.api.Assertions.assertTrue; +import dev.hegel.lowlevel.Abi; +import dev.hegel.lowlevel.Libhegel; import java.io.ByteArrayOutputStream; import java.io.PrintStream; import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Consumer; import org.junit.jupiter.api.Test; @@ -46,7 +53,27 @@ void printedBlobReplaysTheExactFailure() { () -> Hegel.test( tc -> tc.draw(integers().min(0).max(1000), "x"), new Settings().database(Database.disabled()).reproduceFailure(blob))); - assertTrue(stale.getMessage().contains("no longer reproduces"), stale.getMessage()); + assertTrue(stale.getMessage().contains("did not reproduce"), stale.getMessage()); + } + + @Test + void standaloneSingleAttemptReplayRemainsAvailable() { + // The runner drives blob replays through the engine's run loop, but the binding still + // exposes the one-shot hegel_test_case_from_blob for frontends built on hegel-lowlevel. + Libhegel lib = Engine.get(); + RunReport report = + Hegel.run(FAILING, new Settings().database(Database.disabled()).seed(1), Reporter.silent()); + String blob = report.failures().get(0).reproduceBlob().orElseThrow(); + long[] settings = new long[1]; + assertEquals(Abi.OK, lib.settingsNew(settings)); + lib.settingsDatabase(settings[0], ""); + List lines = new ArrayList<>(); + long[] tc = new long[1]; + assertEquals(Abi.OK, lib.testCaseFromBlob(settings[0], blob, lines::add, tc)); + assertTrue(tc[0] != 0); + assertEquals(Abi.OK, lib.markComplete(tc[0], Abi.STATUS_VALID, null)); + lib.testCaseFree(tc[0]); + lib.settingsFree(settings[0]); } @Test @@ -59,8 +86,37 @@ void corruptBlobsAreRejected() { } @Test - void flakyTestsAreDetected() { - java.util.concurrent.atomic.AtomicInteger calls = new java.util.concurrent.atomic.AtomicInteger(); + void aTestThatFailsOnceIsReportedUnconfirmedWithACaveat() { + AtomicInteger calls = new AtomicInteger(); + Consumer onceOnly = tc -> { + tc.draw(integers(), "x"); + if (calls.incrementAndGet() == 1) { + throw new AssertionError("only the first time"); + } + }; + Settings settings = new Settings().database(Database.disabled()).seed(3); + RunReport report = Hegel.run(onceOnly, settings, Reporter.silent()); + assertEquals(RunStatus.FAILED, report.status()); + Failure f = report.failures().get(0); + assertTrue(f.nondeterministic(), f.toString()); + String caveat = f.caveat().orElseThrow(); + assertTrue(caveat.startsWith("unconfirmed failure"), caveat); + // No replay ever failed again, so there is no blob, and the only failing execution was the + // unstamped discovery: the exception is the body's own, the draws were not recorded. + assertEquals(Optional.empty(), f.reproduceBlob()); + assertTrue(f.exception() instanceof AssertionError, String.valueOf(f.exception())); + assertTrue(f.draws().isEmpty(), f.draws().toString()); + + // Hegel.test rethrows the body's own failure, and the printed report carries the caveat. + calls.set(0); + String output = runCapturing(settings, onceOnly, AssertionError.class); + assertTrue(output.contains("note: unconfirmed failure"), output); + assertTrue(!output.contains("reproduceFailure = \""), output); + } + + @Test + void errorStrictnessAbortsOnANondeterministicTest() { + AtomicInteger calls = new AtomicInteger(); HegelException e = assertThrows( HegelException.class, () -> Hegel.test( @@ -70,8 +126,38 @@ void flakyTestsAreDetected() { throw new AssertionError("only the first time"); } }, - new Settings().database(Database.disabled()).seed(3))); - assertTrue(e.getMessage().contains("Flaky"), e.getMessage()); + new Settings() + .database(Database.disabled()) + .seed(3) + .nondeterminismStrictness(NondeterminismStrictness.ERROR))); + assertTrue(e.getMessage().toLowerCase().contains("flaky"), e.getMessage()); + } + + @Test + void confirmedNondeterministicFailureCarriesABlobThatReplays() { + // Fails every other time a large value is drawn: nondeterministic, but reproducible often + // enough for the engine to confirm it, shrink it, and hand back a blob. + AtomicInteger bigDraws = new AtomicInteger(); + Consumer intermittent = tc -> { + int x = tc.draw(integers().min(0).max(1000), "x"); + if (x > 10 && bigDraws.incrementAndGet() % 2 == 0) { + throw new AssertionError("x was too big (this time): " + x); + } + }; + Settings settings = new Settings().database(Database.disabled()).seed(7).printBlob(true); + String output = runCapturing(settings, intermittent, AssertionError.class); + assertTrue(output.contains("note: nondeterministic failure, confirmed"), output); + assertTrue(output.contains("reproduceFailure = \""), output); + String tail = output.substring(output.indexOf("reproduceFailure = \"") + "reproduceFailure = \"".length()); + String blob = tail.substring(0, tail.indexOf('"')); + + // The engine replays a nondeterministic blob until one of its recorded runs fails again. + RunReport replay = Hegel.run( + intermittent, new Settings().database(Database.disabled()).reproduceFailure(blob), Reporter.silent()); + assertEquals(RunStatus.FAILED, replay.status()); + assertTrue(replay.failures().get(0).nondeterministic()); + assertTrue(replay.failures().get(0).exception() instanceof AssertionError); + assertEquals(Optional.empty(), replay.failures().get(0).reproduceBlob()); } @Test diff --git a/shared/src/test/java/dev/hegel/RunReportTest.java b/shared/src/test/java/dev/hegel/RunReportTest.java index 925a27c..21a9d95 100644 --- a/shared/src/test/java/dev/hegel/RunReportTest.java +++ b/shared/src/test/java/dev/hegel/RunReportTest.java @@ -141,41 +141,87 @@ void failedRunCarriesTheReplayedFailure() { assertFalse(r.passed()); assertEquals(1, r.failures().size()); Failure f = r.failures().get(0); - assertEquals("fake-origin-0", f.origin()); - assertEquals("blob-1", f.reproduceBlob()); - assertSame(err, f.exception().get()); - assertFalse(f.flaky()); - // Only the final replay records draws and notes, so one exploration case plus one replay - // yields a single set of each. + // The engine echoes back the origin the runner marked the case with. + assertEquals(Runner.originOf(err, List.of()), f.origin()); + assertEquals(Optional.of("blob-1"), f.reproduceBlob()); + assertSame(err, f.exception()); + assertFalse(f.nondeterministic()); + assertEquals(Optional.empty(), f.caveat()); + // The fake stamps the case for capture, so its draws and notes are the report. assertEquals(List.of("x", "draw_1"), List.copyOf(f.draws().keySet())); assertEquals(7, f.draws().get("x")); assertEquals(List.of("saw 7"), f.notes()); assertThrows(UnsupportedOperationException.class, () -> f.draws().put("y", 1)); assertThrows(UnsupportedOperationException.class, () -> f.notes().add("more")); assertThrows(UnsupportedOperationException.class, () -> r.failures().clear()); - // Two cases ran: the exploration case and the replay. - assertEquals(2, r.statistics().total()); - assertEquals(2, r.statistics().interesting()); + // The engine ran the one case; nothing is replayed client-side. + assertEquals(1, r.statistics().total()); + assertEquals(1, r.statistics().interesting()); assertSame(err, assertThrows(AssertionError.class, r::throwIfFailed)); } @Test - void flakyReplayIsReportedRatherThanThrown() { + void nondeterministicFailureCarriesItsCaveat() { FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; - fake.failureBlobs.add("blob-1"); + fake.failureBlobs.add("nd-blob"); + fake.failureCaveats.add("nondeterministic failure, confirmed: failed 3 of 20 replays"); + AssertionError err = new AssertionError("sometimes"); + RunReport r = report(fake, new Settings().database(Database.disabled()), tc -> { + throw err; + }); + assertEquals(RunStatus.FAILED, r.status()); + Failure f = r.failures().get(0); + assertTrue(f.nondeterministic()); + assertEquals(Optional.of("nondeterministic failure, confirmed: failed 3 of 20 replays"), f.caveat()); + assertEquals(Optional.of("nd-blob"), f.reproduceBlob()); + // The failure is the test's own, caveat or not. + assertSame(err, assertThrows(AssertionError.class, r::throwIfFailed)); + } + + @Test + void unconfirmedFailureIsReportedFromItsDiscovery() { + // A test that failed once and never again: the engine reports the origin unconfirmed, with a + // caveat and no blob, and the only failing execution was the unstamped discovery, whose + // exception is all the runner kept. + FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 3; + fake.captureSequence = new boolean[] {false, true, true}; + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add(null); + fake.failureCaveats.add("nondeterministic failure, unconfirmed: failed 0 of 20 replays"); AtomicInteger calls = new AtomicInteger(); + AssertionError err = new AssertionError("only once"); RunReport r = report(fake, new Settings().database(Database.disabled()), tc -> { + tc.draw(integers(), "x"); if (calls.incrementAndGet() == 1) { - throw new AssertionError("only once"); + throw err; } }); - assertEquals(RunStatus.FAILED, r.status()); Failure f = r.failures().get(0); - assertTrue(f.flaky()); - assertEquals(Optional.empty(), f.exception()); - HegelException e = assertThrows(HegelException.class, r::throwIfFailed); - assertEquals(Runner.FLAKY_DIAGNOSTIC, e.getMessage()); + assertTrue(f.nondeterministic()); + assertEquals(Optional.empty(), f.reproduceBlob()); + assertSame(err, f.exception()); + assertTrue(f.draws().isEmpty()); + assertSame(err, assertThrows(AssertionError.class, r::throwIfFailed)); + } + + @Test + void stampedCaptureOutranksALaterUnstampedOne() { + // Shrink probes (unstamped) keep failing after the stamped replay: the report still comes + // from the stamped execution, and among stamped ones the newest wins. + FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 4; + fake.captureSequence = new boolean[] {false, true, true, false}; + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add("blob-1"); + AtomicInteger calls = new AtomicInteger(); + RunReport r = report(fake, new Settings().database(Database.disabled()), tc -> { + int n = calls.incrementAndGet(); + tc.note("case " + n); + throw new AssertionError("always"); + }); + assertEquals(List.of("case 3"), r.failures().get(0).notes()); } @Test @@ -187,7 +233,7 @@ void checkedExceptionsAreRethrownAsIs() { RunReport r = report(fake, new Settings().database(Database.disabled()), tc -> { throw sneaky(io); }); - assertSame(io, r.failures().get(0).exception().get()); + assertSame(io, r.failures().get(0).exception()); assertSame(io, assertThrows(IOException.class, r::throwIfFailed)); } @@ -226,7 +272,7 @@ void reporterCallbacksArriveInOrder() { fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); RecordingReporter reporter = new RecordingReporter(); - report( + RunReport r = report( fake, new Settings().database(Database.disabled()).testCases(5), tc -> { @@ -237,6 +283,8 @@ void reporterCallbacksArriveInOrder() { reporter); // Engine output goes through the run's callback to the reporter, whenever it arrives. fake.output.accept("engine says hi"); + // The stamped case's draws and notes are not reported live; they are replayed, flagged as + // such, once the engine reports the failure. assertEquals( List.of( "runStarted:5", @@ -247,14 +295,14 @@ void reporterCallbacksArriveInOrder() { "draw:x=3:true", "note:hi:true", "caseFinished:INTERESTING:true", - "failure:fake-origin-0", + "failure:" + r.failures().get(0).origin(), "runFinished:FAILED", "engineOutput:engine says hi"), reporter.events); } @Test - void verboseRunsReportEveryCaseButRecordOnlyTheReplay() { + void verboseRunsReportEveryCaseLiveAndReplayTheCapture() { FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); @@ -268,11 +316,11 @@ void verboseRunsReportEveryCaseButRecordOnlyTheReplay() { throw new AssertionError("boom"); }, reporter); - // The exploration case's draws and notes are reported (flagged non-final) ... + // The case's draws and notes are reported live (flagged as such) ... assertTrue(reporter.events.contains("draw:x=3:false"), reporter.events.toString()); assertTrue(reporter.events.contains("note:hi:false"), reporter.events.toString()); + // ... and replayed with the failure report. assertTrue(reporter.events.contains("draw:x=3:true"), reporter.events.toString()); - // ... but the failure captures the final replay only. assertEquals(Map.of("x", 3), r.failures().get(0).draws()); assertEquals(List.of("hi"), r.failures().get(0).notes()); } @@ -280,6 +328,7 @@ void verboseRunsReportEveryCaseButRecordOnlyTheReplay() { @Test void multipleFailuresAreReportedInEngineOrder() { FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 2; // one case per distinct bug fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); fake.failureBlobs.add("blob-2"); @@ -296,8 +345,9 @@ void multipleFailuresAreReportedInEngineOrder() { }, reporter); assertEquals(2, r.failures().size()); - assertEquals("blob-1", r.failures().get(0).reproduceBlob()); - assertEquals("fake-origin-1", r.failures().get(1).origin()); + assertEquals(Optional.of("blob-1"), r.failures().get(0).reproduceBlob()); + assertTrue(r.failures().get(0).origin().startsWith("AssertionError at ")); + assertTrue(r.failures().get(1).origin().startsWith("IllegalStateException at ")); assertTrue(reporter.events.contains("failuresFound:2")); AssertionError e = assertThrows(AssertionError.class, r::throwIfFailed); assertEquals(2, e.getSuppressed().length); @@ -306,6 +356,8 @@ void multipleFailuresAreReportedInEngineOrder() { @Test void reproduceFailureReportsTheReplayedFailure() { FakeLibhegel fake = new FakeLibhegel(); + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add(null); // a blob replay's failure carries no fresh blob RecordingReporter reporter = new RecordingReporter(); IllegalStateException err = new IllegalStateException("reproduced"); RunReport r = report( @@ -316,20 +368,28 @@ void reproduceFailureReportsTheReplayedFailure() { throw err; }, reporter); + assertEquals("stored-blob", fake.startedBlob); assertEquals(RunStatus.FAILED, r.status()); Failure f = r.failures().get(0); - assertEquals("stored-blob", f.reproduceBlob()); - assertSame(err, f.exception().get()); + assertEquals(Optional.empty(), f.reproduceBlob()); + assertSame(err, f.exception()); // The test suite itself lives in dev.hegel, which counts as infrastructure, so the origin // names the exception and whatever frame sits below the suite. assertTrue(f.origin().startsWith("IllegalStateException at "), f.origin()); assertEquals(Map.of("x", 1), f.draws()); assertEquals(1, r.statistics().total()); assertEquals(1, r.statistics().interesting()); - // No exploration loop: straight to the replay. + // The blob run is pumped like any other, then its failure is reported from the capture. assertEquals( - List.of("runStarted:100", "caseStarted:true", "draw:x=1:true", "caseFinished:INTERESTING:true"), - reporter.events.subList(0, 4)); + List.of( + "runStarted:100", + "caseStarted:false", + "caseFinished:INTERESTING:false", + "failuresFound:1", + "caseStarted:true", + "draw:x=1:true", + "caseFinished:INTERESTING:true"), + reporter.events.subList(0, 7)); assertEquals("runFinished:FAILED", reporter.events.get(reporter.events.size() - 1)); } diff --git a/shared/src/test/java/dev/hegel/RunnerTest.java b/shared/src/test/java/dev/hegel/RunnerTest.java index 6cf0eb5..bc85704 100644 --- a/shared/src/test/java/dev/hegel/RunnerTest.java +++ b/shared/src/test/java/dev/hegel/RunnerTest.java @@ -139,7 +139,7 @@ private static FakeLibhegel runFailing(Consumer body) { } @Test - void replayPassesTheBlobToTheEngine() { + void failuresAreReportedFromTheCaptureWithoutAClientReplay() { FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-xyz"); @@ -148,49 +148,52 @@ void replayPassesTheBlobToTheEngine() { () -> run(fake, new Settings().database(Database.disabled()), tc -> { throw new AssertionError("always"); })); - assertEquals(List.of("blob-xyz"), fake.replayedBlobs); - // One exploration case plus one replay case, all freed. - assertEquals(2, fake.freedTestCases); - } - - @Test - void replayThatPassesIsFlaky() { - AtomicInteger calls = new AtomicInteger(); - HegelException e = assertThrows( - HegelException.class, - () -> runFailing(tc -> { - if (calls.incrementAndGet() == 1) { - throw new AssertionError("only once"); - } - })); - assertTrue(e.getMessage().contains("Flaky"), e.getMessage()); + // The engine owns every replay: nothing is replayed from the blob client-side. + assertTrue(fake.replayedBlobs.isEmpty()); + assertNull(fake.startedBlob); + assertEquals(1, fake.freedTestCases); } @Test - void missingBlobIsAnInternalError() { + void failureWithoutACapturedExecutionIsAnInternalError() { + // The engine reports a failure whose origin never failed in this process: a plumbing bug. FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; - fake.failureBlobs.add(null); + fake.failureBlobs.add("blob-1"); + fake.failureOrigins.add("AssertionError at Elsewhere.java:1"); HegelException e = assertThrows( HegelException.class, () -> run(fake, new Settings().database(Database.disabled()), tc -> {})); - assertTrue(e.getMessage().contains("no reproduce blob"), e.getMessage()); + assertTrue(e.getMessage().contains("no captured failing execution"), e.getMessage()); + assertTrue(e.getMessage().contains("Elsewhere.java:1"), e.getMessage()); } @Test - void undecodableBlobFailsTheRun() { + void unstampedCasesKeepOnlyTheirException() { FakeLibhegel fake = new FakeLibhegel(); + fake.captureSequence = new boolean[] {false}; fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); - fake.fromBlobRc = Abi.E_INVALID_ARG; - fake.lastError = "bad blob"; - HegelException e = assertThrows( - HegelException.class, () -> run(fake, new Settings().database(Database.disabled()), tc -> {})); - assertTrue(e.getMessage().contains("bad blob"), e.getMessage()); + ByteArrayOutputStream buf = new ByteArrayOutputStream(); + AssertionError err = new AssertionError("always"); + RunReport report = Runner.run( + fake, + new Settings().database(Database.disabled()).printBlob(false), + tc -> { + tc.draw(integers(), "x"); + tc.note("unseen"); + throw err; + }, + Reporter.printing(capture(buf))); + assertSame(err, report.failures().get(0).exception()); + assertTrue(report.failures().get(0).draws().isEmpty()); + assertTrue(report.failures().get(0).notes().isEmpty()); + assertEquals("", buf.toString(StandardCharsets.UTF_8)); } @Test void multipleFailuresAggregateWithSuppressedOriginals() { FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 2; // one case per distinct bug fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); fake.failureBlobs.add("blob-2"); @@ -219,15 +222,21 @@ void multipleFailuresAggregateWithSuppressedOriginals() { @Test void aggregateMessageHandlesNullExceptionMessages() { FakeLibhegel fake = new FakeLibhegel(); + fake.caseCount = 2; fake.runStatus = Abi.RUN_STATUS_FAILED; fake.failureBlobs.add("blob-1"); fake.failureBlobs.add("blob-2"); + AtomicInteger calls = new AtomicInteger(); AssertionError e = assertThrows( AssertionError.class, () -> run(fake, new Settings(), tc -> { - throw new IllegalStateException(); // null message + if (calls.incrementAndGet() == 1) { + throw new IllegalStateException(); // null message + } + throw new UnsupportedOperationException(); // null message })); assertTrue(e.getMessage().contains(IllegalStateException.class.getName()), e.getMessage()); + assertTrue(e.getMessage().contains(UnsupportedOperationException.class.getName()), e.getMessage()); } @Test @@ -282,8 +291,10 @@ void nullRunErrorBecomesEmptyMessage() { } @Test - void reproduceFailureReplaysWithoutARun() { + void reproduceFailureStartsABlobRun() { FakeLibhegel fake = new FakeLibhegel(); + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add(null); IllegalStateException err = new IllegalStateException("reproduced"); assertSame( err, @@ -292,38 +303,65 @@ void reproduceFailureReplaysWithoutARun() { () -> run(fake, new Settings().reproduceFailure("stored-blob"), tc -> { throw err; }))); - assertEquals(List.of("stored-blob"), fake.replayedBlobs); - assertNull(fake.output); // runStart was never called + assertEquals("stored-blob", fake.startedBlob); + assertTrue(fake.replayedBlobs.isEmpty()); + assertTrue(fake.runFreed); assertTrue(fake.settingsFreed); } @Test void reproduceFailureReportsAStaleBlob() { + // The engine's blob run passed: none of its replays failed. FakeLibhegel fake = new FakeLibhegel(); HegelException e = assertThrows(HegelException.class, () -> run(fake, new Settings().reproduceFailure("stale"), tc -> {})); - assertTrue(e.getMessage().contains("no longer reproduces"), e.getMessage()); + assertEquals(Runner.STALE_BLOB, e.getMessage()); + assertTrue(e.getMessage().contains("did not reproduce"), e.getMessage()); } @Test void reproduceFailureRejectsAnInvalidBlob() { + // An undecodable blob surfaces as the blob run's error. FakeLibhegel fake = new FakeLibhegel(); - fake.fromBlobRc = Abi.E_INVALID_ARG; - fake.lastError = "corrupt"; + fake.runStatus = Abi.RUN_STATUS_ERROR; + fake.runError = "corrupt"; HegelException e = assertThrows(HegelException.class, () -> run(fake, new Settings().reproduceFailure("???"), tc -> {})); + assertTrue(e.getMessage().contains("not valid"), e.getMessage()); assertTrue(e.getMessage().contains("corrupt"), e.getMessage()); } @Test - void nondeterministicRunFailureIsAnInternalError() { - // Only a concurrent state machine declares a run nondeterministic, and this binding never - // creates one, so the engine reporting that status (whose failures carry no blob) is a bug. + void nondeterminismStrictnessIsSentOnlyWhenSet() { FakeLibhegel fake = new FakeLibhegel(); - fake.runStatus = Abi.RUN_STATUS_FAILED_NONDETERMINISTIC; - HegelException e = assertThrows(HegelException.class, () -> run(fake, new Settings(), tc -> {})); - assertTrue(e.getMessage().contains("nondeterministic"), e.getMessage()); - assertTrue(fake.replayedBlobs.isEmpty()); + run(fake, new Settings().database(Database.disabled()), tc -> {}); + assertNull(fake.strictness); + run( + fake, + new Settings().database(Database.disabled()).nondeterminismStrictness(NondeterminismStrictness.WARN), + tc -> {}); + assertEquals(Integer.valueOf(Abi.NONDETERMINISM_WARN), fake.strictness); + } + + @Test + void effectiveStrictnessIsReadBackFromTheEngine() { + FakeLibhegel fake = new FakeLibhegel(); + fake.resolvedStrictness = Abi.NONDETERMINISM_ERROR; + Settings[] seen = new Settings[1]; + Reporter reporter = new Reporter() { + @Override + public void runStarted(Settings settings) { + seen[0] = settings; + } + }; + Runner.run(fake, new Settings().database(Database.disabled()), tc -> {}, reporter); + assertEquals(NondeterminismStrictness.ERROR, seen[0].nondeterminismStrictness); + assertEquals(NondeterminismStrictness.DEFAULT, new Settings().nondeterminismStrictness); + + FakeLibhegel unknown = new FakeLibhegel(); + unknown.resolvedStrictness = 99; + HegelException e = assertThrows(HegelException.class, () -> run(unknown, new Settings(), tc -> {})); + assertTrue(e.getMessage().contains("99"), e.getMessage()); } @Test @@ -448,22 +486,41 @@ void infrastructurePackagesAreSkippedWhenLocatingTheOrigin() { } @Test - void printBlobIsSkippedForAFlakyReplay() { + void caveatIsPrintedAndABloblessFailurePrintsNoReproducer() { FakeLibhegel fake = new FakeLibhegel(); fake.runStatus = Abi.RUN_STATUS_FAILED; - fake.failureBlobs.add("blob-b64"); - AtomicInteger calls = new AtomicInteger(); + fake.failureBlobs.add(null); + fake.failureCaveats.add("nondeterministic failure, unconfirmed: failed 0 of 20 replays"); ByteArrayOutputStream buf = new ByteArrayOutputStream(); RunReport report = Runner.run( fake, new Settings().database(Database.disabled()).printBlob(true), tc -> { - if (calls.incrementAndGet() == 1) { - throw new AssertionError("only once"); - } + tc.draw(integers().min(0), "x"); + throw new AssertionError("sometimes"); }, Reporter.printing(capture(buf))); - assertTrue(report.failures().get(0).flaky()); - assertEquals("", buf.toString(StandardCharsets.UTF_8)); + assertTrue(report.failures().get(0).nondeterministic()); + String out = buf.toString(StandardCharsets.UTF_8).replace("\r\n", "\n"); + assertEquals("x = 0;\nnote: nondeterministic failure, unconfirmed: failed 0 of 20 replays\n", out); + } + + @Test + void quietRunsPrintNoCaveat() { + FakeLibhegel fake = new FakeLibhegel(); + fake.runStatus = Abi.RUN_STATUS_FAILED; + fake.failureBlobs.add("nd-blob"); + fake.failureCaveats.add("nondeterministic failure, confirmed: failed 9 of 20 replays"); + ByteArrayOutputStream buf = new ByteArrayOutputStream(); + Runner.run( + fake, + new Settings().database(Database.disabled()).printBlob(true).verbosity(Verbosity.QUIET), + tc -> { + throw new AssertionError("sometimes"); + }, + Reporter.printing(capture(buf))); + String out = buf.toString(StandardCharsets.UTF_8); + assertTrue(!out.contains("note:"), out); + assertTrue(out.contains("reproduceFailure = \"nd-blob\""), out); } } diff --git a/shared/src/test/java/dev/hegel/TestCaseTest.java b/shared/src/test/java/dev/hegel/TestCaseTest.java index cecaab1..dc7988a 100644 --- a/shared/src/test/java/dev/hegel/TestCaseTest.java +++ b/shared/src/test/java/dev/hegel/TestCaseTest.java @@ -18,11 +18,15 @@ private static String text(ByteArrayOutputStream buf) { return buf.toString(StandardCharsets.UTF_8).replace("\r\n", "\n"); } - /** A TestCase over a fake binding; only the reporting/target plumbing is under test here. */ + /** + * A TestCase over a fake binding, both stamped for capture and reporting live (verbose) when + * {@code reporting}; only the reporting/target plumbing is under test here. + */ private TestCase newCase(FakeLibhegel fake, boolean reporting, ByteArrayOutputStream buf) { return new TestCase( new LiveDataSource(fake, FakeLibhegel.TC), reporting, + reporting, Reporter.printing(new PrintStream(buf, true, StandardCharsets.UTF_8))); } @@ -75,7 +79,7 @@ void noteRespectsReporting() { } @Test - void finalReplayRecordsDrawsAndNotes() { + void capturedCaseRecordsDrawsAndNotesAndReplaysThem() { TestCase tc = newCase(new FakeLibhegel(), true, new ByteArrayOutputStream()); assertTrue(tc.isFinal()); tc.draw(constant(1), "x"); @@ -87,6 +91,24 @@ void finalReplayRecordsDrawsAndNotes() { assertEquals(want, tc.draws()); assertEquals(List.of("x", "draw_1"), List.copyOf(tc.draws().keySet())); assertEquals(List.of("first"), tc.notes()); + // The recording replays in report order, flagged as a replay. + ByteArrayOutputStream replay = new ByteArrayOutputStream(); + Reporter reporter = Reporter.printing(new PrintStream(replay, true, StandardCharsets.UTF_8)); + reporter.runStarted(new Settings()); + tc.replayTo(reporter); + assertEquals("x = 1;\nfirst\ndraw_1 = 2;\n", text(replay)); + + // A case stamped for capture but not verbose records without reporting live. + ByteArrayOutputStream silent = new ByteArrayOutputStream(); + TestCase captured = new TestCase( + new LiveDataSource(new FakeLibhegel(), FakeLibhegel.TC), + true, + Reporter.printing(new PrintStream(silent, true, StandardCharsets.UTF_8))); + captured.draw(constant(3), "y"); + captured.note("kept"); + assertEquals("", text(silent)); + assertEquals(Map.of("y", 3), captured.draws()); + assertEquals(List.of("kept"), captured.notes()); TestCase exploring = newCase(new FakeLibhegel(), false, new ByteArrayOutputStream()); assertTrue(!exploring.isFinal());