diff --git a/BREAK-REPORT.md b/BREAK-REPORT.md index 094ce4a3..ad69e650 100644 --- a/BREAK-REPORT.md +++ b/BREAK-REPORT.md @@ -360,3 +360,354 @@ integration runs needing the app and databases were not run beyond the Recommend yes, as a follow-up task. 2. No migration, no config, no API changes in this work. Nothing needs a decision to land it. + +--- + +# Audit chain break report + +4 breaks found. 4 fixed and tested. 0 fixed but failing. 0 blocked. + +Middle-link deletion was already rejected (`audit_previous_hash_mismatch`). That case is covered by a test and is not counted above. + +Local SQLite tests only. No production calls. + +## High: a time window hid a deleted tail + +What broke: delete the last audit event for a wallet, then verify with any time window that still covers the survivors (a start date in the past, an end date in the future, or both). The answer was valid. The same deletion with no time window was already reported as `audit_chain_truncated`. + +How to reproduce: record three events, delete the last row, call `verify_audit_chain` with `created_after=2000-01-01` or `created_before=2099-01-01`. + +Root cause: `gateway/app/services/audit_chain.py` loaded the chain head only when both time filters were absent. A window skipped the anchor that detects a missing tail. The head is now loaded on every verify (snapshot at `audit_chain.py:508`, tail check at `audit_chain.py:667`). + +Fix: one snapshot loads every event and the head. Time filters choose which links get a hash walk. The tail is still compared to the head. + +Test: `test_time_window_does_not_hide_a_deleted_tail` (three windows). Before the fix each window returned `valid=True` with 2 events. After the fix, `audit_chain_truncated`. + +## High: sequence holes and duplicate sequence numbers verified as valid + +What broke: `seq` is not part of the signature. Renumber the tail from 3 to 4 and set the head's `last_seq` to 4. Hash order and the head hash still match, so verify returned valid for a chain whose sequences are 1, 2, 4. Collapsing the tail onto the previous sequence (1, 2, 2) and pointing the head at that sequence also returned valid. + +How to reproduce: record three events, update the last row's `seq` and the head's `last_seq` as above, call `verify_audit_chain`. + +Root cause: `gateway/app/services/audit_chain.py` compared only the last event's seq and chain hash to the head. It never required the sequences to be 1, 2, 3, and so on up to that head. The check is `_first_sequence_break` at `audit_chain.py:70`, applied at `audit_chain.py:679`. + +Fix: after the hash walk, if the head row exists, reject the chain unless the stored sequences are exactly 1 through `last_seq`. Reason: `audit_sequence_gap`. A time window does not skip this check. + +Test: `test_renumbered_sequence_gap_is_rejected` and `test_duplicate_sequence_is_rejected`. Before the fix both returned `valid=True`. After the fix, `audit_sequence_gap`. The windowed form of the gap test fails the same way. + +## High: a missing head looked like a complete chain, and the next append split it + +What broke: delete the `audit_chain_heads` row and leave the events. Verify returned valid. The following append treated the wallet as empty, signed the new event as a genesis (no previous hash, seq 1), and stored a new head. The older links then failed verification. + +How to reproduce: record two events, delete the head row, call `verify_audit_chain`, then `record_audit_event` again. + +Root cause: `gateway/app/services/audit_chain.py` ran the tail check only when a head row was present, and `append_chained_audit_event` inserted a new head whenever the select found none (`audit_chain.py:317`). + +Fix: a modern chain (any seq other than 0) with events and no head fails verify with `audit_chain_head_missing` (`audit_chain.py:654`). Append raises `AuditChainHeadMissingError` with that same message and does not insert a row (`audit_chain.py:328`). Chains whose rows are still the old default seq 0, and which have no head, still verify. + +Test: `test_missing_head_on_a_modern_chain_is_rejected` and `test_append_refuses_to_start_a_new_genesis_when_the_head_is_gone`. Before the fix, verify returned `valid=True` and the append did not raise. After the fix, verify returns `audit_chain_head_missing`, the append raises `AuditChainHeadMissingError`, and the event count stays 2. + +## High: a blank wallet id split the wallet-less chain + +What broke: an event with `wallet_id=""` and an event with `wallet_id=None` share one head key (`""`). Verify walked them as two chains. Global verify then reported `audit_chain_truncated` and counted one event. + +How to reproduce: `record_audit_event(..., wallet_id=None)` then `record_audit_event(..., wallet_id="")`, then `verify_audit_chain(wallet_id=None)`. + +Root cause: `append_chained_audit_event` uses `wallet_id or ""` as the head key, while verify filtered `wallet_id IS NULL` and `wallet_id == ""` separately. The shared filter is `_chain_wallet_filter` at `audit_chain.py:57`. Global wallet ids fold `""` into the null chain at `audit_chain.py:436`. + +Fix: NULL and blank wallet ids are one chain on verify, including the predecessor link and the head. Stored wallet ids are unchanged. + +Test: `test_blank_wallet_id_shares_the_wallet_less_chain`. Before the fix, global verify returned `valid=False`, reason `audit_chain_truncated`, `checked_events=1`. After the fix, global verify and `verify_audit_chain(wallet_id="")` both return valid with `checked_events=2`. + +## Tests + +New file: `gateway/tests/test_audit_chain_tamper.py`. + +Before any source change: + +``` +cd gateway && ~/metacode/fleet/heavy .venv/bin/pytest -q tests/test_audit_chain_tamper.py --tb=line -p no:cacheprovider +``` + +Result: `8 failed, 3 passed in 1.74s`. Failures were the sequence gap, the duplicate sequence, all three time windows, the missing head, the append, and the blank wallet. The three passes were middle-link deletion (already rejected), an empty window on a healthy chain, and a seq 0 chain with no head. + +After the fix: + +``` +cd gateway && ~/metacode/fleet/heavy .venv/bin/ruff check app/services/audit_chain.py tests/test_audit_chain_tamper.py +cd gateway && ~/metacode/fleet/heavy .venv/bin/pytest -q tests/test_audit_chain_tamper.py tests/test_audit_chain.py tests/test_audit_snapshot.py tests/test_audit_chain_contention_surface.py --tb=short -p no:cacheprovider +``` + +Ruff: `All checks passed!` + +Pytest: `35 passed in 13.44s`. + +A later run of `tests/test_audit_chain_tamper.py` alone, after the append test was pointed at `AuditChainHeadMissingError`, was `11 passed in 0.44s`. + +## Open questions + +Rewinding the head to an earlier contiguous prefix is still invisible. Delete the tail, set `last_seq` and `last_chain_hash` to the new last event, and leave sequences 1..k untouched. The rows look like the chain stopped there. Seeing that requires an anchor outside this database. This change does not add one. + +A chain that is still all seq 0 and has no head still verifies. The next append on that chain now raises `AuditChainHeadMissingError` instead of signing a new genesis. If any deployment still has those rows, audit writes for that wallet stop until a head is repaired. + +`AuditChainHeadMissingError` is not the contention error (`audit_chain_head_contention`). A request that hits it is not retried as a busy chain. +# Break report: gateway permits + +Summary: 2 breaks found, 2 fixed and tested, 0 fixed but still failing, 0 blocked. Expiry on a live reservation and denied-receipt tampering were probed and held. + +The two breaks are in `gateway/app/services/permits.py`. Local SQLite tests only. No migration. Postgres was not part of this run. + +## 1. High: a bad credit amount can mint budget, wipe it, or crash the check + +What broke. `authorize_and_reserve`, `reserve_budget`, and `POST /v1/permits/verify` accepted a negative `estimated_credits`. On a live permit, reserving `-1` lowered `spent_credits` and the call was reported as allowed, which hands budget back. `NaN` did not get that far: the budget comparison raised `decimal.InvalidOperation` (pre-fix line 867). `release_budget` with a negative amount raised `spent_credits`. `release_budget` with a non-storable amount (`1000000000000`) took the clamp and set `spent_credits` to 0, which clears the reservation. Zero is still a legal amount. + +How to reproduce. Create a live permit, reserve 4, then call `authorize_and_reserve` with `Decimal("-1")`. Before the fix the verdict was allowed and spent fell. The same amount on `reserve_budget` and on `POST /v1/permits/verify` (`estimated_credits` of `"-1"`) was accepted. `Decimal("NaN")` crashed the comparison. `release_budget(Decimal("-1"))` grew spent. `release_budget(Decimal("1000000000000"))` after a reserve of 2 left spent at 0. + +Root cause. `gateway/app/services/permits.py`, the budget comparison inside `_validate_model_for_action`, and the unguarded `spent_credits + amount` / `spent_credits - amount` writes in `authorize_and_reserve`, `reserve_budget`, and `release_budget`. Creation already rejected a non-storable cap. Reservation and release did not. Tool price registration already refused a negative price for this reason. The permit service was still the boundary. + +Fix. `_storable_reservation_amount` (line 115) rejects anything that is not a `Decimal` or that fails `credit_amount_fits_storage` (negative, NaN, Infinity, above the storage limit, bad scale). The read path returns `permit_credits_invalid` before lifecycle checks (line 850), so verify and authorize share it. `reserve_budget` (line 1112) and `release_budget` (line 1268) raise `PermitError("permit_credits_invalid")` before any write. Zero still passes. A storable release larger than the current reservation still clamps to 0. + +Tests. `tests/test_permit_break_it.py`: `test_negative_estimated_credits_cannot_mint_budget`, `test_non_finite_estimated_credits_cannot_reserve`, `test_verify_rejects_negative_estimated_credits`, `test_release_budget_rejects_amounts_that_grow_or_wipe_spend`. All four failed on the pre-fix run and pass now. + +## 2. High: a released receipt stops counting against the aggregate cap + +What broke. `aggregate_value_cap` is a bound on cumulative value. After a positive receipt, `release_budget` puts `spent_credits` back to 0 and leaves the receipt in place. The next reservation checked `max(spent, receipt total) + amount` but stored only `spent + amount`. Once spent numerically caught the old receipt, the floor dropped to 0 and further reservations were bounded by spent alone. Worked case: cap 5, receipt 2, spent released to 0. The first new reservation of 2 was stored as spent 2, not 4. The second reservation of 2 was allowed. Cumulative value was the historical 2 plus 4 of new authority, which is 6, over the cap of 5. Six parallel reservations of 2 admitted 2. The same hole is on `reserve_budget` (the AWI path). A separate case, max credits 3 and cap 100 with the same orphan of 2, allowed a new reservation of 2 because the write never tried to store the orphan. + +How to reproduce. Create a permit with `aggregate_value_cap` 5 and `max_credits` 100. `reserve_budget` 2, write a receipt with `credits_charged` 2 and outcome `failed_unrefunded`, then `release_budget` 2. Spent is 0. Call `authorize_and_reserve` twice for 2. Before the fix both calls were allowed and the first left spent at 2. The concurrent form is six parallel reservations of 2: before the fix, 2 were allowed. + +Root cause. `gateway/app/services/permits.py`, the reservation UPDATE in `authorize_and_reserve` and `reserve_budget`. The cap predicate added a Python floor constant. The SET clause was only `spent_credits + amount`, so a successful reservation did not keep the orphan in the counter. The service comment described that later check as the same arithmetic as a receipt that was never released. It was not: a never-released charge of 2 plus one new reservation of 2 leaves spent at 4 and denies the next 2. + +Fix. `_reservation_charge_expr` (line 1056) puts the receipt sum inside the UPDATE. When a cap is set, the statement adds `max(receipt sum - spent, 0)` and uses that same total in the cap predicate and the `max_credits` predicate. A successful reservation of 2 on the example above stores 4. The next reservation of 2 is denied, reason `permit_aggregate_value_cap_exceeded`, and spent stays 4. The read path counts the same floor against both the cap and `max_credits`, so verify does not allow a call the write must refuse. After a 0-row update the floor is computed again from the refreshed row, so a denial is cap or budget rather than a retry loop. In-flight release with no receipt still frees the cap: the floor is 0 and the write matches the old `spent + amount` form. Alert thresholds on `reserve_budget`, when a cap is set, use the spent value observed before the write, because the new counter can rise by more than `amount`. + +Tests. `tests/test_permit_break_it.py`: `test_released_receipt_history_is_absorbed_on_the_next_reservation`, `test_reserve_budget_absorbs_released_receipt_history`, `test_concurrent_reservations_cannot_refill_a_released_receipt`, `test_absorption_cannot_push_spent_past_max_credits`. All four failed on the pre-fix run (spent stayed 2, the second reserve was allowed, the race admitted 2, the tight max credits case was allowed). They pass now. `tests/test_permit_v2_constraints.py` had locked the loose result (a second reserve allowed, and the race allowed 1 or 2). Those two tests now expect one admission and spent 4. The no-receipt cap tests in that file were not loosened and still pass, including release-then-reserve and the parallel cap with no receipt history. + +Production path checked, not changed. The MCP refund path that calls `release_budget` then signs a receipt uses `credits_charged` 0 (`_finalize_governed_denial`). The upstream refund release followed by that same denial receipt is also 0. Those paths do not leave a positive orphan. The service itself still accepts the state, and the cap tests build it with `release_budget` after a positive receipt. I did not find a current caller that does that in production code. The break is still in the permit service the cap claims to enforce. + +## Probed, not a break: expired permit + +`authorize_and_reserve` and `reserve_budget` already put `expires_at > now` on the reservation UPDATE and classify `permit_expired` before budget. `test_expired_permit_cannot_be_reserved` backdates `expires_at` by 5 minutes, leaves status `active`, and reserves 1. Both paths return `permit_expired` and spent stays 0. That part passed before the fix. The same test then sends `Decimal("-1")` on the expired permit. Before the fix the reason was `permit_expired` and spent did not move (the deadline check returned before the write). After the fix the reason is `permit_credits_invalid`, and spent still does not move. Replay access still ignores expiry, revocation, and remaining budget on purpose. That was not changed. + +## Probed, not a break: denied receipt signing + +`test_denied_receipt_signature_rejects_outcome_and_charge_tamper` signs a denial with `credits_charged` 0, `credits_authorized` 1.10, and reason `permit_expired`. Verify returns valid. Flipping the stored outcome to `success` makes verify return `receipt_signature_invalid`. Restoring the outcome and setting `credits_charged` to 5 does the same. This passed before any code change. Existing coverage in `tests/test_receipts.py` already checks reason-code tamper. `_finalize_governed_denial` in `gateway/app/routers/mcp.py` still omits `constraints_evaluated` on denial receipts. That is an evidence gap, not a signature that accepts a rewritten outcome. MCP was not edited. + +## Tests run + +Pre-fix, before any edit to `permits.py`: + +``` +cd gateway && ~/metacode/fleet/heavy .venv/bin/pytest -q tests/test_permit_break_it.py --tb=line +``` + +9 failed, 2 passed in 1.44s. The two that passed were the zero-credit reservation and the denied-receipt tamper. The nine failures are the breaks above, plus the expired-permit test's negative amount (it wanted `permit_credits_invalid` and got `permit_expired`). + +After the fix, final tree: + +``` +cd gateway && ~/metacode/fleet/heavy .venv/bin/pytest -q tests/test_permit_break_it.py tests/test_permit_v2_constraints.py tests/test_permit_budget_integrity.py --tb=line +``` + +40 passed in 3.04s. + +``` +cd gateway && ~/metacode/fleet/heavy bash -c '.venv/bin/ruff check app/services/permits.py tests/test_permit_break_it.py tests/test_permit_v2_constraints.py && .venv/bin/pytest -q tests/test_permits.py tests/test_denial_details_and_self_service.py tests/test_receipts.py tests/test_permit_budget_drift_visibility.py tests/test_permit_verification_sessions.py tests/test_quickstart_path.py --tb=line -x' +``` + +Ruff: all checks passed. Pytest: 74 passed in 10.42s. + +The full gateway suite was not run. + +## Open questions + +- Should a denial receipt from `_finalize_governed_denial` include `constraints_evaluated` the way success receipts do? Verify still holds without it. Left unchanged. +- Is any caller outside MCP, x402, and AWI releasing a reservation after a receipt with a positive `credits_charged`? Those three paths that were read write 0 on the refund receipt, or they do not release once a receipt exists. The service fix covers the state anyway. +- Postgres was not used. The suite here is SQLite. The new cap write is a correlated sum inside the UPDATE. On SQLite that sum is a float. Amounts in these tests were integers and matched. A sum of many fractional credits could disagree with the Decimal read and surface as `permit_write_contended` instead of a clean cap denial. Not reproduced here. +- Zero-credit authorize still succeeds and does not change spent. A free call can still consume a `max_calls_per_tool` slot. +- A storable `release_budget` larger than current spent still clamps to 0. That is the existing drift behavior for a real refund amount. + +--- + +# Break Report: gateway/failure_lab (Oct 8, 2026) + +## Summary + +Probed the fail-closed surfaces of `gateway/failure_lab` with 8 adversarial +probes (bad observation counts, mismatched observations, malformed operation +identifiers, oversized bodies, type coercions). Found 5 breaks. Fixed 4 with +root-cause patches plus regression tests, all passing. Fixed but failing: 0. +Blocked / left open: 1 (observation binding without identifiers, needs a +contract decision, see finding 5). + +All work is local only. No production config, secrets, migrations, or network +calls were involved. No database needed: every fix is in pure functions +covered by unit tests. + +## Findings (by severity) + +### 1. High: a boolean `True` observation established downstream execution + +What broke: `verify()` with +`downstream_observation={"executions": True, ...}` reported +`DOWNSTREAM_EXECUTION_ESTABLISHED`. In Python `bool` is a subclass of `int`, +so the `isinstance(executions, int)` guard accepted it and `True > 0` +counted as one execution. + +How to reproduce: sign any receipt, call +`verify(bundle, keys, downstream_observation={"executions": True, ...})`, +read the downstream claim. + +Root cause: `gateway/failure_lab/verifier.py`, `_downstream_claim` +(execution-count validation, formerly around line 378). + +Fix: reject booleans explicitly (`isinstance(executions, bool)` fails the +observation as unusable, status NOT_ESTABLISHED). Same for any non-int type. + +Test results: new test `test_a_boolean_is_not_an_execution_count` +(parametrized True/False) in +`gateway/tests/test_failure_lab_verifier.py`. Confirmed it fails on pre-fix +code (old code returned ESTABLISHED for True) via a read-only +`git show HEAD:` comparison script. Passes on fixed code. + +### 2. Medium: an impossible negative execution count was reported as consistent + +What broke: `{"executions": -4}` with a non-success outcome returned +NOT_ESTABLISHED with reason text calling the observation "consistent with" +the outcome. Minus four executions cannot happen, so blessing it as +consistent is a false clean bill for the instrument. + +How to reproduce: `verify()` with `executions=-4`, outcome +`failed_refunded` or `delivery_uncertain`, read the reason string. + +Root cause: same function as finding 1, no lower-bound validation. + +Fix: negative counts return NOT_ESTABLISHED with reason "the supplied +observation reported an impossible execution count". An impossible reading +is an instrument error, not evidence for or against execution. + +Test results: new test +`test_a_negative_count_is_an_unusable_observation_not_consistency`. +Confirmed old code emitted "consistent" wording; passes on fixed code. + +### 3. Medium: an observation naming a different receipt established execution + +What broke: `verify()` never compared any identifier between the +observation and the signed payload, so an observation collected for receipt +B (with executions > 0) marked receipt A as executed. + +How to reproduce: verify receipt A with an observation carrying +`receipt_id` of receipt B and `executions=1`. Old result: ESTABLISHED. + +Root cause: same function; the documented observation shape carries +`operation_id` but the signed payload has no operation field to compare, so +nothing was ever cross-checked. + +Fix: when the observation carries a non-empty string `receipt_id` and the +signed payload names a different one, the claim FAILS with a reason naming +both receipts. Matching or absent identifiers behave exactly as before, so +no existing caller (including scenario T10, whose observation carries no +receipt_id) changes status. The `verify()` docstring now documents the +binding rule and recommends callers always bind observations. + +Test results: new tests `test_an_observation_of_another_receipt_fails_the_claim` +and `test_an_observation_bound_to_this_receipt_still_establishes`. +Confirmed old code returned ESTABLISHED for the mismatch; both pass fixed. + +### 4. Medium: non-string operation ids were coerced into plausible identifiers + +What broke: `classify_request()` applied `str()` to whatever the request +carried as `operation_id`, so a dict, number, or bool became a +plausible-looking string (for example `"{'nested': True}"`). That value +matches no fault plan, so an injected fault would silently never fire while +the crossing record reads as normally observed. + +How to reproduce: feed a `tools/call` (or `/refunds`) body with +`"operation_id": {"nested": true}` to `classify_request`. Old result: +`(kind, "{'nested': True}")`. + +Root cause: `gateway/failure_lab/faults.py`, `classify_request`, both the +MCP and REST branches. + +Fix: accept the value only when it is already a string, else record `None` +(visibly unidentified). The real gateway only ever sends strings +(`StrictStr` in the refund tool), so no legitimate traffic changes. + +Test results: new `gateway/tests/test_failure_lab_failclosed.py` with +parametrized bad-type cases for both paths plus a positive control. +Confirmed old code coerced; passes fixed. + +### 5. Low (open, needs a decision): observations without identifiers are taken on trust + +What did not break but is weak by design: an observation carrying no +`receipt_id` (the documented shape has only executions, operation_id, +source) is assumed to describe the receipt being verified. Finding 3 closes +the case where binding info is present but ignored; it cannot close the +case where no binding info exists, because the signed payload carries no +operation identifier to compare against (only receipt_id, request_hash, and +dispatch linkage). Closing it fully means adding `receipt_id` (or +`request_hash`) to the observation contract and threading it through every +scenario that builds one, starting with T10. + +No code changed for this. Left as an open question for C.Lee below. + +### 6. Low: oversized fault-layer bodies raised instead of refusing cleanly + +What broke: a POST body over the 1 MiB buffer raised raw `ValueError` out +of `FaultInjectionMiddleware`, surfacing to the gateway as an +unclassifiable transport error instead of a clean refusal. The tool was +never touched (fail-closed in effect), but the failure mode was a crash, +not a refusal, and nothing recorded the refusal. + +How to reproduce: POST more than `_MAX_BODY_BYTES` to `/mcp` through +`FaultInjectionMiddleware`. Old behavior: `ValueError` propagates. + +Root cause: `gateway/failure_lab/faults.py`, +`FaultInjectionMiddleware.__call__` called `_read_body` unguarded. + +Fix: catch `ValueError`, record a crossing (`fault="oversized_body_rejected"`, +`reached_tool=False`, `delivered=True`, kind `unclassified` so it never +pollutes dispatch counts), and answer 413 with a JSON error. This mirrors +the diagnostic server, which already refuses oversized submissions with 413 +(see `test_an_oversized_submission_is_refused`). + +Test results: new test +`test_oversized_body_is_refused_as_413_without_reaching_the_tool` in the new +file. Confirmed old code raised `ValueError`; passes fixed. + +## What was probed and held + +- Float (`1.0`) and string (`"1"`) execution counts: already rejected. +- `False` count on a success receipt: already FAILED the claim correctly. +- Unsigned envelope edits: still verify, and are reported as disagreements. +- Envelope `kid` relabelling: cannot steer key selection (signed bytes win). +- Unknown signed `kid`, disabled keys, key-length checks: all fail closed. +- Redaction and secret plumbing (`secret_values` carries admin key, bearer, + control token plus environment secrets): complete, no change needed. +- Retired signing keys staying published: deliberate (historical receipts + stay verifiable), not a bug. +- Scenario verdict logic (T02, T09, T10): guards against empty probe loops + and ledger reconciliation already present, no change needed. + +## Test summary + +Commands run from `gateway/` through the slot queue: + +- `python -m pytest tests/test_failure_lab_verifier.py tests/test_failure_lab_failclosed.py -q`: 27 passed +- `python -m pytest ...verifier ...failclosed ...evidence ...report ...claims ...harness ...portable_keys -q`: 90 passed +- `python -m pytest ...verifier ...failclosed ...scenarios -q`: 86 passed +- `ruff check failure_lab/ tests/test_failure_lab_failclosed.py tests/test_failure_lab_verifier.py`: all checks passed +- `ruff format` applied to `failure_lab/faults.py`; all touched files format-clean +- No-fix confirmation: new assertions executed against pre-fix modules + extracted read-only via `git show HEAD:`; all 5 failed there and pass + with the fix. + +Full gateway suite was not run (time and shared-machine load); scenario +integration runs needing the app and databases were not run beyond the +`test_failure_lab_scenarios` file, which passed. + +## Open questions + +1. Should the downstream-observation contract require `receipt_id` + (finding 5), with T10 and any other builders updated to supply it? + Recommend yes, as a follow-up task. +2. No migration, no config, no API changes in this work. Nothing needs a + decision to land it. diff --git a/gateway/app/services/audit_chain.py b/gateway/app/services/audit_chain.py index e9c19d94..bced1246 100644 --- a/gateway/app/services/audit_chain.py +++ b/gateway/app/services/audit_chain.py @@ -6,7 +6,7 @@ from datetime import datetime from typing import Any, cast -from sqlalchemy import asc, desc, literal, select, true, update +from sqlalchemy import asc, desc, literal, or_, select, true, update from sqlalchemy.engine import CursorResult from sqlalchemy.exc import IntegrityError, OperationalError from sqlalchemy.sql.elements import ColumnElement @@ -102,6 +102,41 @@ def _payload_matching_stored_hash( return None +def _chain_wallet_filter(wallet_id: str | None) -> ColumnElement[bool]: + """Rows that belong to one wallet chain. + + The head table stores "" for a missing wallet id. A blank string is + the same chain as NULL. Walking those rows apart made each half look + truncated or unlinked. + """ + column = cast(ColumnElement[str | None], ControlPlaneAuditEventModel.wallet_id) + if wallet_id: + return cast(ColumnElement[bool], column == wallet_id) + return cast(ColumnElement[bool], or_(column.is_(None), column == "")) + + +def _first_sequence_break( + events: list[ControlPlaneAuditEventModel], +) -> str | None: + """Event id where a run of seq-signed rows stops being contiguous. + + Legacy signatures omit ``seq``, so their stored numbers cannot establish + contiguity within a legacy run. A following seq-signed row must continue + from the final legacy row's stored sequence. ``events`` must already be + ordered by seq, then created_at. + """ + expected_seq: int | None = 1 + for event in events: + payload = _payload_matching_stored_hash(event) + if payload is not None and "seq" not in payload: + expected_seq = event.seq + 1 + continue + if expected_seq is not None and event.seq != expected_seq: + return event.event_id + expected_seq = event.seq + 1 + return None + + def _sign_with_previous( model: ControlPlaneAuditEventModel, previous_hash: str | None, @@ -149,12 +184,7 @@ async def sign_audit_model(model: ControlPlaneAuditEventModel) -> None: async with factory() as session: result = await session.execute( select(ControlPlaneAuditEventModel) - .where( - cast( - ColumnElement[bool], - ControlPlaneAuditEventModel.wallet_id == model.wallet_id, - ) - ) + .where(_chain_wallet_filter(model.wallet_id)) .order_by( desc(cast(ColumnElement[Any], ControlPlaneAuditEventModel.seq)), desc(cast(ColumnElement[Any], ControlPlaneAuditEventModel.created_at)), @@ -175,6 +205,16 @@ class AuditEventConflictError(RuntimeError): """A caller reused an audit event identity for different signed evidence.""" +class AuditChainHeadMissingError(RuntimeError): + """Events are stored for this chain but the head row is gone. + + Appending would sign a new genesis (empty previous hash, seq 1) and + split the links that are already there. Callers get this instead. + """ + + reason = "audit_chain_head_missing" + + class AuditChainContendedError(RuntimeError): """The per-wallet chain head stayed contended for the whole retry budget. @@ -298,6 +338,47 @@ async def append_chained_audit_event( ).first() observed_seq = row[0] if row else 0 previous_hash = row[1] if row else None + if row is None: + # No head does not mean an empty chain. A genesis signed + # over rows that are already stored splits the hash links. + terminal = ( + await session.execute( + select( + cast(Any, ControlPlaneAuditEventModel.seq), + cast(Any, ControlPlaneAuditEventModel.chain_hash), + ) + .where(_chain_wallet_filter(model.wallet_id)) + .order_by( + desc( + cast( + ColumnElement[Any], + ControlPlaneAuditEventModel.seq, + ) + ), + desc( + cast( + ColumnElement[Any], + ControlPlaneAuditEventModel.created_at, + ) + ), + ) + .limit(1) + ) + ).first() + if terminal is not None: + # A first writer may have committed between the head + # and event reads. Only that race warrants a retry. + if await session.get(AuditChainHeadModel, wallet_key): + raise _HeadConflict() + if terminal[0] != 0: + # A seq-signed row exists, so a head was written + # and has since gone. Refuse rather than guess. + raise AuditChainHeadMissingError("audit_chain_head_missing") + # Every stored row predates seq (migration 017 set + # them to 0) and no head was ever written. Continue + # the legacy chain from its terminal link at seq 1, + # which verification accepts after a legacy run. + previous_hash = terminal[1] model.seq = observed_seq + 1 _sign_with_previous(model, previous_hash, signing_key_id=key.key_id) now = utc_now() @@ -403,7 +484,11 @@ async def verify_audit_chain( wallet_ids_result = await session.execute( select(cast(Any, ControlPlaneAuditEventModel.wallet_id)).distinct() ) - event_wallets = {row[0] for row in wallet_ids_result.all()} + # Blank wallet ids share the "" head with NULL. Fold them into + # one entry so this walk does not count that chain twice. + event_wallets = { + (None if not row[0] else row[0]) for row in wallet_ids_result.all() + } # Also enumerate wallets from the chain-head table. Deriving the list # from events alone let an attacker who deleted ALL of a wallet's # events erase that wallet from the check entirely: with no events it @@ -455,108 +540,73 @@ async def _verify_single_chain( created_after: datetime | None = None, created_before: datetime | None = None, ) -> AuditChainVerification: - """Verify one wallet's audit chain (``wallet_id=None`` -> the wallet-less - chain of ``wallet_id IS NULL`` events).""" + """Verify one wallet's audit chain. + + ``wallet_id=None`` and ``wallet_id=""`` are the wallet-less chain + (NULL and blank ids together). + """ created_after = to_naive_utc(created_after) if created_after else None created_before = to_naive_utc(created_before) if created_before else None - stmt = select(ControlPlaneAuditEventModel).order_by( - asc(cast(ColumnElement[Any], ControlPlaneAuditEventModel.seq)), - asc(cast(ColumnElement[Any], ControlPlaneAuditEventModel.created_at)), - ) - if wallet_id is None: - stmt = stmt.where(cast(Any, ControlPlaneAuditEventModel.wallet_id).is_(None)) - else: - stmt = stmt.where( - cast( - ColumnElement[bool], ControlPlaneAuditEventModel.wallet_id == wallet_id - ) - ) - if created_after: - stmt = stmt.where( - cast( - ColumnElement[bool], - ControlPlaneAuditEventModel.created_at >= created_after, - ) - ) - if created_before: - stmt = stmt.where( - cast( - ColumnElement[bool], - ControlPlaneAuditEventModel.created_at <= created_before, - ) + stmt = ( + select(ControlPlaneAuditEventModel) + .where(_chain_wallet_filter(wallet_id)) + .order_by( + asc(cast(ColumnElement[Any], ControlPlaneAuditEventModel.seq)), + asc(cast(ColumnElement[Any], ControlPlaneAuditEventModel.created_at)), ) + ) factory = get_session_factory() async with factory() as session: - head = None - if created_after is None and created_before is None: - # One statement gives events and head the same database snapshot. - # A separate head SELECT could see a legitimate concurrent append - # and label the earlier event snapshot as a truncated chain. - # Anchor the outer joins to one row so an empty/deleted event set - # still loads its head and actual truncation remains detectable. - anchor = select(literal(1).label("anchor")).subquery() - event_rows = aliased(ControlPlaneAuditEventModel, stmt.subquery()) - head_rows = aliased( - AuditChainHeadModel, - select(AuditChainHeadModel) - .where( - cast( - ColumnElement[bool], - AuditChainHeadModel.wallet_key == (wallet_id or ""), - ) - ) - .subquery(), - ) - snapshot = await session.execute( - select(event_rows, head_rows) - .select_from(anchor) - .outerjoin(event_rows, true()) - .outerjoin(head_rows, true()) - .order_by( - cast(ColumnElement[Any], event_rows.seq), - cast(ColumnElement[Any], event_rows.created_at), - ) - ) - rows = snapshot.all() - events = [row[0] for row in rows if row[0] is not None] - head = rows[0][1] if rows else None - else: - result = await session.execute(stmt) - events = list(result.scalars().all()) - - # A window that excludes the wallet's genesis event starts mid-chain: the - # first in-window event legitimately links to an event *outside* the - # window. Seed the expected predecessor from the last stored event before - # the window instead of from genesis, otherwise every non-genesis window - # reports a valid chain as tampered (``audit_previous_hash_mismatch``). - previous_hash: str | None = None - if events and created_after: - first_seq = events[0].seq - pred_stmt = ( - select(cast(Any, ControlPlaneAuditEventModel.chain_hash)) + # One statement gives events and head the same database snapshot. + # A separate head SELECT could see a legitimate concurrent append + # and label the earlier event snapshot as a truncated chain. + # Time filters used to skip this load, so a wide window hid a + # deleted tail. Apply those filters in memory after the snapshot. + # Anchor the outer joins to one row so an empty or deleted event + # set still loads its head and truncation remains detectable. + anchor = select(literal(1).label("anchor")).subquery() + event_rows = aliased(ControlPlaneAuditEventModel, stmt.subquery()) + head_rows = aliased( + AuditChainHeadModel, + select(AuditChainHeadModel) .where( - cast(ColumnElement[bool], ControlPlaneAuditEventModel.seq < first_seq) - ) - .order_by(desc(cast(ColumnElement[Any], ControlPlaneAuditEventModel.seq))) - .limit(1) - ) - if wallet_id is None: - pred_stmt = pred_stmt.where( - cast(Any, ControlPlaneAuditEventModel.wallet_id).is_(None) - ) - else: - pred_stmt = pred_stmt.where( cast( ColumnElement[bool], - ControlPlaneAuditEventModel.wallet_id == wallet_id, + AuditChainHeadModel.wallet_key == (wallet_id or ""), ) ) - async with factory() as session: - previous_hash = (await session.execute(pred_stmt)).scalar_one_or_none() + .subquery(), + ) + snapshot = await session.execute( + select(event_rows, head_rows) + .select_from(anchor) + .outerjoin(event_rows, true()) + .outerjoin(head_rows, true()) + .order_by( + cast(ColumnElement[Any], event_rows.seq), + cast(ColumnElement[Any], event_rows.created_at), + ) + ) + rows = snapshot.all() + all_events = [row[0] for row in rows if row[0] is not None] + head = rows[0][1] if rows else None + + if created_after is not None or created_before is not None: + events = [ + event + for event in all_events + if (created_after is None or event.created_at >= created_after) + and (created_before is None or event.created_at <= created_before) + ] + else: + events = all_events + + previous_hash: str | None = None first_event_id = events[0].event_id if events else None last_event_id = events[-1].event_id if events else None - for event in events: + window_event_ids = {event.event_id for event in events} + for event in all_events: payload = _payload_matching_stored_hash(event) if payload is None: return AuditChainVerification( @@ -576,35 +626,36 @@ async def _verify_single_chain( "audit_previous_hash_mismatch", event.event_id, ) - if not event.signature or not event.signature_key_id: - return AuditChainVerification( - False, - len(events), - first_event_id, - last_event_id, - "audit_signature_missing", - event.event_id, - ) - signed_payload = { - **payload, - "payload_hash": event.payload_hash, - "alg": "Ed25519", - "kid": event.signature_key_id, - } - ok = await get_signing_key_service().verify_payload( - signed_payload, - signature=event.signature, - key_id=event.signature_key_id, - ) - if not ok: - return AuditChainVerification( - False, - len(events), - first_event_id, - last_event_id, - "audit_signature_invalid", - event.event_id, + if event.event_id in window_event_ids: + if not event.signature or not event.signature_key_id: + return AuditChainVerification( + False, + len(events), + first_event_id, + last_event_id, + "audit_signature_missing", + event.event_id, + ) + signed_payload = { + **payload, + "payload_hash": event.payload_hash, + "alg": "Ed25519", + "kid": event.signature_key_id, + } + ok = await get_signing_key_service().verify_payload( + signed_payload, + signature=event.signature, + key_id=event.signature_key_id, ) + if not ok: + return AuditChainVerification( + False, + len(events), + first_event_id, + last_event_id, + "audit_signature_invalid", + event.event_id, + ) expected_chain_hash = sha256_hex( { "previous_hash": event.previous_hash, @@ -623,23 +674,45 @@ async def _verify_single_chain( ) previous_hash = event.chain_hash - # Tail-truncation check: the last verified event must match the head the - # append path recorded. If the head says seq=N/chain_hash=H but the loaded - # chain ends earlier (or is empty), the tail was deleted. - if head is not None: - last_event = events[-1] if events else None + # The head and sequence also belong to the whole chain. Skipping them + # when the head row is gone lets a deleted tail or renumbered seq verify. + anchor_first = all_events[0].event_id if all_events else None + anchor_last = all_events[-1].event_id if all_events else None + if head is None: + if any(event.seq != 0 for event in all_events): + broken = next(event for event in all_events if event.seq != 0) + return AuditChainVerification( + False, + len(all_events), + anchor_first, + anchor_last, + "audit_chain_head_missing", + broken.event_id, + ) + else: + last_event = all_events[-1] if all_events else None if last_event is None or ( last_event.seq != head.last_seq or last_event.chain_hash != head.last_chain_hash ): return AuditChainVerification( False, - len(events), - first_event_id, - last_event_id, + len(all_events), + anchor_first, + anchor_last, "audit_chain_truncated", last_event.event_id if last_event else None, ) + broken_id = _first_sequence_break(all_events) + if broken_id is not None: + return AuditChainVerification( + False, + len(all_events), + anchor_first, + anchor_last, + "audit_sequence_gap", + broken_id, + ) return AuditChainVerification( True, diff --git a/gateway/tests/test_audit_chain.py b/gateway/tests/test_audit_chain.py index 0534dc42..2a687e59 100644 --- a/gateway/tests/test_audit_chain.py +++ b/gateway/tests/test_audit_chain.py @@ -694,8 +694,7 @@ async def test_windowed_verify_of_valid_chain_is_valid(client, clean_database): assert windowed.valid is True, windowed assert windowed.checked_events == 2 - # The seeded predecessor is really checked: tampering the chain_hash of - # the last event *outside* the window breaks the first in-window link. + # The full chain walk checks the hash of an event outside the window. async with factory() as session: event = ( await session.execute( @@ -712,4 +711,5 @@ async def test_windowed_verify_of_valid_chain_is_valid(client, clean_database): created_after=third_created - timedelta(microseconds=1), ) assert tampered.valid is False - assert tampered.reason == "audit_previous_hash_mismatch" + assert tampered.reason == "audit_chain_hash_mismatch" + assert tampered.broken_event_id == rows[1].event_id diff --git a/gateway/tests/test_audit_chain_tamper.py b/gateway/tests/test_audit_chain_tamper.py new file mode 100644 index 00000000..2fdb9434 --- /dev/null +++ b/gateway/tests/test_audit_chain_tamper.py @@ -0,0 +1,578 @@ +"""Adversarial checks for audit hash-chain tamper and sequence gaps. + +These tests mutate stored rows the way a database writer can: delete a +link, renumber seq, drop the head, or split the wallet-less chain. They +do not call the network. +""" + +from __future__ import annotations + +from datetime import datetime, timedelta +from uuid import uuid4 + +import pytest +from sqlalchemy import select + +from app.core.time import utc_now +from app.db.database import get_session_factory +from app.db.models import AuditChainHeadModel, ControlPlaneAuditEventModel +from app.services.audit_chain import ( + AuditChainContendedError, + AuditChainHeadMissingError, + _sign_with_previous, + verify_audit_chain, +) +from app.services.audit_log import record_audit_event +from app.services.signing_keys import get_signing_key_service, sha256_hex +from tests.test_audit_chain import _payload_without_seq + + +@pytest.mark.anyio +async def test_middle_deletion_breaks_the_hash_link(clean_database): + """Removing a middle link must not leave a valid chain. + + The survivor's previous hash still names the deleted link, so the + walk cannot close. This was already rejected before the sequence + check; the test pins that reason. + """ + wallet_id = "wlt-middle-delete" + for n in range(3): + await record_audit_event( + event="trust.middle", wallet_id=wallet_id, metadata={"n": n} + ) + + factory = get_session_factory() + async with factory() as session: + middle = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by(ControlPlaneAuditEventModel.seq) + ) + ) + .scalars() + .all() + )[1] + await session.delete(middle) + await session.commit() + + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is False + assert result.reason == "audit_previous_hash_mismatch" + + +@pytest.mark.anyio +@pytest.mark.parametrize("deleted_seqs", [(4,), (3,), (3, 4), (2,)]) +async def test_windowed_verify_finds_deleted_legacy_and_modern_links( + clean_database, deleted_seqs +): + wallet_id = f"wlt-{uuid4()}" + base = utc_now() + for seq in range(1, 7): + await record_audit_event( + event="trust.window-link", + wallet_id=wallet_id, + created_at=base + timedelta(seconds=seq), + metadata={"seq": seq}, + ) + + factory = get_session_factory() + async with factory() as session: + rows = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by(ControlPlaneAuditEventModel.seq) + ) + ) + .scalars() + .all() + ) + assert [row.seq for row in rows] == list(range(1, 7)) + previous_hash = None + for row in rows[:3]: + row.previous_hash = previous_hash + payload = _payload_without_seq(row) + row.payload_hash = sha256_hex(payload) + payload["payload_hash"] = row.payload_hash + assert row.signature_key_id is not None + row.signature, row.signature_key_id, _ = ( + get_signing_key_service().sign_payload_with_key_id( + payload, row.signature_key_id + ) + ) + row.chain_hash = sha256_hex( + { + "previous_hash": previous_hash, + "payload_hash": row.payload_hash, + "signature": row.signature, + } + ) + previous_hash = row.chain_hash + for row in rows[3:]: + assert row.signature_key_id is not None + _sign_with_previous(row, previous_hash, signing_key_id=row.signature_key_id) + previous_hash = row.chain_hash + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + head.last_chain_hash = previous_hash + await session.commit() + + window = {"created_after": rows[-1].created_at} + intact = await verify_audit_chain(wallet_id=wallet_id, **window) + assert intact.valid is True, intact + assert intact.checked_events == 1 + assert intact.first_event_id == intact.last_event_id == rows[-1].event_id + + async with factory() as session: + for row in rows: + if row.seq in deleted_seqs: + stored = await session.get(ControlPlaneAuditEventModel, row.event_id) + assert stored is not None + await session.delete(stored) + await session.commit() + + result = await verify_audit_chain(wallet_id=wallet_id, **window) + assert result.valid is False + assert result.reason == "audit_previous_hash_mismatch" + assert result.broken_event_id == rows[max(deleted_seqs)].event_id + + if deleted_seqs == (3,): + # A writer can repair linkage and re-sign the modern suffix. The + # first seq-signed row must still reveal the gap after legacy rows. + async with factory() as session: + survivors = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by(ControlPlaneAuditEventModel.seq) + ) + ) + .scalars() + .all() + ) + previous_hash = survivors[1].chain_hash + for row in survivors[2:]: + assert row.signature_key_id is not None + _sign_with_previous( + row, previous_hash, signing_key_id=row.signature_key_id + ) + previous_hash = row.chain_hash + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + head.last_chain_hash = previous_hash + await session.commit() + + relinked = await verify_audit_chain(wallet_id=wallet_id, **window) + assert relinked.valid is False + assert relinked.reason == "audit_sequence_gap" + assert relinked.broken_event_id == rows[3].event_id + + +@pytest.mark.anyio +async def test_renumbered_sequence_gap_is_rejected(clean_database): + """Renumbering breaks a signature; a signed gap breaks contiguity.""" + wallet_id = "wlt-seq-gap" + for n in range(3): + await record_audit_event( + event="trust.gap", wallet_id=wallet_id, metadata={"n": n} + ) + + factory = get_session_factory() + async with factory() as session: + rows = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by(ControlPlaneAuditEventModel.seq) + ) + ) + .scalars() + .all() + ) + rows[-1].seq += 1 + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + head.last_seq = rows[-1].seq + session.add(rows[-1]) + session.add(head) + await session.commit() + + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is False + assert result.reason == "audit_payload_hash_mismatch" + + async with factory() as session: + rows = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by(ControlPlaneAuditEventModel.seq) + ) + ) + .scalars() + .all() + ) + tail = rows[-1] + assert tail.signature_key_id is not None + # Seq-less signatures are exempt; only a valid seq-signed gap reaches + # the contiguity check. + _sign_with_previous( + tail, rows[-2].chain_hash, signing_key_id=tail.signature_key_id + ) + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + head.last_chain_hash = tail.chain_hash + await session.commit() + + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is False + assert result.reason == "audit_sequence_gap" + assert result.broken_event_id is not None + + windowed = await verify_audit_chain( + wallet_id=wallet_id, + created_after=datetime(2000, 1, 1), + ) + assert windowed.valid is False + assert windowed.reason == "audit_sequence_gap" + + +@pytest.mark.anyio +async def test_duplicate_sequence_is_rejected(clean_database): + """A duplicate breaks its signature and, if re-signed, contiguity.""" + wallet_id = "wlt-dup-seq" + base = utc_now() + for n in range(3): + await record_audit_event( + event="trust.dup", + wallet_id=wallet_id, + created_at=base + timedelta(seconds=n), + metadata={"n": n}, + ) + + factory = get_session_factory() + async with factory() as session: + rows = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by(ControlPlaneAuditEventModel.seq) + ) + ) + .scalars() + .all() + ) + rows[-1].seq = rows[-2].seq + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + head.last_seq = rows[-1].seq + head.last_chain_hash = rows[-1].chain_hash + session.add(rows[-1]) + session.add(head) + await session.commit() + + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is False + assert result.reason == "audit_payload_hash_mismatch" + + async with factory() as session: + rows = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by( + ControlPlaneAuditEventModel.seq, + ControlPlaneAuditEventModel.created_at, + ) + ) + ) + .scalars() + .all() + ) + tail = rows[-1] + assert tail.signature_key_id is not None + _sign_with_previous( + tail, rows[-2].chain_hash, signing_key_id=tail.signature_key_id + ) + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + head.last_chain_hash = tail.chain_hash + await session.commit() + + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is False + assert result.reason == "audit_sequence_gap" + + +@pytest.mark.anyio +@pytest.mark.parametrize( + "window", + [ + {"created_after": datetime(2000, 1, 1)}, + {"created_before": datetime(2099, 1, 1)}, + { + "created_after": datetime(2000, 1, 1), + "created_before": datetime(2099, 1, 1), + }, + ], +) +async def test_time_window_does_not_hide_a_deleted_tail(clean_database, window): + """A wide time window must not skip the head anchor. + + Deleting the tail leaves a hash-valid prefix. Verification used to + load the head only when both time filters were absent, so any window + that still covered the survivors returned valid. + """ + wallet_id = "wlt-window-tail" + for n in range(3): + await record_audit_event( + event="trust.windowtail", wallet_id=wallet_id, metadata={"n": n} + ) + + factory = get_session_factory() + async with factory() as session: + last = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by(ControlPlaneAuditEventModel.seq.desc()) + ) + ) + .scalars() + .first() + ) + await session.delete(last) + await session.commit() + + plain = await verify_audit_chain(wallet_id=wallet_id) + assert plain.valid is False + assert plain.reason == "audit_chain_truncated" + + windowed = await verify_audit_chain(wallet_id=wallet_id, **window) + assert windowed.valid is False + assert windowed.reason == "audit_chain_truncated" + + +@pytest.mark.anyio +async def test_window_that_excludes_every_event_keeps_a_valid_chain(clean_database): + """An empty window is not a truncated chain when the full chain is intact.""" + wallet_id = "wlt-window-empty" + await record_audit_event(event="trust.keep", wallet_id=wallet_id) + result = await verify_audit_chain( + wallet_id=wallet_id, + created_before=datetime(2000, 1, 1), + ) + assert result.valid is True + assert result.checked_events == 0 + + +@pytest.mark.anyio +async def test_missing_head_on_a_modern_chain_is_rejected(clean_database): + """Dropping the head row must not make the remaining events look complete. + + The head is the anchor for tail deletion. With the row gone, the old + verifier skipped that check and returned valid. + """ + wallet_id = "wlt-head-gone" + for n in range(2): + await record_audit_event( + event="trust.head", wallet_id=wallet_id, metadata={"n": n} + ) + + factory = get_session_factory() + async with factory() as session: + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + await session.delete(head) + await session.commit() + + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is False + assert result.reason == "audit_chain_head_missing" + + +@pytest.mark.anyio +async def test_append_refuses_to_start_a_new_genesis_when_the_head_is_gone( + clean_database, +): + """The next append must not sign a fresh genesis over a live chain. + + Without a head the append treated the wallet as empty, stored seq 1 + with an empty previous hash, and left the older links unverifiable. + """ + wallet_id = "wlt-head-append" + for n in range(2): + await record_audit_event( + event="trust.headappend", wallet_id=wallet_id, metadata={"n": n} + ) + + factory = get_session_factory() + async with factory() as session: + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + await session.delete(head) + await session.commit() + + with pytest.raises(AuditChainHeadMissingError, match="audit_chain_head_missing"): + await record_audit_event( + event="trust.headappend", wallet_id=wallet_id, metadata={"n": 2} + ) + + async with factory() as session: + count = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel).where( + ControlPlaneAuditEventModel.wallet_id == wallet_id + ) + ) + ) + .scalars() + .all() + ) + assert len(count) == 2 + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is False + assert result.reason == "audit_chain_head_missing" + + +@pytest.mark.anyio +async def test_missing_head_error_is_non_retryable(clean_database, monkeypatch): + wallet_id = f"wlt-{uuid4()}" + await record_audit_event(event="trust.headappend", wallet_id=wallet_id) + + factory = get_session_factory() + async with factory() as session: + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + await session.delete(head) + await session.commit() + + from app.services import audit_chain + + attempts = 0 + + def counted_factory(): + nonlocal attempts + attempts += 1 + return factory() + + monkeypatch.setattr(audit_chain, "get_session_factory", lambda: counted_factory) + with pytest.raises(AuditChainHeadMissingError) as excinfo: + await record_audit_event(event="trust.headappend", wallet_id=wallet_id) + + assert excinfo.value.reason == "audit_chain_head_missing" + assert not isinstance(excinfo.value, AuditChainContendedError) + assert attempts == 1 + + +@pytest.mark.anyio +async def test_blank_wallet_id_shares_the_wallet_less_chain(clean_database): + """NULL and "" are one chain, because they already share one head key. + + Verification used to walk them separately. Each half then failed: the + NULL side looked truncated, and the blank side did not link to genesis. + """ + await record_audit_event(event="system.a", wallet_id=None, metadata={"n": 1}) + await record_audit_event(event="system.b", wallet_id="", metadata={"n": 2}) + + global_result = await verify_audit_chain(wallet_id=None) + assert global_result.valid is True, global_result + assert global_result.checked_events == 2 + + blank = await verify_audit_chain(wallet_id="") + assert blank.valid is True, blank + assert blank.checked_events == 2 + + +@pytest.mark.anyio +async def test_legacy_seq_zero_chain_without_a_head_still_verifies(clean_database): + """Rows from before seq existed are all seq 0 and have no head. + + Those rows must keep verifying. A modern chain (seq starting at 1) + with the head removed must not. + """ + wallet_id = "wlt-legacy-seq0" + base = utc_now() + for n in range(2): + await record_audit_event( + event="trust.legacy", + wallet_id=wallet_id, + created_at=base + timedelta(seconds=n), + metadata={"n": n}, + ) + + factory = get_session_factory() + async with factory() as session: + rows = ( + ( + await session.execute( + select(ControlPlaneAuditEventModel) + .where(ControlPlaneAuditEventModel.wallet_id == wallet_id) + .order_by(ControlPlaneAuditEventModel.created_at) + ) + ) + .scalars() + .all() + ) + previous_hash = None + for row in rows: + row.seq = 0 + row.previous_hash = previous_hash + payload = _payload_without_seq(row) + row.payload_hash = sha256_hex(payload) + payload["payload_hash"] = row.payload_hash + assert row.signature_key_id is not None + row.signature, row.signature_key_id, _ = ( + get_signing_key_service().sign_payload_with_key_id( + payload, row.signature_key_id + ) + ) + row.chain_hash = sha256_hex( + { + "previous_hash": previous_hash, + "payload_hash": row.payload_hash, + "signature": row.signature, + } + ) + previous_hash = row.chain_hash + session.add(row) + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + await session.delete(head) + await session.commit() + + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is True, result + assert result.checked_events == 2 + + # The next append continues the legacy chain instead of refusing it. + await record_audit_event( + event="trust.legacy", wallet_id=wallet_id, metadata={"n": 2} + ) + async with factory() as session: + appended = ( + await session.execute( + select(ControlPlaneAuditEventModel).where( + ControlPlaneAuditEventModel.wallet_id == wallet_id, + ControlPlaneAuditEventModel.seq == 1, + ) + ) + ).scalar_one() + assert appended.previous_hash == previous_hash + head = await session.get(AuditChainHeadModel, wallet_id) + assert head is not None + assert head.last_seq == 1 + assert head.last_chain_hash == appended.chain_hash + result = await verify_audit_chain(wallet_id=wallet_id) + assert result.valid is True, result + assert result.checked_events == 3