diff --git a/docs/agent-guide/implemented/bugfix/2026-10-19-authoritative-transition-diagnostics.md b/docs/agent-guide/implemented/bugfix/2026-10-19-authoritative-transition-diagnostics.md new file mode 100644 index 0000000..ba087fe --- /dev/null +++ b/docs/agent-guide/implemented/bugfix/2026-10-19-authoritative-transition-diagnostics.md @@ -0,0 +1,103 @@ +# Authoritative Transition Diagnostics + +## Problem + +The installed a0746e97 worker discards begin_authoritative error variants and +publishes one generic rejection with Storage_busy. The Worker errors publisher +then replaces every authoritative failure with one generic ledger entry. +This prevents distinguishing closed databases, sync-version conflicts, +discontinuous cursors and unsupported transaction formats. + +## Decision + +Keep the existing overlay variants and worker Error codes. Use one pure +authoritative transition error mapper in the effect runner, shared by begin and +apply. All messages and diagnostic category signals must be fixed strings; +discard raw decode, integrity, persistence and fatal-state text. +Preserve the category through the existing reducer Diagnostic signal and the +service's finite allowlist, so Last error and Worker errors both distinguish it. +Do not change recovery, synchronization policy, protected types, Dune, account +access, phone installation or graph data. + +The runner mapper stays private. Deterministic runtime tests will exercise actual +cursor and format rejection through Runner and Core, without fabricated producer +results. Separate propagation tests cover all four typed errors at the reducer +boundary, and the service's existing pure diagnostic row projection is exported +through its ordinary (non-spec) mli for direct tests of the actual UI allowlist. +These propagation tests do not claim producer coverage of close/token races. +The two deterministic producer races remain explicitly uncovered because the +existing public runner interface has no such scheduling boundary. No protected +spec or Dune change is made. + +## Alternatives considered + +### Only replace the runner message + +Rejected because Worker errors would still normalize every failure. Mapping tests +that inject hand-written worker errors would not verify producer classification. +Adding a second error enum or moving overlay policy into the contract would add +unnecessary coupling and potentially require prohibited Dune changes. + +## Acceptance criteria + +- All four begin-authoritative variants have distinct fixed messages and accurate + existing worker Error codes; only genuinely busy states use Storage_busy. +- Producer mapping and actual downstream diagnostics projection are covered by + deterministic tests where publicly reachable, including raw-sensitive-string exclusion. +- Existing source and public type boundaries are respected; production code + changes require no protected-interface extension. +- A real Web/Simulator reproduction must use the same verified dedicated graph, + ordinary UI edits and authenticated production synchronization, not worker injection. + +## Risks + +- Discarding raw lower-level messages intentionally limits detail to safe categories. +- Host-restamped Simulator builds retain cross_abi_validated=false. +- The current 1446 ordinary App has no authenticated session. Its normal Cognito + login form requires credentials unavailable in the configured environment. + No credentials or tokens may be copied from the Mac browser. + +## Questions + +- None for the authorized implementation. Explicit parent direction on 2026-10-19 + keeps the producer mapper private, accepts the honest deterministic coverage gap, + and forbids modifying protected interfaces. Simulator authentication and any + future producer test-interface extension remain separate pending user actions. + +## Consequences + +The private shared mapper now emits fixed messages and origin categories using +existing Error codes. Only actual prepare/commit busy variants use Storage_busy. +Checksum/integrity/persistence use the broad Ambiguous_sync_state code with their +exact fixed origin category; a fatal-state commit closes the database and uses +Closed_session. Neither ownership failure nor disk corruption is inferred. +Core retains the origin category in its Diagnostic signal. The service's finite +allowlist retains that category in the existing applyAuthoritative ledger row. +No recovery, retry, commit, graph, authentication or asset behavior changed. + +Three actual runner/database rejection cases failed against the old uniform +Storage_busy result and passed after the change. Four-category downstream +propagation and the actual service row allowlist also failed before and passed +afterward. The producer mapper remains private: database-close and sync-version +preparation races have no deterministic public hook and are not claimed covered. +The service projection test is compositional coverage, not mounted UI acceptance. + +A clean full @all build and forced full runtest passed: 21 Alcotest suites, +977 cases, plus standalone programs. The new decision passes validation; the +whole document check retains four pre-existing invalid historical documents. +Independent review's two classification findings were corrected and final diff +review found no further blockers. All temporary evidence is outside the repo. + +An external synthetic probe uses real Runner/Database commits and only delays +consumption of their actual completion. Its control consumes checkpoint 1; +backgrounding before consumption instead leaves the manager at 0 while SQLite +is durably at 1. Foreground plus a normal repeated pull then actually rejects +cursor 1 as authoritativeCursorDiscontinuous. No completion is fabricated and +no product recovery fix was made. This proves a local client race path, not +real Web synchronization or the cause of the user's phone screenshot. + +The Mac Web test graph is verified online with local/remote tx 872 and no edits +submitted. Existing Simulator 1446 lacks a logged-in ordinary Journal session; +its normal Cognito login form requires user input. No credentials were copied, +no iPhone was used and no graph was deleted. Protected spec and Dune files remain +unchanged. Simulator packaging, if prepared later, retains experimental ABI limits. diff --git a/docs/agent-guide/implemented/bugfix/2026-10-20-authoritative-decode-detail-diagnostics.md b/docs/agent-guide/implemented/bugfix/2026-10-20-authoritative-decode-detail-diagnostics.md new file mode 100644 index 0000000..7b8e40a --- /dev/null +++ b/docs/agent-guide/implemented/bugfix/2026-10-20-authoritative-decode-detail-diagnostics.md @@ -0,0 +1,100 @@ +# Authoritative Decode Detail Diagnostics + +## Problem + +Build 50 classifies the phone failure as Authoritative_decode_failed but loses +which preparation, decoding or in-memory application step failed. The existing +string includes raw Transit, exception and potentially graph-specific content, +so simply displaying it would leak data and still mislabel application exceptions. + +## Decision + +Keep the protected Authoritative_decode_failed of string interface and all sync +behavior. Add a shared, strictly validated bounded diagnostic codec in ordinary +logseq_db_types sources. Only fixed stages, reasons, type/operation/attribute +allowlists and nonnegative bounded structural counters may cross this carrier. +Codec functions keep their current signatures; authoritative decoding emits +safe records, database adds stage/transaction context, and runner/reducer/service +propagate only validated records into Last error and Worker errors. Legacy or +invalid carriers retain the existing generic category, never arbitrary text. + +Catch Datascript.with_tx exceptions as application-exception plus a fixed +exception constructor class. Do not inspect exception strings to guess causes. +An unresolved lookup ref can be visible only as a structural lookup-reference +count, not a fabricated precise failure reason or failing operation index. +Do not repair empty transactions, cursor races, retries, or recovery. + +The pure reducer cannot originate wire decoding errors: its public events carry +already completed effects. Previous public runner/database reproductions reached +both preparation and application failures. Extend that narrow producer suite; +service projection tests independently verify its actual sanitizing boundary. +No private module access, protected spec or Dune changes are required. + +## Alternatives considered + +### Expose raw lower errors or extend protected variants + +Rejected: raw errors can contain user values and paths. Extending protected specs +requires separate explicit authorization and is unnecessary with a strict safe +codec over the existing string carrier. Exception-message substring guessing +would falsely imply a known cause and is excluded. + +## Acceptance criteria + +- Actual wire decoding failures distinguish preparation, decode and application + stages, fixed reasons, transaction/operation position and useful safe types. +- Last error and Worker errors preserve the same safe detail. Legacy/malicious + diagnostic inputs cannot display arbitrary content or malformed counters. +- Synthetic boundary cases cover empty/invalid transactions, operations/fields, + protected values, entity/reference/source-tx/numeric types, missing lookups and + application exceptions. Tests assert checkpoint and data remain uncommitted. +- Full build/tests and independent review pass before building a correct signed + iphoneos artifact, backing up Build 50 data and installing an in-place update. +- No raw payload, plaintext, UUID, account/graph names, token, custom attribute, + original exception string, protected spec or Dune change is introduced. + +## Risks + +- Application exceptions remain intentionally broad because the public API does + not identify their failing operation or expose typed cause information. +- Historical raw strings must fail closed to a generic diagnostic. +- Host-restamped device builds remain cross_abi_validated=false; USB launch does + not prove complete ABI/runtime acceptance or the original incident cause. + +## Questions + +- None. The user explicitly approved diagnostic detail and iPhone update on + 2026-10-20. Independent review is requested; no PR, push or merge is authorized. + +## Consequences + +A shared abstract diagnostic codec restricts the existing string carrier to +bounded fixed enums and structural counters; unknown or malformed carriers are +never displayed. Both Last error and Worker errors use its safe formatter. +Preparation, operation decode and transaction application retain distinct stages. +Codec failures retain 1-based transaction/operation context and finite type, +operation and builtin-attribute descriptors. Custom names become custom/other. +Application exceptions retain only a fixed constructor class and bounded +best-effort lookup-reference/operation counts, never an operation attribution. +The original recursive exception fallback remains nonfatal with unknown decode +classification. Protected spec, Dune and transaction acceptance branches remain +unchanged; no empty-transaction, cursor-race or retry fix was made. + +The actual runner/database producer suite now has 22 deterministic cases, +including preparation/decode/application failures, second transaction/operation +position, CAS expected/value, numeric/ref types and bounded statistics. It checks +Last error, emitted signal, unchanged checkpoint and exclusion of synthetic +content, paths, UUID and raw application exception text. Service tests cover the +real row projection, unknown/duplicate/malformed/oversized fields, invalid enums +and counters, legacy raw input, byte budget and roundtrip. No private module or +protected interface was opened for tests. The first full build and runtest pass +21 Alcotest suites / 996 cases plus standalone programs; final verification after +preserving the recursive catch is recorded externally before packaging. + +Independent review found and corrected a test-only cursor fixture regression, +CAS expected-field attribution, potentially unbounded diagnostic statistics, and +the original recursive catch scope. The public interfaces cannot deterministically +induce every unexpected recursive exception or decrypt-count mismatch; these +are not claimed individually reproduced. The actual phone subtype remains to +be observed in the next installed diagnostics build, not inferred from fixtures. +Installation evidence and final hashes remain in this task's external evidence. diff --git a/docs/agent-guide/implemented/bugfix/2026-10-20-authoritative-empty-transaction-compatibility.md b/docs/agent-guide/implemented/bugfix/2026-10-20-authoritative-empty-transaction-compatibility.md new file mode 100644 index 0000000..d472963 --- /dev/null +++ b/docs/agent-guide/implemented/bugfix/2026-10-20-authoritative-empty-transaction-compatibility.md @@ -0,0 +1,86 @@ +# Authoritative Empty Transaction Compatibility + +## Problem + +Build 51 shows prepare / empty-transaction / transaction=1 on the phone. +The first transaction in the current authoritative batch parses as an empty +Transit array or list; the client rejects it before application. The upstream +normalized transaction log permits empty operations while advancing its cursor. +The diagnostic ordinal is not a server cursor. + +## Decision + +Allow an empty operation array/list only at the private codec authoritative-input +entry points. Keep strict defaults for durable outgoing-record validation and +insertion helpers. Preserve the empty transaction as one element of the batch, +using all existing candidate, origin validation, staging, replay and atomic +checkpoint/outbox/revision commit paths. No cursor shortcut or dropped batch slot. +Do not modify protected specs, Dune, sync policy, race recovery, attachments, +upstream normalization or any other suspected issue. User authorization explicitly +limits this task to the confirmed empty authoritative transaction gap and iPhone update. + +The pure sync reducer stores opaque wire text and cannot originate the decoding +failure through pure events without fabricating a producer error. The previous +actual runner/database reproductions confirmed ownership in the overlay codec; +new regressions use the public Database boundary, the narrowest accessible owner. +Existing runner diagnostic regressions for empty input must change to success, +while all other existing failure/privacy cases remain intact. + +## Alternatives considered + +### Drop empty transactions or accept empty input globally + +Rejected: dropping changes cursor/report/origin alignment; a global parser change +also loosens outgoing validation. Manually moving the checkpoint would bypass +atomicity and token/generation guards. + +## Acceptance criteria + +- Old source fails the public Database empty authoritative regressions; the minimal + change passes array/list, durable reopen and empty/nonempty/empty batch cases. +- No-content changes advance the server checkpoint and sync token without an + unnecessary logical projection change. Real mixed transactions still apply. +- Late malformed transaction and storage failure roll back the whole batch; + stale token, empty outer batch, malformed input and outgoing-empty validation + retain their existing rejection behavior. +- Final full checks and independent review use the final source. Correct signed + iphoneos output is installed in place only after fresh Build 51 backup and + rollback preservation; actual device version/startup are read back. + +## Risks + +- Existing origin validation can still reject an empty transaction that does not + match an accepted local submission; this integrity guard must remain unchanged. +- This fixes client compatibility, not why upstream produced an empty transaction. +- Experimental host-restamp remains cross_abi_validated=false. + +## Questions + +- None. The user explicitly approved only this compatibility fix and iPhone + update on 2026-10-20, and forbade unrelated speculative fixes. + +## Consequences + +The private parser/protected-value collector/decoder keep allow_empty=false by + default, and only the three authoritative-input calls pass true. Legal empty +arrays/lists now produce empty operations through the existing candidate, +origin, storage, replay and atomic commit paths. No production storage, sync +manager, outbox, recovery, attachment, upstream or unrelated behavior changed. +The protected spec and Dune remain unchanged. + +Eight public Database cases cover empty arrays/lists, mixed batch slots, +checkpoint/token persistence after reopen and subsequent cursor application, +late decode and actual SQLite metadata commit failures with disk rollback, +token guards and malformed neighbour/outer-batch rejection. Seven failed on the +old producer; all passed after the change. The late failure explicitly checks +stage=decode/transaction=3. The existing outgoing corruption fixture now also +checks [] still fails closed. Existing runner empty/second-transaction regressions +now assert actual success, checkpoint advancement and no Last error; they failed +on the old source and pass with this change. All other diagnostic failures and +privacy cases are retained. The complete storage suite (85 cases) also passes. +Final full build/test results, independent review, source and install evidence +are recorded externally before phone delivery. + +No additional issue was fixed. The user authorization and excluded scope remain +as stated above; the cause of upstream empty generation and the independent +attachment error are not investigated or changed in this implementation. diff --git a/logseq_db_types/lib/authoritative_diagnostic.ml b/logseq_db_types/lib/authoritative_diagnostic.ml new file mode 100644 index 0000000..4b64040 --- /dev/null +++ b/logseq_db_types/lib/authoritative_diagnostic.ml @@ -0,0 +1,305 @@ +type stage = + | Prepare + | Decode + | Application + +type reason = + | Transit_syntax + | Empty_transaction + | Transaction_shape + | Operation_shape + | Operation_keyword + | Unsupported_operation + | Operation_fields + | Entity_shape + | Reference_shape + | Integer_range + | Source_tx_type + | Protected_value_type + | Decrypted_value_count + | Decimal_value + | Integer_value + | Application_exception + | Unknown + +type t = (string * Yojson.Safe.t) list + +let stages = [ "prepare"; "decode"; "application" ] + +let reasons = + [ "transit-syntax" + ; "empty-transaction" + ; "transaction-shape" + ; "operation-shape" + ; "operation-keyword" + ; "unsupported-operation" + ; "operation-fields" + ; "entity-shape" + ; "reference-shape" + ; "integer-range" + ; "source-tx-type" + ; "protected-value-type" + ; "decrypted-value-count" + ; "decimal-value" + ; "integer-value" + ; "application-exception" + ; "unknown" + ] +;; + +let operations = + [ "add"; "retract"; "retract-attribute"; "retract-entity"; "cas"; "other" ] +;; + +let attributes = + [ "block/title" + ; "block/name" + ; "block/uuid" + ; "block/parent" + ; "block/page" + ; "block/order" + ; "block/refs" + ; "db/ident" + ; "db/valueType" + ; "db/cardinality" + ; "custom" + ] +;; + +let fields = [ "entity"; "attribute"; "value"; "expected"; "source-tx"; "unknown" ] + +let kinds = + [ "null" + ; "bool" + ; "string" + ; "integer" + ; "int64" + ; "float" + ; "binary" + ; "decimal" + ; "big-integer" + ; "date" + ; "uuid" + ; "uri" + ; "keyword" + ; "symbol" + ; "array" + ; "map" + ; "set" + ; "list" + ; "tagged" + ; "unknown" + ] +;; + +let exceptions = [ "invalid-argument"; "failure"; "not-found"; "other" ] + +let counter_keys = + [ "transaction"; "operationIndex"; "operationCount"; "lookupRefs"; "arity" ] +;; + +let string_lists = + [ "stage", stages + ; "reason", reasons + ; "operation", operations + ; "attribute", attributes + ; "field", fields + ; "type", kinds + ; "exception", exceptions + ] +;; + +let key_order = + [ "stage" + ; "reason" + ; "transaction" + ; "operationIndex" + ; "operationCount" + ; "lookupRefs" + ; "arity" + ; "operation" + ; "attribute" + ; "field" + ; "type" + ; "exception" + ] +;; + +let prefix = "authoritative-decode:v1:" +let detail_name = "authoritativeDiagnostic" +let signal_prefix = "worker.applyAuthoritative.authoritativeDecodeFailed:" +let maximum_counter = 1_000_000_000 + +let stage_name = function + | Prepare -> "prepare" + | Decode -> "decode" + | Application -> "application" +;; + +let reason_name = function + | Transit_syntax -> "transit-syntax" + | Empty_transaction -> "empty-transaction" + | Transaction_shape -> "transaction-shape" + | Operation_shape -> "operation-shape" + | Operation_keyword -> "operation-keyword" + | Unsupported_operation -> "unsupported-operation" + | Operation_fields -> "operation-fields" + | Entity_shape -> "entity-shape" + | Reference_shape -> "reference-shape" + | Integer_range -> "integer-range" + | Source_tx_type -> "source-tx-type" + | Protected_value_type -> "protected-value-type" + | Decrypted_value_count -> "decrypted-value-count" + | Decimal_value -> "decimal-value" + | Integer_value -> "integer-value" + | Application_exception -> "application-exception" + | Unknown -> "unknown" +;; + +let canonical entries = + List.filter_map + (fun key -> Option.map (fun value -> key, value) (List.assoc_opt key entries)) + key_order +;; + +let fixed values fallback value = if List.mem value values then value else fallback + +let operation_name = function + | "db/add" | "add" -> "add" + | "db/retract" | "retract" -> "retract" + | "db.fn/retractAttribute" | "db/retractAttribute" | "retract-attribute" -> + "retract-attribute" + | "db/retractEntity" | "retract-entity" -> "retract-entity" + | "db.fn/cas" | "db/cas" | "cas" -> "cas" + | _ -> "other" +;; + +let add_optional key value entries = + match value with + | None -> entries + | Some value -> (key, value) :: List.remove_assoc key entries +;; + +let number key value = + Option.map + (fun n -> + `Int + (max + (if key = "transaction" || key = "operationIndex" then 1 else 0) + (min maximum_counter n))) + value +;; + +let descriptor values fallback value = + Option.map (fun v -> `String (fixed values fallback v)) value +;; + +let with_context + ?stage + ?transaction + ?operation_index + ?operation_count + ?lookup_refs + ?arity + ?operation + ?attribute + ?field + entries + = + entries + |> add_optional "stage" (Option.map (fun x -> `String (stage_name x)) stage) + |> add_optional "transaction" (number "transaction" transaction) + |> add_optional "operationIndex" (number "operationIndex" operation_index) + |> add_optional "operationCount" (number "operationCount" operation_count) + |> add_optional "lookupRefs" (number "lookupRefs" lookup_refs) + |> add_optional "arity" (number "arity" arity) + |> add_optional "operation" (Option.map (fun x -> `String (operation_name x)) operation) + |> add_optional "attribute" (descriptor attributes "custom" attribute) + |> add_optional "field" (descriptor fields "unknown" field) + |> canonical +;; + +let create + ~stage + ~reason + ?transaction + ?operation_index + ?operation_count + ?lookup_refs + ?arity + ?operation + ?attribute + ?field + ?actual + ?exception_kind + () + = + [ "stage", `String (stage_name stage); "reason", `String (reason_name reason) ] + |> with_context + ?transaction + ?operation_index + ?operation_count + ?lookup_refs + ?arity + ?operation + ?attribute + ?field + |> add_optional "type" (descriptor kinds "unknown" actual) + |> add_optional "exception" (descriptor exceptions "other" exception_kind) + |> canonical +;; + +let encode value = prefix ^ Yojson.Safe.to_string (`Assoc value) + +let decode source = + if String.length source > 512 || not (String.starts_with ~prefix source) + then None + else ( + try + match + Yojson.Safe.from_string + (String.sub + source + (String.length prefix) + (String.length source - String.length prefix)) + with + | `Assoc entries -> + let keys = List.map fst entries in + let valid = + List.length keys = List.length (List.sort_uniq String.compare keys) + && List.mem_assoc "stage" entries + && List.mem_assoc "reason" entries + && List.for_all + (fun (key, value) -> + match List.assoc_opt key string_lists, value with + | Some allowed, `String s -> List.mem s allowed + | None, `Int n when List.mem key counter_keys -> + (n >= if key = "transaction" || key = "operationIndex" then 1 else 0) + && n <= maximum_counter + | _ -> false) + entries + && + if List.assoc_opt "stage" entries = Some (`String "application") + then + List.assoc_opt "reason" entries = Some (`String "application-exception") + && not (List.mem_assoc "operationIndex" entries) + else List.assoc_opt "reason" entries <> Some (`String "application-exception") + in + if valid then Some (canonical entries) else None + | _ -> None + with + | Yojson.Json_error _ | Failure _ | Invalid_argument _ -> None) +;; + +let message value = + let part (key, value) = + key + ^ "=" + ^ + match value with + | `String s -> s + | `Int n -> string_of_int n + | _ -> "unknown" + in + "Authoritative transaction rejected (" ^ String.concat "; " (List.map part value) ^ ")." +;; diff --git a/logseq_db_types/lib/authoritative_diagnostic.mli b/logseq_db_types/lib/authoritative_diagnostic.mli new file mode 100644 index 0000000..b55afd8 --- /dev/null +++ b/logseq_db_types/lib/authoritative_diagnostic.mli @@ -0,0 +1,64 @@ +(** A bounded diagnostic containing no transaction values or exception text. + String descriptors are reduced to finite allowlists by [create]. + [Application] records use [Application_exception] and omit [operation_index]; + other stages do not use that reason. Invalid decoded combinations fail closed. *) +type t + +type stage = + | Prepare + | Decode + | Application + +type reason = + | Transit_syntax + | Empty_transaction + | Transaction_shape + | Operation_shape + | Operation_keyword + | Unsupported_operation + | Operation_fields + | Entity_shape + | Reference_shape + | Integer_range + | Source_tx_type + | Protected_value_type + | Decrypted_value_count + | Decimal_value + | Integer_value + | Application_exception + | Unknown + +val create + : stage:stage + -> reason:reason + -> ?transaction:int + -> ?operation_index:int + -> ?operation_count:int + -> ?lookup_refs:int + -> ?arity:int + -> ?operation:string + -> ?attribute:string + -> ?field:string + -> ?actual:string + -> ?exception_kind:string + -> unit + -> t + +val with_context + : ?stage:stage + -> ?transaction:int + -> ?operation_index:int + -> ?operation_count:int + -> ?lookup_refs:int + -> ?arity:int + -> ?operation:string + -> ?attribute:string + -> ?field:string + -> t + -> t + +val encode : t -> string +val decode : string -> t option +val message : t -> string +val detail_name : string +val signal_prefix : string diff --git a/logseq_db_worker/lib/effect_runner/effect_runner.ml b/logseq_db_worker/lib/effect_runner/effect_runner.ml index 1b069aa..c431388 100644 --- a/logseq_db_worker/lib/effect_runner/effect_runner.ml +++ b/logseq_db_worker/lib/effect_runner/effect_runner.ml @@ -1455,29 +1455,107 @@ let unprotect_request t key request = | Ok plaintexts -> Ok (request, plaintexts)) ;; -let authoritative_transition_error = function - | Overlay.Authoritative_database_closed -> "authoritative database closed" - | Authoritative_sync_token_conflict -> "authoritative sync token conflict" - | Authoritative_cursor_discontinuous -> "authoritative cursor discontinuous" - | Authoritative_checksum_mismatch -> "authoritative checksum mismatch" - | Authoritative_crypto_required -> "authoritative crypto required" - | Authoritative_crypto_unexpected -> "authoritative crypto unexpected" - | Authoritative_crypto_error _ -> "authoritative crypto result invalid" - | Authoritative_decode_failed message -> "authoritative decode failed: " ^ message - | Authoritative_origin_unresolved -> "authoritative origin unresolved" - | Authoritative_preparation_consumed -> "authoritative preparation consumed" - | Authoritative_prepare_busy -> "authoritative preparation busy" - | Authoritative_integrity_failure message -> - "authoritative integrity failure: " ^ message - | Authoritative_commit_database_closed -> "authoritative database closed at commit" - | Authoritative_commit_token_conflict -> "authoritative commit token conflict" - | Authoritative_commit_generation_invalidated -> - "authoritative commit generation invalidated" - | Authoritative_commit_busy -> "authoritative commit busy" - | Authoritative_commit_persistence_failed message -> - "authoritative commit persistence failed: " ^ message - | Authoritative_commit_fatal_state message -> - "authoritative commit fatal state: " ^ message +let authoritative_transition_error error = + let module Error = Logseq_db_worker_contract.Error in + let module Diagnostic = Logseq_db_types.Authoritative_diagnostic in + let diagnostic = + match error with + | Overlay.Authoritative_decode_failed source -> Diagnostic.decode source + | _ -> None + in + let code, category, message = + match error with + | Overlay.Authoritative_database_closed -> + ( Error.Closed_session + , "authoritativeDatabaseClosed" + , "Authoritative database is closed." ) + | Overlay.Authoritative_sync_token_conflict -> + ( Error.Conflict + , "authoritativeSyncTokenConflict" + , "Authoritative sync version changed." ) + | Overlay.Authoritative_cursor_discontinuous -> + ( Error.Ambiguous_sync_state + , "authoritativeCursorDiscontinuous" + , "Authoritative transaction cursors are not continuous." ) + | Overlay.Authoritative_checksum_mismatch -> + ( Error.Ambiguous_sync_state + , "authoritativeChecksumMismatch" + , "Authoritative checksum does not match." ) + | Overlay.Authoritative_crypto_required -> + ( Error.Invalid_request + , "authoritativeCryptoRequired" + , "Authoritative transition requires decrypted values." ) + | Overlay.Authoritative_crypto_unexpected -> + ( Error.Invalid_request + , "authoritativeCryptoUnexpected" + , "Authoritative transition received unexpected decrypted values." ) + | Overlay.Authoritative_crypto_error _ -> + ( Error.Unsupported_value + , "authoritativeCryptoInvalid" + , "Authoritative decrypted values are invalid." ) + | Overlay.Authoritative_decode_failed _ -> + ( Error.Unsupported_semantics + , "authoritativeDecodeFailed" + , Option.fold + ~none:"Authoritative transaction format is unsupported." + ~some:Diagnostic.message + diagnostic ) + | Overlay.Authoritative_origin_unresolved -> + ( Error.Ambiguous_sync_state + , "authoritativeOriginUnresolved" + , "Authoritative transaction origin is unresolved." ) + | Overlay.Authoritative_preparation_consumed -> + ( Error.Invalid_request + , "authoritativePreparationConsumed" + , "Authoritative preparation was already consumed." ) + | Overlay.Authoritative_prepare_busy -> + Error.Storage_busy, "authoritativePrepareBusy", "Authoritative preparation is busy." + | Overlay.Authoritative_integrity_failure _ -> + ( Error.Ambiguous_sync_state + , "authoritativeIntegrityFailure" + , "Authoritative integrity validation failed." ) + | Overlay.Authoritative_commit_database_closed -> + ( Error.Closed_session + , "authoritativeCommitDatabaseClosed" + , "Authoritative database closed before commit." ) + | Overlay.Authoritative_commit_token_conflict -> + ( Error.Conflict + , "authoritativeCommitTokenConflict" + , "Authoritative sync version changed before commit." ) + | Overlay.Authoritative_commit_generation_invalidated -> + ( Error.Conflict + , "authoritativeCommitGenerationInvalidated" + , "Authoritative database generation changed before commit." ) + | Overlay.Authoritative_commit_busy -> + Error.Storage_busy, "authoritativeCommitBusy", "Authoritative commit is busy." + | Overlay.Authoritative_commit_persistence_failed _ -> + ( Error.Ambiguous_sync_state + , "authoritativeCommitPersistenceFailed" + , "Authoritative commit could not be persisted." ) + | Overlay.Authoritative_commit_fatal_state _ -> + ( Error.Closed_session + , "authoritativeCommitFatalState" + , "Authoritative storage requires recovery." ) + in + let origin = + Error.create_cause + ~component:Error.Managed_sync + ~operation:"applyAuthoritative" + ~code:(Some category) + ~message + |> Result.get_ok + in + let details = + match diagnostic with + | None -> [] + | Some detail -> + [ Error. + { name = Diagnostic.detail_name + ; value = Detail_string (Diagnostic.encode detail) + } + ] + in + Error.create_with_origin ~code ~message ~details ~origin |> Result.get_ok ;; let deletion_authorized t (request : Sync.mirror_deletion) = @@ -1902,7 +1980,7 @@ let handle_sync_worker_effect t = function ~expected:(Overlay.sync_view_token sync) request.input with - | Error _ -> Error (effect_error "The authoritative transition was rejected.") + | Error error -> Error (authoritative_transition_error error) | Ok (prepared, crypto) -> let decrypted = match crypto with @@ -1917,7 +1995,7 @@ let handle_sync_worker_effect t = function Database.apply_authoritative session.database prepared ~decrypted with | Error error -> - Error (effect_error (authoritative_transition_error error)) + Error (authoritative_transition_error error) | Ok (Database.Authoritative_deferred defer) -> Ok (sync_result diff --git a/logseq_db_worker/lib/pure_reducer/core.ml b/logseq_db_worker/lib/pure_reducer/core.ml index 27ebb1c..67a8e24 100644 --- a/logseq_db_worker/lib/pure_reducer/core.ml +++ b/logseq_db_worker/lib/pure_reducer/core.ml @@ -489,8 +489,23 @@ let step state event = { scope = Sync.effect_scope_of_graph request.scope.graph; message })) state in + let origin = (Logseq_db_worker_contract.Error.trace error).origin in + let signal = + match origin.component, origin.operation, origin.code with + | Managed_sync, "applyAuthoritative", Some "authoritativeDecodeFailed" -> + let module Diagnostic = Logseq_db_types.Authoritative_diagnostic in + let detail = List.find_map (fun (detail:Logseq_db_worker_contract.Error.detail) -> + match detail.name,detail.value with + | name,Detail_string source when name=Diagnostic.detail_name -> Diagnostic.decode source + | _ -> None) (Logseq_db_worker_contract.Error.details error) in + Option.fold ~none:"worker.applyAuthoritative.authoritativeDecodeFailed" + ~some:(fun value -> Diagnostic.signal_prefix ^ Diagnostic.encode value) detail + | Managed_sync, "applyAuthoritative", Some category -> + "worker.applyAuthoritative." ^ category + | _ -> "worker.applyAuthoritative" + in { next = transitioned.next - ; effects = Publish (Diagnostic "worker.applyAuthoritative") :: transitioned.effects + ; effects = Publish (Diagnostic signal) :: transitioned.effects } | Some (Sync.Apply_outbox_transition request) -> let transitioned = diff --git a/logseq_db_worker/lui/logseq_db_worker_lui_service.ml b/logseq_db_worker/lui/logseq_db_worker_lui_service.ml index 639d566..fad5cc4 100644 --- a/logseq_db_worker/lui/logseq_db_worker_lui_service.ml +++ b/logseq_db_worker/lui/logseq_db_worker_lui_service.ml @@ -432,6 +432,12 @@ let production_dependencies () = (* Diagnostic strings are internal signals, never display text. Unknown values receive a fixed fallback so exception/path/content strings cannot escape. *) let diagnostic_report = function + | code when String.starts_with ~prefix:Logseq_db_types.Authoritative_diagnostic.signal_prefix code -> + let module Diagnostic = Logseq_db_types.Authoritative_diagnostic in + let prefix=Diagnostic.signal_prefix in + let detail=Diagnostic.decode (String.sub code (String.length prefix) (String.length code-String.length prefix)) in + "applyAuthoritative", "sync", + Option.fold ~none:"Authoritative transaction format is unsupported." ~some:Diagnostic.message detail | "worker.readTagTitles" -> "readTagTitles", "presentation", "Tag labels could not be read. Reopen this page, or reopen Journal." | "worker.closeDatabase" -> "closeDatabase", "cleanup", "The current graph could not finish closing. Reopen Journal before trying again." | "worker.confirmMirrorRetirement" -> "confirmMirrorRetirement", "localDeletion", "Local copy removal could not be confirmed as durable. Reopen Journal and check the local copy before trying again." @@ -439,6 +445,42 @@ let diagnostic_report = function | "worker.inspectMirror" -> "inspectMirror", "localRestore", "Local graph inspection failed. Retry or choose another graph." | "worker.attachGraph" -> "attachGraph", "localRestore", "The local graph could not be opened. Retry or choose another graph." | "worker.inspectSync" -> "inspectSync", "sync", "Sync information is unavailable. Retry sync." + | "worker.applyAuthoritative.authoritativeDatabaseClosed" -> + "applyAuthoritative", "sync", "Authoritative database is closed." + | "worker.applyAuthoritative.authoritativeSyncTokenConflict" -> + "applyAuthoritative", "sync", "Authoritative sync version changed." + | "worker.applyAuthoritative.authoritativeCursorDiscontinuous" -> + "applyAuthoritative", "sync", "Authoritative transaction cursors are not continuous." + | "worker.applyAuthoritative.authoritativeChecksumMismatch" -> + "applyAuthoritative", "sync", "Authoritative checksum does not match." + | "worker.applyAuthoritative.authoritativeCryptoRequired" -> + "applyAuthoritative", "sync", "Authoritative transition requires decrypted values." + | "worker.applyAuthoritative.authoritativeCryptoUnexpected" -> + "applyAuthoritative", "sync", "Authoritative transition received unexpected decrypted values." + | "worker.applyAuthoritative.authoritativeCryptoInvalid" -> + "applyAuthoritative", "sync", "Authoritative decrypted values are invalid." + | "worker.applyAuthoritative.authoritativeDecodeFailed" -> + "applyAuthoritative", "sync", "Authoritative transaction format is unsupported." + | "worker.applyAuthoritative.authoritativeOriginUnresolved" -> + "applyAuthoritative", "sync", "Authoritative transaction origin is unresolved." + | "worker.applyAuthoritative.authoritativePreparationConsumed" -> + "applyAuthoritative", "sync", "Authoritative preparation was already consumed." + | "worker.applyAuthoritative.authoritativePrepareBusy" -> + "applyAuthoritative", "sync", "Authoritative preparation is busy." + | "worker.applyAuthoritative.authoritativeIntegrityFailure" -> + "applyAuthoritative", "sync", "Authoritative integrity validation failed." + | "worker.applyAuthoritative.authoritativeCommitDatabaseClosed" -> + "applyAuthoritative", "sync", "Authoritative database closed before commit." + | "worker.applyAuthoritative.authoritativeCommitTokenConflict" -> + "applyAuthoritative", "sync", "Authoritative sync version changed before commit." + | "worker.applyAuthoritative.authoritativeCommitGenerationInvalidated" -> + "applyAuthoritative", "sync", "Authoritative database generation changed before commit." + | "worker.applyAuthoritative.authoritativeCommitBusy" -> + "applyAuthoritative", "sync", "Authoritative commit is busy." + | "worker.applyAuthoritative.authoritativeCommitPersistenceFailed" -> + "applyAuthoritative", "sync", "Authoritative commit could not be persisted." + | "worker.applyAuthoritative.authoritativeCommitFatalState" -> + "applyAuthoritative", "sync", "Authoritative storage requires recovery." | "worker.applyAuthoritative" -> "applyAuthoritative", "sync", "Remote changes could not be applied. Retry sync or reopen Journal." | "worker.applyOutbox" -> "applyOutbox", "sync", "Local sync progress could not be saved. Retry sync or reopen Journal." | "worker.resetAccount" -> "resetAccount", "account", "Local account cleanup could not finish. Retry or reopen Journal." diff --git a/logseq_db_worker/lui/logseq_db_worker_lui_service.mli b/logseq_db_worker/lui/logseq_db_worker_lui_service.mli index 70ec5a1..8951e01 100644 --- a/logseq_db_worker/lui/logseq_db_worker_lui_service.mli +++ b/logseq_db_worker/lui/logseq_db_worker_lui_service.mli @@ -258,6 +258,10 @@ val create val service : (Logseq_db_worker.Config.t, request, response, push) Journal_worker.Service.t +(** Fixed operation, phase and display text for an internal diagnostic signal. + Unknown signals use a safe fallback without displaying the input. *) +val diagnostic_report : string -> string * string * string + (** Complete the current token challenge through bounded control storage. Stale/duplicate replies cannot replace the current flight. Authentication changes revoke old tickets before their ordered DB work is admitted. *) diff --git a/logseq_db_worker/test/test_effect_runner.ml b/logseq_db_worker/test/test_effect_runner.ml index 40c4852..ca8015a 100644 --- a/logseq_db_worker/test/test_effect_runner.ml +++ b/logseq_db_worker/test/test_effect_runner.ml @@ -1629,10 +1629,403 @@ let test_deletion_journal_failure_preserves_worker () = deletion_runtime_fixture let test_deletion_confirmation_failure_reports () = deletion_runtime_fixture ~confirmation_failure:true ~busy:false () +(* Producer mapping lives in the runner. Real cursor/format failures reach it + through the public reducer and database; no failed external result is injected. *) +let authoritative_detail_cases = + [ "empty", "[]", [ "stage=prepare"; "reason=empty-transaction"; "transaction=1" ] + ; "top-level", "{}", [ "stage=prepare"; "reason=transaction-shape"; "type=map" ] + ; "syntax", "[", [ "stage=prepare"; "reason=transit-syntax" ] + ; "entry", "[42]", [ "reason=operation-shape"; "operationIndex=1"; "type=integer" ] + ; ( "keyword" + , "[[\"PRIVATE_SYNTHETIC_PAYLOAD\",99999]]" + , [ "reason=operation-keyword"; "type=string" ] ) + ; ( "attribute" + , "[[\"~:db/add\",99999,\"PRIVATE_SYNTHETIC_PAYLOAD\",1]]" + , [ "reason=operation-fields"; "operation=add"; "arity=4" ] ) + ; ( "protected" + , "[[\"~:db/add\",99999,\"~:block/title\",false]]" + , [ "reason=protected-value-type"; "attribute=block/title"; "type=bool" ] ) + ; ( "extra-fields" + , "[[\"~:db/add\",99999,\"~:block/PRIVATE_SYNTHETIC_PAYLOAD\",1,2,3]]" + , [ "stage=decode"; "reason=operation-fields"; "arity=6"; "attribute=custom" ] ) + ; ( "source-tx" + , "[[\"~:db/add\",99999,\"~:block/foo\",1,\"PRIVATE_SYNTHETIC_PAYLOAD\"]]" + , [ "reason=source-tx-type"; "field=source-tx"; "type=string" ] ) + ; ( "entity-range" + , "[[\"~:db/add\",\"~i4611686018427387904\",\"~:block/foo\",1]]" + , [ "reason=integer-range"; "field=entity"; "type=int64" ] ) + ; ( "big-integer" + , "[[\"~:db/add\",99999,\"~:block/foo\",\"~n9223372036854775808\"]]" + , [ "reason=integer-value"; "type=big-integer" ] ) + ; ( "reference" + , "[[\"~:db/add\",[\"~:block/uuid\",\"~u22222222-2222-4222-8222-222222222222\"],\"~:block/parent\",null]]" + , [ "reason=reference-shape"; "field=value"; "type=null" ] ) + ; ( "missing-lookup" + , "[[\"~:db/add\",[\"~:block/uuid\",\"~u99999999-9999-4999-8999-999999999999\"],\"~:block/foo\",42]]" + , [ "stage=application" + ; "reason=application-exception" + ; "exception=invalid-argument" + ; "lookupRefs=1" + ; "operationCount=1" + ] ) + ; ( "application-nil" + , "[[\"~:db/add\",99999,\"~:block/PRIVATE_SYNTHETIC_PAYLOAD\",null]]" + , [ "stage=application" + ; "reason=application-exception" + ; "exception=invalid-argument" + ; "operationCount=1" + ] ) + ; ( "cas-expected" + , "[[\"~:db.fn/cas\",99999,\"~:block/title\",false,true]]" + , [ "stage=prepare" + ; "reason=protected-value-type" + ; "field=expected" + ; "operation=cas" + ] ) + ; ( "cas-value" + , "[[\"~:db.fn/cas\",99999,\"~:block/title\",null,false]]" + , [ "stage=prepare"; "reason=protected-value-type"; "field=value" ] ) + ; ( "second-transaction" + , "[]" + , [ "stage=prepare"; "reason=empty-transaction"; "transaction=2" ] ) + ; ( "second-operation" + , "[[\"~:db/retractEntity\",99998],[\"~:PRIVATE_SYNTHETIC_PAYLOAD\",99999]]" + , [ "transaction=1" + ; "operationIndex=2" + ; "operation=other" + ; "reason=unsupported-operation" + ] ) + ; ( "application-bounded-stats" + , "[[\"~:db/add\",99999,\"~:block/foo\",null],[\"~:db/add\",99999,\"~:block/bar\",[" + ^ String.concat "," (List.init 3000 (fun _ -> "0")) + ^ "]]]" + , [ "stage=application" + ; "reason=application-exception" + ; "operationCount=2" + ; "!lookupRefs=" + ] ) + ] +;; + +let test_authoritative_rejection_diagnostics kind () = + let module G = Logseq_db_types.Graph_types in + let module Error = Logseq_db_worker.Error in + T.with_managed (fun fixture -> + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let graph_id = + G.Uuid.of_string "94000000-0000-4000-8000-000000000707" |> Result.get_ok + in + let graph : Sync.graph = + { graph_id + ; name = "Synthetic authoritative diagnostics" + ; encrypted = false + ; schema = { major = 65; minor = 33; exact = true } + } + in + let root = fixture.config.application_support_directory in + let worker_root = Filename.concat root "logseq-db-worker" in + Unix.mkdir worker_root 0o700; + let mirrors = Filename.concat worker_root "synced-graphs" in + Unix.mkdir mirrors 0o700; + let directory = Filename.concat mirrors (G.Uuid.to_string graph_id) in + Unix.mkdir directory 0o700; + let sqlite = Sqlite3.db_open (Filename.concat directory "db.sqlite") in + let open Yojson.Safe.Util in + let storage = T.read_json (T.fixture "storage/logseq-65.33-create-page.json") in + Sqlite3.Rc.check (Sqlite3.exec sqlite (storage |> member "tableSql" |> to_string)); + let stmt = + Sqlite3.prepare sqlite "INSERT INTO kvs(addr,content,addresses) VALUES(?,?,?)" + in + storage + |> member "rows" + |> to_list + |> List.iter (fun row -> + Sqlite3.Rc.check (Sqlite3.reset stmt); + Sqlite3.Rc.check (Sqlite3.bind_int stmt 1 (row |> member "addr" |> to_int)); + Sqlite3.Rc.check + (Sqlite3.bind_text stmt 2 (row |> member "content" |> to_string)); + Sqlite3.Rc.check + (Sqlite3.bind + stmt + 3 + (match member "addresses" row with + | `Null -> Sqlite3.Data.NULL + | `String value -> TEXT value + | _ -> assert false)); + Sqlite3.Rc.check (Sqlite3.step stmt)); + Sqlite3.Rc.check (Sqlite3.finalize stmt); + let cursor n = Logseq_db_types.Server_cursor.of_int64 n |> Result.get_ok in + let checkpoint = + Logseq_db_types.Sync_checkpoint.create + ~graph_id + ~schema:{ major = 65; minor = 33 } + ~applied_server_t:(cursor 0L) + ~checksum:"0123456789abcdef" + |> Result.get_ok + in + Logseq_db_storage.Sync_checkpoint_store.initialize_database sqlite checkpoint + |> Result.get_ok; + Logseq_db_storage.Sync_outbox_store.initialize_database sqlite |> Result.get_ok; + ignore (Sqlite3.db_close sqlite); + let limits = + Sync.limits + ~maximum_response_bytes: + (match kind with + | `Detail ("application-bounded-stats", _, _) -> 16384 + | _ -> 1024) + ~maximum_artifact_bytes:1048576 + ~submission_batch_size:32 + |> Result.get_ok + in + let sync = + Sync.config ~managed_sync_origin:(Uri.of_string "https://api.logseq.io") ~limits + |> Result.get_ok + in + let state = + ref (Core.initial (Core.config ~worker:fixture.config ~sync) |> Result.get_ok) + in + let events = Queue.create () + and emitted = ref [] + and failures = ref [] + and connection = ref None in + let post event = Queue.add event events in + let sync_runner = + Runner.sync_runner + ~shutdown:(fun () -> ()) + ~submit:(function + | Sync.Request (ticket, Sync.Fetch_catalog _) -> + post + (Core.Sync_event + (Sync.Runner_completed (Sync.Completion (ticket, Ok [ graph ])))) + | Sync.Request (ticket, Sync.Save_catalog _) -> + post + (Core.Sync_event + (Sync.Runner_completed (Sync.Completion (ticket, Ok ())))) + | Sync.Start_websocket request -> connection := Some request.scope + | _ -> ()) + () + in + let dependencies = + Runner.dependencies + ~runtime: + (Runner.runtime ~fork:(fun ~sw:_ task -> task ()) ~sleep:(fun _ -> ()) + |> Result.get_ok) + ~config:fixture.config + ~overlay:fixture.overlay + ~sync_runner + ~publish:(fun output -> emitted := output :: !emitted) + |> Result.get_ok + in + let runner = + Runner.create + ~sw + dependencies + ~post + ~recovery_current:(fun _ -> false) + ~upload_current:(fun _ -> false) + ~asset_current:(fun _ -> false) + |> Result.get_ok + in + let drive event = + post event; + while not (Queue.is_empty events) do + let event = Queue.take events in + (match event with + | Core.Runner_completed (Sync_worker_effect_completed (_, Error error)) -> + failures := error :: !failures + | _ -> ()); + let transition = Core.step !state event in + state := transition.next; + List.iter (Runner.submit runner) transition.effects + done + in + Fun.protect + ~finally:(fun () -> Runner.shutdown runner) + (fun () -> + drive + (Core.Sync_event + (Sync.Sign_in_requested + { account_generation = 0; presentation_generation = 0 })); + drive + (Core.Sync_event + (Sync.Account_authenticated { user_id = Some "synthetic-user" })); + drive (Core.Sync_event (Sync.Graph_selected graph_id)); + Alcotest.(check bool) + "actual mirror opens" + true + ((Core.view !state).graph.phase = Graph_open); + let connection = Option.get !connection in + drive (Core.Sync_event (Sync.Websocket_opened connection)); + let t, payload, code, category, message, checks = + match kind with + | `Cursor -> + ( 2L + , "[[\"~:db/retractEntity\",99999]]" + , Error.Ambiguous_sync_state + , "authoritativeCursorDiscontinuous" + , Some "Authoritative transaction cursors are not continuous." + , [] ) + | `Format -> + ( 1L + , "[[\"~:PRIVATE_SYNTHETIC_PAYLOAD\",99999]]" + , Error.Unsupported_semantics + , "authoritativeDecodeFailed" + , None + , [ "stage=prepare"; "reason=unsupported-operation"; "operation=other" ] + ) + | `Apply_format -> + ( 1L + , "[[\"~:db/add\",null,\"~:block/foo\",\"PRIVATE_SYNTHETIC_PAYLOAD\"]]" + , Error.Unsupported_semantics + , "authoritativeDecodeFailed" + , None + , [ "stage=decode"; "reason=entity-shape"; "field=entity"; "type=null" ] + ) + | `Detail (name, payload, checks) -> + ( (if name = "second-transaction" then 2L else 1L) + , payload + , Error.Unsupported_semantics + , "authoritativeDecodeFailed" + , None + , checks ) + in + let tx : Logseq_sync_pure_reducer.Sync_protocol.Server.pull_transaction = + { t = cursor t; tx = payload; outliner_op = None } + in + let txs = + match kind with + | `Detail ("second-transaction", _, _) -> + [ { tx with t = cursor 1L; tx = "[[\"~:db/retractEntity\",99998]]" } + ; tx + ] + | _ -> [ tx ] + in + drive + (Core.Sync_event + (Sync.Websocket_message + (connection, Pull_ok { t = cursor t; checksum = None; txs }))); + let empty_success=match kind with `Detail (("empty"|"second-transaction"),_,_) -> true | _ -> false in + if empty_success then ( + Alcotest.(check bool) "legal empty authoritative batch succeeds" true (!failures=[]); + Alcotest.(check bool) "empty cursor reaches actual manager" true + ((Core.view !state).sync.snapshot.applied_server_t=Some(cursor t)); + Alcotest.(check (option string)) "successful empty pull has no Last error" None + (Core.view !state).sync.snapshot.last_error; + Alcotest.(check bool) "successful empty pull keeps graph open" true + ((Core.view !state).graph.phase=Graph_open)) else ( + let error = + match !failures with + | [ error ] -> error + | _ -> Alcotest.fail "expected actual authoritative failure" + in + Alcotest.(check bool) + "failure code reflects actual category" + true + (Error.code error = code); + let message_text = Error.message error in + let includes text needle = + let rec loop i = + i + String.length needle <= String.length text + && (String.sub text i (String.length needle) = needle || loop (i + 1)) + in + loop 0 + in + Option.iter + (fun expected -> + Alcotest.(check string) + "producer retains a fixed safe reason" + expected + message_text) + message; + List.iter + (fun expected -> + let absent = String.starts_with ~prefix:"!" expected in + let part = + if absent + then String.sub expected 1 (String.length expected - 1) + else expected + in + Alcotest.(check bool) + ("safe detail " ^ expected) + (not absent) + (includes message_text part)) + checks; + Alcotest.(check (option string)) + "typed origin retains category" + (Some category) + (Error.trace error).origin.code; + Alcotest.(check (option string)) + "actual Last error retains category" + (Some message_text) + (Core.view !state).sync.snapshot.last_error; + Alcotest.(check bool) + "actual Worker diagnostic retains category" + true + (List.exists + (function + | Core.Diagnostic signal -> + String.starts_with + ~prefix:("worker.applyAuthoritative." ^ category) + signal + | _ -> false) + !emitted); + if checks <> [] + then + Alcotest.(check bool) + "validated detail travels in actual diagnostic signal" + true + (List.exists + (function + | Core.Diagnostic signal -> + includes signal "authoritative-decode:v1:" + | _ -> false) + !emitted); + Alcotest.(check bool) + "failure precedes checkpoint advancement" + true + ((Core.view !state).sync.snapshot.applied_server_t = Some (cursor 0L)); + let encoded = + Yojson.Safe.to_string (Error.to_yojson error) + ^ "\n" + ^ String.concat + "\n" + (List.filter_map + (function + | Core.Diagnostic signal -> Some signal + | _ -> None) + !emitted) + in + let contains needle = + let rec loop i = + i + String.length needle <= String.length encoded + && (String.sub encoded i (String.length needle) = needle || loop (i + 1)) + in + loop 0 + in + List.iter + (fun secret -> + Alcotest.(check bool) "source content excluded" false (contains secret)) + ([ root + ; "PRIVATE_SYNTHETIC_PAYLOAD" + ; "99999999-9999-4999-8999-999999999999" + ; "Nothing found" + ; "Cannot store nil" + ] + @ if String.length payload > 30 then [ payload ] else [])))))) +;; + + let () = Alcotest.run "logseq db worker effect runner" - [ ( "ID token cache" + [ ( "authoritative diagnostics", [ + Alcotest.test_case "real cursor rejection reaches Diagnostics" `Quick (test_authoritative_rejection_diagnostics `Cursor); + Alcotest.test_case "real format rejection reaches Diagnostics" `Quick (test_authoritative_rejection_diagnostics `Format); + Alcotest.test_case "apply format failure excludes raw source" `Quick (test_authoritative_rejection_diagnostics `Apply_format) ] @ + List.map (fun ((name,_,_) as detail) -> Alcotest.test_case ("safe detail "^name) `Quick + (test_authoritative_rejection_diagnostics (`Detail detail))) authoritative_detail_cases) + ; ( "ID token cache" , [ Alcotest.test_case "caps reuse at twenty-four hours" `Quick diff --git a/logseq_db_worker/test/test_lui_service.ml b/logseq_db_worker/test/test_lui_service.ml index 60fdba4..b81f15f 100644 --- a/logseq_db_worker/test/test_lui_service.ml +++ b/logseq_db_worker/test/test_lui_service.ml @@ -1262,10 +1262,114 @@ let test_restore_error_reaches_diagnostics () = | _ -> ()); !cleared))) ;; +(* The row allowlist belongs to the service publisher, not the pure reducer. *) +let test_authoritative_diagnostic_rows () = + List.iter (fun (category, expected) -> + let operation, phase, message = Service.diagnostic_report + ("worker.applyAuthoritative." ^ category) in + T.require (operation="applyAuthoritative" && phase="sync") "authoritative row retains its operation"; + T.require (message=expected) "authoritative row erased category %s" category) + [ "authoritativeDatabaseClosed", "Authoritative database is closed." + ; "authoritativeSyncTokenConflict", "Authoritative sync version changed." + ; "authoritativeCursorDiscontinuous", "Authoritative transaction cursors are not continuous." + ; "authoritativeDecodeFailed", "Authoritative transaction format is unsupported." ]; + let unsafe = "worker.applyAuthoritative.PRIVATE_SYNTHETIC_PAYLOAD" in + T.require (Service.diagnostic_report unsafe = Service.diagnostic_report "unknown") + "unknown signal must never display caller content" +;; + +let test_authoritative_detail_rows () = + let prefix = + "worker.applyAuthoritative.authoritativeDecodeFailed:authoritative-decode:v1:" + in + let carrier = + "{\"stage\":\"decode\",\"reason\":\"entity-shape\",\"transaction\":1,\"operationIndex\":2,\"arity\":4,\"operation\":\"add\",\"attribute\":\"custom\",\"field\":\"entity\",\"type\":\"null\"}" + in + let op, phase, message = Service.diagnostic_report (prefix ^ carrier) in + let contains text needle = + let rec loop i = + i + String.length needle <= String.length text + && (String.sub text i (String.length needle) = needle || loop (i + 1)) + in + loop 0 + in + T.require (op = "applyAuthoritative" && phase = "sync") "detail row preserves operation"; + List.iter + (fun part -> T.require (contains message part) "detail row omitted %s" part) + [ "stage=decode" + ; "reason=entity-shape" + ; "transaction=1" + ; "operationIndex=2" + ; "attribute=custom" + ; "type=null" + ]; + List.iter + (fun malicious -> + let _, _, message = Service.diagnostic_report (prefix ^ malicious) in + T.require + (not (contains message "PRIVATE_SYNTHETIC_PAYLOAD")) + "unsafe diagnostic echoed data"; + T.require + (not (contains message "stage=decode")) + "invalid detail was partly accepted") + [ "PRIVATE_SYNTHETIC_PAYLOAD" + ; "{\"stage\":\"decode\",\"reason\":\"PRIVATE_SYNTHETIC_PAYLOAD\"}" + ; "{\"stage\":\"decode\",\"reason\":\"entity-shape\",\"attribute\":\"PRIVATE_SYNTHETIC_PAYLOAD\"}" + ; "{\"stage\":\"decode\",\"reason\":\"entity-shape\",\"transaction\":-1}" + ; "{\"stage\":\"decode\",\"reason\":\"entity-shape\",\"operationIndex\":0}" + ; "{\"stage\":\"decode\",\"reason\":\"entity-shape\",\"arity\":1000000001}" + ; "{\"stage\":\"decode\",\"stage\":\"prepare\",\"reason\":\"entity-shape\"}" + ; "{\"stage\":\"decode\",\"reason\":\"entity-shape\",\"raw\":\"PRIVATE_SYNTHETIC_PAYLOAD\"}" + ; "{\"stage\":\"decode\",\"reason\":\"entity-shape\",\"type\":\"PRIVATE_SYNTHETIC_PAYLOAD\"}" + ; "{\"stage\":\"application\",\"reason\":\"application-exception\",\"operationIndex\":1}" + ; "{\"stage\":\"decode\",\"reason\":\"entity-shape\",\"transaction\":true}" + ; String.make 2048 'x' + ]; + let _, _, legacy = + Service.diagnostic_report + "worker.applyAuthoritative.authoritativeDecodeFailed:PRIVATE_SYNTHETIC_PAYLOAD" + in + T.require (not (contains legacy "PRIVATE_SYNTHETIC_PAYLOAD")) "legacy raw string leaked" +;; + +let test_authoritative_detail_budget () = + let module D = Logseq_db_types.Authoritative_diagnostic in + let detail = + D.create + ~stage:Decode + ~reason:Decrypted_value_count + ~transaction:max_int + ~operation_index:max_int + ~operation_count:max_int + ~lookup_refs:max_int + ~arity:max_int + ~operation:"db.fn/retractAttribute" + ~attribute:"db/cardinality" + ~field:"source-tx" + ~actual:"big-integer" + ~exception_kind:"invalid-argument" + () + in + let encoded = D.encode detail + and message = D.message detail in + T.require + (String.length encoded <= 512 && String.length message <= 512) + "diagnostic exceeds Error budget"; + T.require + (Option.map D.message (D.decode encoded) = Some message) + "largest detail failed roundtrip"; + let _, _, row = Service.diagnostic_report (D.signal_prefix ^ encoded) in + T.require (row = message) "actual row lost maximum bounded detail" +;; + + let () = T.run "lui worker service" - [ T.case "ignored restore failure reaches scoped safe Diagnostics" test_restore_error_reaches_diagnostics + [ T.case "authoritative detail byte budget" test_authoritative_detail_budget + ; T.case "strict authoritative detail rows" test_authoritative_detail_rows + ; T.case "authoritative category row allowlist" test_authoritative_diagnostic_rows + ; T.case "ignored restore failure reaches scoped safe Diagnostics" test_restore_error_reaches_diagnostics ; T.case "local copy confirmed deletion awaits physical removal" (test_local_copy_presence "delete") ; T.case "local copy busy failure recovers" (test_local_copy_presence "busy") ; T.case "local copy absent metadata" (test_local_copy_presence "absent") diff --git a/logseq_db_worker/test/test_pure_reducer.ml b/logseq_db_worker/test/test_pure_reducer.ml index 0ac27a8..5fb0c4a 100644 --- a/logseq_db_worker/test/test_pure_reducer.ml +++ b/logseq_db_worker/test/test_pure_reducer.ml @@ -1131,10 +1131,49 @@ let test_local_restore_worker_failures_are_terminal () = ) [false;true] ;; +(* This tests downstream propagation only: the runner owns producer classification. + Closed/version preparation races have no deterministic public runner hook. *) +let test_authoritative_failure_categories_reach_diagnostics () = + let module Error = Logseq_db_worker.Error in + List.iter (fun (code, category, message) -> + let ready, graph = worker_open_graph_transition () in + let connection = List.find_map (function + | Core.Run_sync (Sync.Start_websocket request) -> Some request.scope + | _ -> None) ready.effects |> Option.get in + let opened = Core.step ready.next (Sync_event (Sync.Websocket_opened connection)) in + let cursor = Logseq_db_types.Server_cursor.of_int64 1L |> Result.get_ok in + let tx : Logseq_sync_pure_reducer.Sync_protocol.Server.pull_transaction = + {t=cursor;tx="[[\"~:db/retractEntity\",99999]]";outliner_op=None} in + let pending = Core.step opened.next (Sync_event (Sync.Websocket_message + (connection, Pull_ok {t=cursor;checksum=None;txs=[tx]}))) in + let ticket = List.find_map (function + | Core.Run_worker (Core.Request (ticket, Handle_sync_worker_effect (Sync.Apply_authoritative_batch _))) -> Some ticket + | _ -> None) pending.effects |> Option.get in + let origin = Error.create_cause ~component:Managed_sync ~operation:"applyAuthoritative" + ~code:(Some category) ~message |> Result.get_ok in + let error = Error.create_with_origin ~code ~message ~details:[] ~origin |> Result.get_ok in + let failure = Core.Runner_completed (Sync_worker_effect_completed (ticket, Error error)) in + let failed = Core.step pending.next failure in + Alcotest.(check (option string)) "Last error retains category" (Some message) + (Core.view failed.next).sync.snapshot.last_error; + Alcotest.(check bool) "Worker diagnostics retain category" true + (List.exists (function Core.Publish (Diagnostic signal) -> + signal="worker.applyAuthoritative." ^ category | _ -> false) failed.effects); + Alcotest.(check bool) "local graph remains open" true + ((Core.view failed.next).graph.phase=Graph_open && graph.graph_generation=(Core.view failed.next).graph.generation); + Alcotest.(check int) "duplicate completion is inert" 0 + (List.length (Core.step failed.next failure).effects)) + [ Error.Closed_session, "authoritativeDatabaseClosed", "Authoritative database is closed." + ; Conflict, "authoritativeSyncTokenConflict", "Authoritative sync version changed." + ; Ambiguous_sync_state, "authoritativeCursorDiscontinuous", "Authoritative transaction cursors are not continuous." + ; Unsupported_semantics, "authoritativeDecodeFailed", "Authoritative transaction format is unsupported." ] +;; + let () = Alcotest.run "logseq db worker pure reducer" - [ ( "snapshot failure", [ + [ ( "authoritative diagnostics", [Alcotest.test_case "four categories propagate without closing graph" `Quick test_authoritative_failure_categories_reach_diagnostics]) + ; ( "snapshot failure", [ Alcotest.test_case "terminal failure and explicit retry" `Quick test_snapshot_failure_is_terminal_and_retryable; Alcotest.test_case "success and late graph switch completions" `Quick test_snapshot_success_and_late_switch_completions ]) ; ( "asset session" diff --git a/logseq_overlay_db/lib/database.ml b/logseq_overlay_db/lib/database.ml index 230dfca..dc9d957 100644 --- a/logseq_overlay_db/lib/database.ml +++ b/logseq_overlay_db/lib/database.ml @@ -7036,6 +7036,18 @@ let authoritative_crypto_item_id database revision index = (Graph.Uuid.to_string database.graph_uuid)) ;; +module Authoritative_diagnostic = Logseq_db_types.Authoritative_diagnostic + +let authoritative_decode_context ~stage ~transaction source = + let detail = + match Authoritative_diagnostic.decode source with + | Some detail -> detail + | None -> Authoritative_diagnostic.create ~stage ~reason:Unknown () + in + Authoritative_diagnostic.with_context ~stage ~transaction detail + |> Authoritative_diagnostic.encode +;; + let begin_authoritative database ~expected batch = Eio.Mutex.use_rw ~protect:true database.lock (fun () -> if database.closed @@ -7063,13 +7075,16 @@ let begin_authoritative database ~expected batch = |> Types.encoded_transaction_to_string) transactions in - let rec collect acc = function + let rec collect index acc = function | [] -> Ok (List.rev acc |> List.concat) | wire :: rest -> - Result.bind (Sync_tx_codec.protected_values wire) (fun values -> - collect (values :: acc) rest) + Result.bind + (Sync_tx_codec.protected_values ~allow_empty:true wire + |> Result.map_error + (authoritative_decode_context ~stage:Prepare ~transaction:index)) + (fun values -> collect (index + 1) (values :: acc) rest) in - match collect [] wires with + match collect 1 [] wires with | Error message -> Error (Types.Authoritative_decode_failed message) | Ok protected -> let request = @@ -7597,6 +7612,77 @@ let same_authoritative_decrypted_input left right = | None, Some _ | Some _, None -> false ;; +(* Structural statistics only; neither reference values nor exception strings leave this layer. *) +let authoritative_lookup_refs operations = + (* A diagnostic must not replace the caught application failure. Bound both + traversal work and depth, and omit incomplete statistics. *) + try + let remaining = ref 1024 in + let spend depth = + if depth > 64 || !remaining = 0 then raise Exit; + decr remaining + in + let rec entity depth = function + | Datascript.Lookup_ref (_, v) -> + spend depth; + 1 + value (depth + 1) v + | _ -> + spend depth; + 0 + and value depth v = + spend depth; + match v with + | Datascript.Ref_to e -> entity (depth + 1) e + | List vs | Vector vs | Set vs -> + List.fold_left (fun n v -> n + value (depth + 1) v) 0 vs + | Tuple vs -> + List.fold_left + (fun n v -> n + Option.fold ~none:0 ~some:(value (depth + 1)) v) + 0 + vs + | Map vs -> + List.fold_left + (fun n (k, v) -> n + value (depth + 1) k + value (depth + 1) v) + 0 + vs + | _ -> 0 + in + Some + (List.fold_left + (fun n op -> + spend 0; + match op with + | Datascript.Add (e, _, v) -> n + entity 1 e + value 1 v + | Retract (e, _, v) -> n + entity 1 e + Option.fold ~none:0 ~some:(value 1) v + | RetractAttr (e, _) | RetractEntity e -> n + entity 1 e + | CompareAndSet (e, _, old, v) -> + n + entity 1 e + Option.fold ~none:0 ~some:(value 1) old + value 1 v + | _ -> n) + 0 + operations) + with + | _ -> None +;; + +let authoritative_exception_kind = function + | Invalid_argument _ -> "invalid-argument" + | Failure _ -> "failure" + | Not_found -> "not-found" + | _ -> "other" +;; + +let authoritative_application_error ~transaction operations exn = + Authoritative_diagnostic.create + ~stage:Application + ~reason:Application_exception + ~transaction + ~operation_count:(List.length operations) + ?lookup_refs:(authoritative_lookup_refs operations) + ~exception_kind:(authoritative_exception_kind exn) + () + |> Authoritative_diagnostic.encode +;; + let prepare_authoritative_candidate preparation ~decrypted = match preparation.authoritative_state with | Authoritative_ready candidate @@ -7618,41 +7704,79 @@ let prepare_authoritative_candidate preparation ~decrypted = | Some _, Some _ -> Error Types.Authoritative_crypto_unexpected in Result.bind decrypted_values (fun plaintexts -> - let rec decode database decoded expected_tx_data roots_after remaining wires = + let rec decode index database decoded expected_tx_data roots_after remaining wires = match wires with | [] -> if remaining = [] then Ok (List.rev decoded, List.rev expected_tx_data, List.rev roots_after, database) - else Error "decrypted value count does not match protected values" + else + Error + (Authoritative_diagnostic.create + ~stage:Decode + ~reason:Decrypted_value_count + () + |> Authoritative_diagnostic.encode) | wire :: rest -> - Result.bind (Sync_tx_codec.protected_values wire) (fun protected -> - Result.bind - (split_prefix (List.length protected) remaining) - (fun (plain, remaining) -> - Result.bind - (Sync_tx_codec.decode ~db:database ~decrypted_values:plain wire) - (fun operations -> - try - let report = - Datascript.with_tx - ~tx_meta:[ "skip-store?", Datascript.Bool true ] - database - operations - in - decode - report.db_after - (operations :: decoded) - (report.tx_data :: expected_tx_data) - (report.db_after :: roots_after) - remaining - rest - with - | exn -> Error (Printexc.to_string exn)))) + let current = + Result.bind (Sync_tx_codec.protected_values ~allow_empty:true wire) (fun protected -> + Result.bind + (split_prefix (List.length protected) remaining + |> Result.map_error (fun _ -> + Authoritative_diagnostic.create + ~stage:Decode + ~reason:Decrypted_value_count + () + |> Authoritative_diagnostic.encode)) + (fun (plain, remaining) -> + Result.bind + (Sync_tx_codec.decode ~allow_empty:true ~db:database ~decrypted_values:plain wire) + (fun operations -> + try + let report = + Datascript.with_tx + ~tx_meta:[ "skip-store?", Datascript.Bool true ] + database + operations + in + Ok (report, operations, remaining) + with + | exn -> + Error + (authoritative_application_error + ~transaction:index + operations + exn)))) + |> Result.map_error (fun source -> + match Authoritative_diagnostic.decode source with + | Some detail -> + Authoritative_diagnostic.with_context ~transaction:index detail + |> Authoritative_diagnostic.encode + | None -> + authoritative_decode_context ~stage:Decode ~transaction:index source) + in + Result.bind current (fun (report, operations, remaining) -> + (* Preserve the existing catch around recursive decoding, but do not + attribute an unexpected later decode exception to this with_tx. *) + try + decode + (index + 1) + report.db_after + (operations :: decoded) + (report.tx_data :: expected_tx_data) + (report.db_after :: roots_after) + remaining + rest + with exn -> + Error (Authoritative_diagnostic.create ~stage:Decode ~reason:Unknown + ?transaction:(if rest=[] then None else Some(index+1)) + ~exception_kind:(authoritative_exception_kind exn) () + |> Authoritative_diagnostic.encode)) in match decode + 1 (authoritative_database preparation.authoritative_owner) [] [] @@ -8863,15 +8987,28 @@ let logical_change_footprint ~maximum:before.owner.dependencies.limits.change_max_items in let favorite_memberships, favorite_overflow = - favorite_membership_dependents authoritative_before authoritative_after transaction_data + favorite_membership_dependents + authoritative_before + authoritative_after + transaction_data ~maximum:before.owner.dependencies.limits.change_max_items in let asset_holders, asset_overflow = - asset_reference_dependents before after authoritative_before authoritative_after transaction_data + asset_reference_dependents + before + after + authoritative_before + authoritative_after + transaction_data ~maximum:before.owner.dependencies.limits.change_max_items in - let asset_pages, asset_blocks = List.partition (fun uuid -> - Option.is_some (logical_page_at before uuid) || Option.is_some (logical_page_at after uuid)) asset_holders in + let asset_pages, asset_blocks = + List.partition + (fun uuid -> + Option.is_some (logical_page_at before uuid) + || Option.is_some (logical_page_at after uuid)) + asset_holders + in let outbox_blocks, outbox_pages = List.fold_left (fun (blocks, pages) (footprint : effect_footprint) -> @@ -8926,7 +9063,8 @@ let logical_change_footprint let after_value = logical_block_at after uuid in before_value <> after_value) block_candidates - |> fun changed -> List.sort_uniq Graph.Uuid.compare (changed @ favorite_memberships @ asset_blocks) + |> fun changed -> + List.sort_uniq Graph.Uuid.compare (changed @ favorite_memberships @ asset_blocks) in let parents, membership_pages = List.fold_left @@ -9466,7 +9604,9 @@ let commit_authoritative_candidate database prepared = |> Result.get_ok in let outbox_delta, durable_outbox = - outbox_write_set database ~revision:(database.sync_revision + 1) + outbox_write_set + database + ~revision:(database.sync_revision + 1) candidate_outbox in let receipts = @@ -9530,7 +9670,10 @@ let commit_authoritative_candidate database prepared = transitioned_records) in let changed = - fanout_overflow || block_uuids <> [] || page_uuids <> [] || structure_interests <> [] + fanout_overflow + || block_uuids <> [] + || page_uuids <> [] + || structure_interests <> [] in database.projection <- (if changed then candidate_projection else before_projection); diff --git a/logseq_overlay_db/lib/sync_tx_codec.ml b/logseq_overlay_db/lib/sync_tx_codec.ml index eec9238..6a48a32 100644 --- a/logseq_overlay_db/lib/sync_tx_codec.ml +++ b/logseq_overlay_db/lib/sync_tx_codec.ml @@ -18,6 +18,80 @@ let map_all decode values = loop [] values ;; +module Diagnostic = Logseq_db_types.Authoritative_diagnostic + +let transit_kind = function + | Transit.Null -> "null" + | Bool _ -> "bool" + | String _ -> "string" + | Int _ -> "integer" + | Int64 _ -> "int64" + | Float _ -> "float" + | Binary _ -> "binary" + | Big_decimal _ -> "decimal" + | Big_int _ -> "big-integer" + | Date _ -> "date" + | Uuid _ -> "uuid" + | Uri _ -> "uri" + | Keyword _ -> "keyword" + | Symbol _ -> "symbol" + | Array _ -> "array" + | Map _ -> "map" + | Set _ -> "set" + | List _ -> "list" + | Tagged _ -> "tagged" +;; + +let diagnostic_error ?field ?actual reason = + Error (Diagnostic.encode (Diagnostic.create ~stage:Decode ~reason ?field ?actual ())) +;; + +let diagnostic_context + ?field + ?operation_index + ?operation_count + ?arity + ?operation + ?attribute + result + = + Result.map_error + (fun source -> + let detail = + match Diagnostic.decode source with + | Some detail -> detail + | None -> Diagnostic.create ~stage:Decode ~reason:Unknown () + in + Diagnostic.with_context + ?field + ?operation_index + ?operation_count + ?arity + ?operation + ?attribute + detail + |> Diagnostic.encode) + result +;; + +let operation_context index count operation result = + let arity, name, attribute = + match operation with + | Transit.Array (Keyword op :: _entity :: Keyword attr :: _ as fields) -> + Some (List.length fields), Some op, Some attr + | Array (Keyword op :: _ as fields) -> Some (List.length fields), Some op, None + | Array fields -> Some (List.length fields), None, None + | _ -> None, None, None + in + diagnostic_context + ~operation_index:index + ~operation_count:count + ?arity + ?operation:name + ?attribute + result +;; + let lookup_value entity attr = match Datascript.Entity.entity_attr_raw entity attr with | Some (Datascript.One_value value) -> Some value @@ -256,13 +330,13 @@ let integer = function | Int64 value when Int64.compare value (Int64.of_int min_int) >= 0 && Int64.compare value (Int64.of_int max_int) <= 0 -> Ok (Int64.to_int value) - | _ -> Error "expected a Transit integer" + | value -> diagnostic_error ~actual:(transit_kind value) Integer_range ;; let integer64 = function | Transit.Int value -> Ok (Int64.of_int value) | Int64 value -> Ok value - | _ -> Error "expected a Transit integer" + | value -> diagnostic_error ~actual:(transit_kind value) Integer_value ;; let rec value_of_transit = function @@ -276,11 +350,11 @@ let rec value_of_transit = function | Big_decimal value -> (match float_of_string_opt value with | Some value -> Ok (Float value) - | None -> Error "invalid Transit decimal") + | None -> diagnostic_error ~actual:"decimal" Decimal_value) | Big_int value -> (match Int64.of_string_opt value with | Some value -> value_of_transit (Transit.Int64 value) - | None -> Error "invalid Transit integer") + | None -> diagnostic_error ~actual:"big-integer" Integer_value) | Date value -> Ok (Instant value) | Uuid value -> Ok (Uuid value) | Uri value -> Ok (String value) @@ -303,7 +377,7 @@ let rec value_of_transit = function bind (value_of_transit value) (fun value -> Ok (Vector [ String tag; value ])) ;; -let entity_ref_of_transit = function +let entity_ref_of_transit ?(reason = Diagnostic.Entity_shape) = function | Transit.Int value -> Ok (Entity_id value) | Int64 value -> bind (integer (Transit.Int64 value)) (fun value -> Ok (Entity_id value)) @@ -312,7 +386,7 @@ let entity_ref_of_transit = function | Keyword value -> Ok (Ident value) | Array [ Keyword attr; value ] -> bind (value_of_transit value) (fun value -> Ok (Lookup_ref (attr, value))) - | _ -> Error "transaction entity must be an eid, tempid, ident, or lookup ref" + | value -> diagnostic_error ~actual:(transit_kind value) reason ;; let schema_attr db attr = List.assoc_opt attr (Datascript.schema db) @@ -320,14 +394,16 @@ let schema_attr db attr = List.assoc_opt attr (Datascript.schema db) let tx_value ~db ~attr value = match schema_attr db attr with | Some { Datascript.value_type = Some RefType; _ } -> - bind (entity_ref_of_transit value) (fun entity -> Ok (Ref_to entity)) + bind (entity_ref_of_transit ~reason:Reference_shape value) (fun entity -> + Ok (Ref_to entity)) | None | Some _ -> value_of_transit value ;; let valid_source_tx = function | Transit.Int _ -> Ok () | Int64 value -> bind (integer (Transit.Int64 value)) (fun _ -> Ok ()) - | _ -> Error "normalized transaction source tx must be an integer" + | value -> + diagnostic_error ~field:"source-tx" ~actual:(transit_kind value) Source_tx_type ;; let plaintext_value attribute value plaintexts = @@ -335,8 +411,8 @@ let plaintext_value attribute value plaintexts = then ( match value, plaintexts with | Transit.String _, plaintext :: rest -> Ok (Transit.String plaintext, rest) - | String _, [] -> Error "decrypted value count does not match protected values" - | _, _ -> Error "protected sync attributes must contain strings") + | String _, [] -> diagnostic_error Decrypted_value_count + | value, _ -> diagnostic_error ~actual:(transit_kind value) Protected_value_type) else Ok (value, plaintexts) ;; @@ -345,14 +421,22 @@ let decode_add_or_retract ~db constructor fields plaintexts = | [ entity; Transit.Keyword attr; value ] | [ entity; Transit.Keyword attr; value; Transit.Int _ ] | [ entity; Transit.Keyword attr; value; Transit.Int64 _ ] -> - bind (entity_ref_of_transit entity) (fun entity -> - bind (plaintext_value attr value plaintexts) (fun (value, plaintexts) -> - bind (tx_value ~db ~attr value) (fun value -> - Ok (constructor entity attr value, plaintexts)))) - | [ _; _; _; source_tx ] -> + bind + (diagnostic_context ~field:"entity" (entity_ref_of_transit entity)) + (fun entity -> + bind (plaintext_value attr value plaintexts) (fun (value, plaintexts) -> + bind + (diagnostic_context ~field:"value" (tx_value ~db ~attr value)) + (fun value -> Ok (constructor entity attr value, plaintexts)))) + | [ _; attribute; _; source_tx ] -> bind (valid_source_tx source_tx) (fun () -> - Error "normalized add/retract operation has invalid fields") - | _ -> Error "normalized add/retract operation has invalid fields" + diagnostic_error + ~field:"attribute" + ~actual:(transit_kind attribute) + Operation_fields) + | [ _; attribute; _ ] -> + diagnostic_error ~field:"attribute" ~actual:(transit_kind attribute) Operation_fields + | _ -> diagnostic_error Operation_fields ;; let decode_operation ~db operation plaintexts = @@ -372,50 +456,75 @@ let decode_operation ~db operation plaintexts = | Array [ Keyword ("db.fn/retractAttribute" | "db/retractAttribute"); entity; Keyword attr ] -> - bind (entity_ref_of_transit entity) (fun entity -> - Ok (RetractAttr (entity, attr), plaintexts)) + bind + (diagnostic_context ~field:"entity" (entity_ref_of_transit entity)) + (fun entity -> Ok (RetractAttr (entity, attr), plaintexts)) | Array [ Keyword "db/retractEntity"; entity ] -> - bind (entity_ref_of_transit entity) (fun entity -> - Ok (RetractEntity entity, plaintexts)) + bind + (diagnostic_context ~field:"entity" (entity_ref_of_transit entity)) + (fun entity -> Ok (RetractEntity entity, plaintexts)) | Array [ Keyword ("db.fn/cas" | "db/cas"); entity; Keyword attr; expected; value ] -> - bind (entity_ref_of_transit entity) (fun entity -> - bind - (match expected with - | Transit.Null -> Ok (None, plaintexts) - | expected -> - bind (plaintext_value attr expected plaintexts) (fun (expected, plaintexts) -> - bind (tx_value ~db ~attr expected) (fun expected -> - Ok (Some expected, plaintexts)))) - (fun (expected, plaintexts) -> - bind (plaintext_value attr value plaintexts) (fun (value, plaintexts) -> - bind (tx_value ~db ~attr value) (fun value -> - Ok (CompareAndSet (entity, attr, expected, value), plaintexts))))) + bind + (diagnostic_context ~field:"entity" (entity_ref_of_transit entity)) + (fun entity -> + bind + (match expected with + | Transit.Null -> Ok (None, plaintexts) + | expected -> + bind + (diagnostic_context + ~field:"expected" + (plaintext_value attr expected plaintexts)) + (fun (expected, plaintexts) -> + bind + (diagnostic_context ~field:"expected" (tx_value ~db ~attr expected)) + (fun expected -> Ok (Some expected, plaintexts)))) + (fun (expected, plaintexts) -> + bind (plaintext_value attr value plaintexts) (fun (value, plaintexts) -> + bind + (diagnostic_context ~field:"value" (tx_value ~db ~attr value)) + (fun value -> + Ok (CompareAndSet (entity, attr, expected, value), plaintexts))))) | Array (Keyword operation :: _) -> - Error ("unsupported normalized transaction operation: " ^ operation) - | Array _ -> Error "normalized transaction operation must start with a keyword" - | _ -> Error "normalized transaction entries must be operation arrays" + diagnostic_error + (if + List.mem + operation + [ "db/add" + ; "db/retract" + ; "db.fn/cas" + ; "db/cas" + ; "db.fn/retractAttribute" + ; "db/retractAttribute" + ; "db/retractEntity" + ] + then Operation_fields + else Unsupported_operation) + | Array fields -> + diagnostic_error + ?actual:(List.nth_opt fields 0 |> Option.map transit_kind) + Operation_keyword + | value -> diagnostic_error ~actual:(transit_kind value) Operation_shape ;; -let parse source = +let parse ?(allow_empty = false) source = try match Codec.of_string source with - | Transit.Array [] | List [] -> Error "normalized transaction must not be empty" + | (Transit.Array [] | List []) when not allow_empty -> diagnostic_error Empty_transaction | Array operations | List operations -> Ok operations - | _ -> Error "normalized transaction must be a Transit array or list" + | value -> diagnostic_error ~actual:(transit_kind value) Transaction_shape with - | Transit.Decode_error message - | Yojson.Json_error message - | Failure message - | Invalid_argument message -> Error ("invalid transaction Transit: " ^ message) + | Transit.Decode_error _ | Yojson.Json_error _ | Failure _ | Invalid_argument _ -> + diagnostic_error Transit_syntax ;; -let protected_values source = - let value attribute value = +let protected_values ?(allow_empty = false) source = + let value ?(field = "value") attribute value = if List.mem attribute protected_attributes then ( match value with | Transit.String ciphertext -> Ok [ ciphertext ] - | _ -> Error "protected sync attributes must contain encrypted strings") + | value -> diagnostic_error ~field ~actual:(transit_kind value) Protected_value_type) else Ok [] in let operation = function @@ -428,37 +537,63 @@ let protected_values source = bind (match expected with | Transit.Null -> Ok [] - | expected -> value attribute expected) + | expected -> value ~field:"expected" attribute expected) (fun expected -> bind (value attribute value_) (fun value -> Ok (expected @ value))) | Array [ Keyword ("db.fn/retractAttribute" | "db/retractAttribute"); _; Keyword _ ] | Array [ Keyword "db/retractEntity"; _ ] -> Ok [] | Array (Keyword operation :: _) -> - Error ("unsupported normalized transaction operation: " ^ operation) - | Array _ -> Error "normalized transaction operation must start with a keyword" - | _ -> Error "normalized transaction entries must be operation arrays" + diagnostic_error + (if + List.mem + operation + [ "db/add" + ; "db/retract" + ; "db.fn/cas" + ; "db/cas" + ; "db.fn/retractAttribute" + ; "db/retractAttribute" + ; "db/retractEntity" + ] + then Operation_fields + else Unsupported_operation) + | Array fields -> + diagnostic_error + ?actual:(List.nth_opt fields 0 |> Option.map transit_kind) + Operation_keyword + | value -> diagnostic_error ~actual:(transit_kind value) Operation_shape in - bind (parse source) (fun operations -> - let rec loop values = function + bind (parse ~allow_empty source) (fun operations -> + let operation_count = List.length operations in + let rec loop index values = function | [] -> Ok (List.concat (List.rev values)) | operation_ :: rest -> - bind (operation operation_) (fun current -> loop (current :: values) rest) + bind + (operation_context index operation_count operation_ (operation operation_)) + (fun current -> loop (index + 1) (current :: values) rest) in - loop [] operations) + loop 1 [] operations) ;; -let decode ~db ~decrypted_values source = - bind (parse source) (fun operations -> - let rec loop decoded plaintexts = function +let decode ?(allow_empty = false) ~db ~decrypted_values source = + bind (parse ~allow_empty source) (fun operations -> + let operation_count = List.length operations in + let rec loop index decoded plaintexts = function | [] -> if plaintexts = [] then Ok (List.rev decoded) - else Error "decrypted value count does not match protected values" + else diagnostic_error Decrypted_value_count | operation :: rest -> - bind (decode_operation ~db operation plaintexts) (fun (operation, plaintexts) -> - loop (operation :: decoded) plaintexts rest) + bind + (operation_context + index + operation_count + operation + (decode_operation ~db operation plaintexts)) + (fun (operation, plaintexts) -> + loop (index + 1) (operation :: decoded) plaintexts rest) in - loop [] decrypted_values operations) + loop 1 [] decrypted_values operations) ;; let inserted_orders source = diff --git a/logseq_overlay_db/lib/sync_tx_codec.mli b/logseq_overlay_db/lib/sync_tx_codec.mli index ab0742d..4479495 100644 --- a/logseq_overlay_db/lib/sync_tx_codec.mli +++ b/logseq_overlay_db/lib/sync_tx_codec.mli @@ -1,4 +1,5 @@ -val protected_values : string -> (string list, string) result +(** Empty operations are accepted only when explicitly enabled for authoritative input. *) +val protected_values : ?allow_empty:bool -> string -> (string list, string) result val protected_plaintexts : Datascript.tx_op list -> (string list, string) result val synchronized_operations : Datascript.tx_op list -> Datascript.tx_op list val encode : db:Datascript.db -> Datascript.tx_op list -> (string, string) result @@ -10,7 +11,8 @@ val encode_protected -> (string, string) result val decode - : db:Datascript.db + : ?allow_empty:bool + -> db:Datascript.db -> decrypted_values:string list -> string -> (Datascript.tx_op list, string) result diff --git a/logseq_overlay_db/test/test_overlay_storage.ml b/logseq_overlay_db/test/test_overlay_storage.ml index 9b75b86..bc7145c 100644 --- a/logseq_overlay_db/test/test_overlay_storage.ml +++ b/logseq_overlay_db/test/test_overlay_storage.ml @@ -1499,7 +1499,7 @@ let pre_uuid_status_outbox_preserves_identity () = "opening legacy outbox changed operation fingerprints") ;; -let noncanonical_outbox_payload_fails_closed () = +let noncanonical_outbox_payload_fails_closed ?(payload="tampered") () = let behavior = "noncanonical durable outbox payload fails closed" in T.with_temp_directory "overlay-noncanonical-outbox-" (fun support -> let database_path = T.seed_mirror support in @@ -1532,7 +1532,7 @@ let noncanonical_outbox_payload_fails_closed () = (match Yojson.Safe.from_string source with | `Assoc fields -> `Assoc - (("normalized_transaction", `String "tampered") + (("normalized_transaction", `String payload) :: List.remove_assoc "normalized_transaction" fields) |> Yojson.Safe.to_string | _ -> Alcotest.fail "queued fixture record is not JSON") @@ -3086,8 +3086,9 @@ let cases = duplicate_outbox_member_fails_closed ; Alcotest.test_case "noncanonical outbox payload fails closed" - `Quick - noncanonical_outbox_payload_fails_closed + `Quick (noncanonical_outbox_payload_fails_closed ~payload:"tampered") + ; Alcotest.test_case "empty outgoing transaction stays corrupt" `Quick + (noncanonical_outbox_payload_fails_closed ~payload:"[]") ; Alcotest.test_case "inconsistent origin evidence fails closed" `Quick diff --git a/logseq_overlay_db/test/test_overlay_sync.ml b/logseq_overlay_db/test/test_overlay_sync.ml index 846d017..c9f0fc1 100644 --- a/logseq_overlay_db/test/test_overlay_sync.ml +++ b/logseq_overlay_db/test/test_overlay_sync.ml @@ -43,6 +43,302 @@ let commit_plain_authoritative database ~cursor wire behavior = | Authoritative_deferred _ -> Alcotest.fail "ordinary authoritative batch was deferred" ;; +(* The pure sync reducer carries opaque wire text. The public Database boundary + executes the actual codec, candidate, durability and replay for this defect. *) +let empty_authoritative_batch wires = + let transactions = + List.mapi + (fun index wire -> + authoritative_transaction + ~cursor:(server_cursor (index + 1)) + ~transaction:(encoded wire)) + wires + in + authoritative_batch + ~maximum_count:16 + ~maximum_bytes:4096 + ~transactions + ~through:(server_cursor (List.length wires)) + ~checksum:None + |> T.require_ok ~behavior:"construct empty-compatible authoritative batch" +;; + +let apply_empty_compatible_batch database batch = + let behavior = "apply empty-compatible authoritative batch" in + let sync = Database.inspect_sync database |> T.require_ok ~behavior in + let prepared, crypto = + Database.begin_authoritative database ~expected:(sync_view_token sync) batch + |> T.require_ok ~behavior + in + T.require (Option.is_none crypto) "empty/plain batch requested crypto"; + match + Database.apply_authoritative database prepared ~decrypted:None + |> T.require_ok ~behavior + with + | Authoritative_applied commit -> commit + | Authoritative_deferred _ -> Alcotest.fail "empty/plain batch unexpectedly deferred" +;; + +let empty_batch_state database = + let behavior = "inspect empty authoritative regression" in + let sync = Database.inspect_sync database |> T.require_ok ~behavior in + let snapshot = Database.current_snapshot database |> T.require_ok ~behavior in + Fun.protect + ~finally:(fun () -> Database.release_snapshot snapshot) + (fun () -> + let version = Database.snapshot_version snapshot in + let updated = + match + Database.get_outline_blocks snapshot [ T.authoritative_block_uuid ] + |> T.require_ok ~behavior + with + | [ Present_block { value; _ } ] -> value.block.updated_at_ms + | _ -> Alcotest.fail "authoritative fixture block missing" + in + sync, version.projection_revision, updated) +;; + +let empty_batch_update_wire = + "[[\"~:db/add\",[\"~:block/uuid\",\"~u22222222-2222-4222-8222-222222222224\"],\"~:block/updated-at\",1704067200123]]" +;; + +let empty_authoritative_preserves_logical_state wire database = + let before, projection, updated = empty_batch_state database in + let commit = + apply_empty_compatible_batch database (empty_authoritative_batch [ wire ]) + in + let after, after_projection, after_updated = empty_batch_state database in + T.require + (Server_cursor.equal (sync_view_checkpoint after) (server_cursor 1)) + "empty tx did not advance checkpoint"; + T.require + (not (sync_token_equal (sync_view_token before) (sync_view_token after))) + "empty tx did not advance sync revision"; + T.require + (Projection_revision.equal projection after_projection && updated = after_updated) + "empty tx changed logical data/revision"; + T.require + (commit.logical_change_summary = No_logical_change) + "empty tx emitted logical change"; + T.require (sync_view_submissions after = []) "empty tx fabricated outbox submissions"; + match + Database.begin_authoritative + database + ~expected:(sync_view_token before) + (empty_authoritative_batch [ wire ]) + with + | Error Authoritative_sync_token_conflict -> () + | _ -> Alcotest.fail "successful empty transaction left its old token usable" +;; + +let empty_authoritative_mixed_batch database = + let _before, projection, _updated = empty_batch_state database in + let commit = + apply_empty_compatible_batch + database + (empty_authoritative_batch [ "[]"; empty_batch_update_wire; "[]" ]) + in + let after, after_projection, updated = empty_batch_state database in + T.require + (Server_cursor.equal commit.checkpoint (server_cursor 3) + && Server_cursor.equal (sync_view_checkpoint after) (server_cursor 3)) + "mixed batch dropped empty cursor slots"; + T.require + (updated = 1_704_067_200_123L + && not (Projection_revision.equal projection after_projection)) + "mixed batch failed to apply real operation" +;; + +let with_empty_authoritative_storage run = + let behavior = "empty authoritative durability" in + T.with_temp_directory "overlay-empty-authoritative-" (fun support -> + let path = T.seed_mirror support in + let dependencies = T.dependencies ~behavior in + Eio_main.run (fun _ -> + Eio.Switch.run (fun sw -> + let handles = ref [] in + let open_database () = + let inspection = + Database.inspect_mirror + ~application_support_directory:support + ~graph_id:T.graph_uuid + |> T.require_ok ~behavior + in + let database = + Database.open_ + ~sw + dependencies + inspection + ~graph_name:"empty-authoritative-fixture" + |> T.require_ok ~behavior + in + handles := database :: !handles; + database + in + Fun.protect + ~finally:(fun () -> + List.iter + (fun db -> + try ignore (Database.close db) with + | _ -> ()) + !handles) + (fun () -> run path open_database (open_database ()))))) +;; + +let empty_authoritative_survives_reopen () = + with_empty_authoritative_storage (fun _path open_database database -> + let before, projection, updated = empty_batch_state database in + ignore (apply_empty_compatible_batch database (empty_authoritative_batch [ "[]" ])); + let committed, _, _ = empty_batch_state database in + Database.close database |> T.require_ok ~behavior:"close empty authoritative"; + let reopened = open_database () in + let restored, restored_projection, restored_updated = empty_batch_state reopened in + T.require + (Server_cursor.equal (sync_view_checkpoint restored) (server_cursor 1)) + "reopen lost empty server cursor"; + T.require + (sync_token_equal (sync_view_token committed) (sync_view_token restored) + && not (sync_token_equal (sync_view_token before) (sync_view_token restored))) + "reopen lost sync revision"; + T.require + (Projection_revision.equal projection restored_projection + && updated = restored_updated) + "reopen changed empty logical state"; + ignore + (commit_plain_authoritative + reopened + ~cursor:(server_cursor 2) + empty_batch_update_wire + "next cursor after empty reopen")) +;; + +let empty_authoritative_failure_is_atomic ~storage () = + with_empty_authoritative_storage (fun path open_database database -> + let before, projection, updated = empty_batch_state database in + let invalid = "[[\"~:db/add\",null,\"~:block/foo\",1]]" in + let wires = + if storage + then [ "[]"; empty_batch_update_wire; "[]" ] + else [ "[]"; empty_batch_update_wire; invalid ] + in + let batch = empty_authoritative_batch wires in + let prepared, _ = + Database.begin_authoritative database ~expected:(sync_view_token before) batch + |> T.require_ok ~behavior:"prepare empty-containing rollback" + in + let sql command = + let db = Sqlite3.db_open ~mode:`NO_CREATE path in + Fun.protect + ~finally:(fun () -> ignore (Sqlite3.db_close db)) + (fun () -> Sqlite3.Rc.check (Sqlite3.exec db command)) + in + if storage + then + sql + "CREATE TRIGGER fail_empty_checkpoint BEFORE UPDATE ON sync_meta BEGIN SELECT \ + RAISE(ABORT, 'synthetic empty metadata failure'); END"; + (match Database.apply_authoritative database prepared ~decrypted:None with + | Error (Authoritative_commit_persistence_failed _) when storage -> () + | Error (Authoritative_decode_failed source) when not storage -> + let module D = Logseq_db_types.Authoritative_diagnostic in + let detail = + match D.decode source with + | Some detail -> D.message detail + | None -> Alcotest.fail "late failure lost safe diagnostic" + in + let contains needle = + let rec loop i = + i + String.length needle <= String.length detail + && (String.sub detail i (String.length needle) = needle || loop (i + 1)) + in + loop 0 + in + T.require + (contains "stage=decode" && contains "transaction=3") + "empty slots shifted the late failure transaction ordinal" + | _ -> Alcotest.fail "empty-containing failure returned wrong outcome"); + if storage then sql "DROP TRIGGER fail_empty_checkpoint"; + let assert_unchanged database = + let after, after_projection, after_updated = empty_batch_state database in + T.require + (Server_cursor.equal (sync_view_checkpoint before) (sync_view_checkpoint after) + && sync_token_equal (sync_view_token before) (sync_view_token after)) + "failed batch advanced checkpoint/revision"; + T.require + (Projection_revision.equal projection after_projection && updated = after_updated) + "failed mixed batch exposed partial data"; + T.require (sync_view_submissions after = []) "failed batch changed outbox" + in + assert_unchanged database; + Database.close database |> T.require_ok ~behavior:"close failed empty batch"; + assert_unchanged (open_database ())) +;; + +let empty_authoritative_keeps_token_guards database = + let before = + Database.inspect_sync database |> T.require_ok ~behavior:"before empty guards" + in + let empty = empty_authoritative_batch [ "[]" ] in + let stale, _ = + Database.begin_authoritative database ~expected:(sync_view_token before) empty + |> T.require_ok ~behavior:"prepare empty token conflict" + in + ignore + (commit_plain_authoritative + database + ~cursor:(server_cursor 1) + empty_batch_update_wire + "advance competing commit"); + (match + Database.begin_authoritative database ~expected:(sync_view_token before) empty + with + | Error Authoritative_sync_token_conflict -> () + | _ -> Alcotest.fail "empty payload bypassed begin token guard"); + (match Database.apply_authoritative database stale ~decrypted:None with + | Error Authoritative_commit_token_conflict -> () + | _ -> Alcotest.fail "empty payload bypassed commit token guard"); + let after, _, updated = empty_batch_state database in + T.require + (Server_cursor.equal (sync_view_checkpoint after) (server_cursor 1) + && updated = 1_704_067_200_123L) + "stale empty preparation changed winning transaction" +;; + +let empty_authoritative_keeps_invalid_input_rejection database = + let before = + Database.inspect_sync database + |> T.require_ok ~behavior:"before invalid empty neighbours" + in + List.iter + (fun wire -> + match + Database.begin_authoritative + database + ~expected:(sync_view_token before) + (empty_authoritative_batch [ wire ]) + with + | Error (Authoritative_decode_failed _) -> () + | _ -> Alcotest.fail "empty support accepted malformed transaction") + [ "["; "null"; "{}"; "[42]" ]; + (match + authoritative_batch + ~maximum_count:16 + ~maximum_bytes:4096 + ~transactions:[] + ~through:(server_cursor 0) + ~checksum:None + with + | Error Authoritative_batch_empty -> () + | _ -> Alcotest.fail "empty outer batch accepted"); + let after = + Database.inspect_sync database + |> T.require_ok ~behavior:"after invalid empty neighbours" + in + T.require + (sync_token_equal (sync_view_token before) (sync_view_token after)) + "invalid input changed revision" +;; + let revision_codecs_reject_unknown_versions () = T.require (Result.is_error (Generation.of_string "generation:v0:1") @@ -4023,5 +4319,15 @@ let database_cases = let () = Alcotest.run "logseq_overlay_db sync" - [ "validated wire inputs", pure_cases; "durable sync view", database_cases ] + [ "empty authoritative compatibility", + [ T.database_case "empty array advances only sync state" (empty_authoritative_preserves_logical_state "[]") + ; T.database_case "empty Transit list advances only sync state" (empty_authoritative_preserves_logical_state + (Transit_native.Transit.Json.to_string (Transit_core.Json.List []))) + ; T.database_case "mixed batch preserves empty cursor slots" empty_authoritative_mixed_batch + ; Alcotest.test_case "empty cursor and revision survive reopen" `Quick empty_authoritative_survives_reopen + ; Alcotest.test_case "late decode failure rolls back mixed batch" `Quick (empty_authoritative_failure_is_atomic ~storage:false) + ; Alcotest.test_case "storage failure rolls back mixed batch" `Quick (empty_authoritative_failure_is_atomic ~storage:true) + ; T.database_case "empty payload keeps both token guards" empty_authoritative_keeps_token_guards + ; T.database_case "malformed neighbours and empty outer batch reject" empty_authoritative_keeps_invalid_input_rejection ] + ; "validated wire inputs", pure_cases; "durable sync view", database_cases ] ;;