Skip to content

feat(shadow): log the paths of mismatching fields in confirmed-mismatch warnings - #242

Merged
lan17 merged 234 commits into
mainfrom
claude/dialcache-shadow-reporting-1d5357
Oct 27, 2026
Merged

lan17 merged 234 commits into
mainfrom
claude/dialcache-shadow-reporting-1d5357

Conversation

@lan17

@lan17 lan17 commented Oct 24, 2026 •

Copy link
Copy Markdown
Owner

Problem

Shadow validation is DialCache's background correctness check. For a sampled share of keys, it
compares the value cached in Redis with a fresh result from the source, the function the cache
wraps, off the caller's path. When the two disagree it reads Redis again: if the cached payload is
still there, the mismatch is confirmed. Each outcome, such as match, mismatch or superseded,
is counted on the shadowValidation metric.

When the mismatch rate rises, the only way to learn what disagreed has been
shadow.logMismatches: true. It logs the cache key and the complete JSON of both values, each cut
at 8 KiB. That is all or nothing: to learn which field keeps disagreeing, a team has to put whole
records into its logs, and for a large record the cut often lands before the field that differs.
The questions a mismatch rate raises are structural: is it always the same field, is a field missing
on one side, did a list change length, or is the difference something JSON cannot show at all? None
of them need the values.

Pull request #138 built an earlier, TypeScript-only version in August 2026. It predates the Go,
Rust and Python ports, can no longer be rebased, and is superseded here; its reviewed decisions are
carried over. Closes #243, which tracks the ports and the cross-port evidence.

Change

A confirmed-mismatch warning can now say where the cached and source values differ without logging
what they hold. A new shadow.mismatchLogging group chooses what the warning carries:

  • diff: "paths" logs the JSON Pointers of the differing fields.
  • diff: "values" also logs what each side holds at those pointers.
  • value: true logs both whole values, as logMismatches: true did.

logMismatches stays as shorthand for everything on. A use case can replace the built-in path
computation with its own hook, and a new metric reports whether each warning was actually logged.
All of it is off by default and runs in the background shadow check after the verdict, so it never
changes the shadowValidation outcome, and no failure reaches the caller.

The design record design/2026-10-24-shadow-mismatch-paths.md holds the decisions (D0 to D15), the
alternatives rejected, and the shapes each port uses. D12 makes the Quint shadow profile predict
each confirmed-mismatch warning's fields, null fields, scalar paths and entries, age and
shadowMismatchLogging outcome, checked in every port's replay. Names below are TypeScript's. Change type:
Portable behavior (C60's wording and a new contract for the path computation) and Public API (every
port mirrors the config, the warning fields, the hook and the metric).

What a warning looks like

A GetUser use case on the users-api namespace has this user cached, and the source has changed
it since:

// Decoded from Redis
const cached = { id: "u_123", email: "ana@example.com", plan: "free",
  roles: ["viewer"], profile: { name: "Ana", locale: "en-US" } };
// Returned by the source
const source = { id: "u_123", email: "ana@example.com", plan: "pro",
  roles: ["viewer", "editor"],
  profile: { name: "Ana", locale: "en-GB", timezone: "Europe/London" } };

With shadow: { ramp: 5, mismatchLogging: { diff: "paths" } }, once the mismatch is confirmed,
DialCache makes this call to the configured logger:

logger.warn("DialCache shadow validation mismatch", {
  cacheNamespace: "users-api",
  useCase: "GetUser",
  keyType: "user_id",
  outcome: "mismatch",
  cachedValueAgeSeconds: 42.5, // varies
  cacheKey: "{users-api:user_id:u_123}#GetUser",
  mismatchPaths: ["/plan", "/profile/locale", "/profile/timezone", "/roles"],
  mismatchPathCount: 4,
});

/id, /email and /profile/name match, so they are absent. Paths follow member names in UTF-16
order, so /roles comes after /profile. Only the source has /profile/timezone, and /roles
changed length, so that array is reported once, whole, rather than element by element.
cachedValueAgeSeconds, new on every warning, is the cached value's age at the verdict. With
diff: "values", the warning also carries diffJson:

[{"path":"/plan","cached":"free","source":"pro"},{"path":"/profile/locale","cached":"en-US","source":"en-GB"},{"path":"/profile/timezone","source":"Europe/London"},{"path":"/roles","cached":["viewer"],"source":["viewer","editor"]}]

docs/shadow-validation.md walks through this example and a table of smaller cases. Native tests
pin both of the example's warnings and every row of the table.

Choosing what to log

shadow.mismatchLogging holds two independent settings, both off by default. Like other config
leaves, each can be overridden on its own by a runtime config:

Leaf Values Adds to the warning
diff "off", "paths", "values" "paths": where the values differ. "values": also what each side holds there
value true, false The JSON of both whole values

"paths" logs no field values, so it is the level meant to stay on in production for user data. It
still logs the cache key, and member names appear in the pointers, so an object keyed by
identifiers logs those identifiers. "values" and value: true log application data and suit an
investigation.

shadow.logMismatches stays permanently, because it is live in consumers' runtime config stores and
removing it would silently stop their logging. true stands for { value: true, diff: "values" }
and false for both off. A use case has a static default config, and a runtime config provider can
return an overlay. Within each of the two, the shorthand expands first and a leaf written beside it
overrides the expansion; then the overlay's leaves win over the default's. So a runtime
logMismatches: false with no leaves beside it turns everything off, whatever the default sets.

In a runtime config, an invalid leaf acts as off, and an invalid logMismatches expands to both
leaves off before the leaves written beside it apply. Either records one config_resolution error,
once for each shadow check that actually runs. In a static default config, an invalid leaf makes
cached() or getOrLoad() throw. A mismatchLogging that is not an object is rejected the way a
malformed ttlSec map is: the constructor throws, and a runtime config with one makes that call run
uncached. DialCacheKeyConfig.disabled() turns both leaves off.

How the paths are computed

The built-in computation never logs anything outside the JSON that value would log. Each value is
rendered to JSON once, and the value dumps and the built-in diff both come from that rendering, so a
member that a toJSON method leaves out of the dump stays out of the diff too. The two renderings
are parsed and walked together:

  • Object members are compared in UTF-16 code-unit order of their names, so every port emits paths in
    the same order. A member present on one side only is reported once, with only that side.
  • Arrays of equal length are compared by index. Arrays of different length are reported once, at the
    array, so one inserted element does not report every later index.
  • Any other differing pair is reported at its RFC 6901 JSON Pointer, and roots of different kinds at
    the root pointer "".
  • The walk descends at most 32 levels; a differing pair deeper than that is reported at the
    container 32 levels down.
  • A top-level undefined, which the JSON serializer stores for a negative-cache entry, is an
    absent value, not a rendering failure (D13). Against a present value it gives one entry at ""
    carrying only the present side; the absent side's value field is null and the outcome stays
    logged. A cycle, a bigint or a throwing toJSON still has no rendering and logs
    render_error.

The output is capped at 64 entries (mismatchPathCount keeps the full count), 512 bytes per
pointer, 1 KiB per value inside diffJson and 8 KiB for diffJson. The caps bound what is logged,
not the work: the full count needs a full parse and walk of both renderings, so the CPU and memory
cost grows with the values' size (about 86 ms per warning for a 5.6 MB value, against 13 ms for the value dumps
alone). The record lists this under Deviations, and whether to bound it is an open question below. An empty mismatchPaths means
both values have the same JSON, so the comparator, Node's util.isDeepStrictEqual unless the use
case sets shadowComparator, saw a difference JSON cannot show, such as a Date against its ISO
string.

Computing the differences yourself

For values the JSON walk explains poorly, such as lists that should be matched by ID, a use case can
compute the differences itself. shadowMismatchDiff(cached, source), an option beside
shadowComparator on cached() and getOrLoad(), returns the entries { path, cached?, source? };
a hook that knows only where values differ returns paths alone. It follows the comparator's rules:
synchronous, deterministic, side-effect-free, non-mutating and bounded. Its entries are logged as it
returns them, so the toJSON guarantee above covers only the built-in computation.

If the hook throws, returns a promise-like or a non-array, or returns a member that is not an object
with a string path, then mismatchPaths, mismatchPathCount and diffJson are logged as null.
DialCache never falls back to the built-in computation, so a broken hook cannot log more than its
author intended. A shadowMismatchDiff that is not a function makes cached() or getOrLoad()
throw a TypeError, as shouldAttemptStaleRecovery does.

Knowing whether a warning was logged

Logging can fail quietly, so a new optional metrics-adapter method, shadowMismatchLogging, reports
one outcome per warning attempt:

Outcome Meaning
logged Every selected field rendered and the logger returned
hook_error shadowMismatchDiff or the built-in walk threw or returned an invalid list
render_error A value had no JSON rendering, or building the fields failed
logger_error The application's logger threw, or the promise it returned rejected (D15); this used to go unnoticed

The first failing stage wins. When a hook_error or render_error leaves fields null, the
warning carries that outcome as loggingOutcome. The metric fires only while logging is on, so
count mismatches with shadowValidation, not with this metric. Datadog exports it as
dialcache.shadow_mismatch_logging.count and Prometheus as
dialcache_shadow_mismatch_logging_counter.

Compatibility

  • Existing configs. No migration. logMismatches: true keeps logging every confirmed mismatch,
    and its warnings gain cachedValueAgeSeconds, mismatchPaths, mismatchPathCount and
    diffJson. The diff reads each whole rendering while the value dumps stop at 8 KiB, so for a
    larger value it also logs differing fields past that cut. That is deliberate (D10): truncation was
    never redaction, and a team that set logMismatches: true has already accepted these values in
    its logs. Each confirmed mismatch also costs more CPU and memory in proportion to the values'
    size, and a warning can carry up to about 40 KiB more. To keep the previous fields plus
    cachedValueAgeSeconds, write mismatchLogging: { diff: "off" } beside the shorthand. To log
    where values differ without logging them, replace logMismatches with
    mismatchLogging: { diff: "paths" }: a leaf beside logMismatches: true leaves the other
    expanded leaf on, and a service still on an earlier DialCache version reads only the shorthand.
  • TypeScript API. Additive: new optional config fields, the shadowMismatchDiff option, the
    shadowMismatchLogging adapter method, and the exported types ShadowMismatchDiffLevel,
    ShadowMismatchLoggingConfig, ShadowMismatchDiff, ShadowMismatchEntry,
    ShadowMismatchLoggingMetricLabels and ShadowMismatchLoggingOutcome. Apart from the new
    leaves, config is read as on main: unknown fields stay ignored, and an inherited shadow group
    still applies (D14).
    DialCacheKeyConfig.disabled().shadow now carries mismatchLogging: { value: false, diff: "off" }
    in place of logMismatches: false.
  • Datadog. shadow_mismatch_logging.count is now the longest metric suffix, so the adapter
    constructor throws a RangeError for a custom namespace longer than 170 characters (175 before),
    keeping every name within Datadog's 200-character limit.
  • Go and Rust. At the maintainer's choice the ports replace their logging config and details
    types and delete what becomes unused, so Go and Rust callers can see compile errors. Go adds
    ShadowPolicy.MismatchLogging, replaces ResolvedShadow.LogMismatches with LogValue and
    LogDiff, and removes ShadowMismatchLogDetails, ShadowMismatchDetails and
    PreviewShadowLogJSON. Rust's ShadowPolicy gains mismatch_logging, ResolvedShadow and
    ShadowMismatchDetails change shape, Operation holds a private renderer, and exhaustive matches
    over Event and MetricKind gain a variant. docs/upgrading.md names each change. At the
    maintainer's request the title carries no !.
  • Rust previews. A custom preview replaces the JSON for the whole warning, so a redacting
    preview also bounds the difference fields: they come only from a mismatch_diff hook, and are
    render_error without one. An operation whose preview is explicitly None keeps main's
    meaning and logs no values, at the maintainer's choice. Operations built with with_codec have
    differences only through the hook (Rust: let with_codec operations supply a JSON renderer for mismatch paths #245 tracks a JSON renderer for them).
  • Python. cachedValueJson and sourceValueJson now escape lone surrogates as JsonSerializer
    does, and a static default with an invalid new leaf raises ConfigError, as invalid static
    log_mismatches already did. A logger whose warning() returns an awaitable is awaited for the
    outcome (D15).

Formal evidence

DialCache keeps its ports in step with a shared formal layer: numbered contracts in
formal/guides/CONTRACTS.md, Quint models that generate call histories every port must replay
identically, and a catalog of mutants (deliberate bugs) that each port's tests must catch.

  • Contracts. C60 now says logging is on when value is true or diff is above "off" once
    logMismatches expands, and states the outcome rule: one shadowMismatchLogging event per warning
    attempt, none with logging off, ranked hook_error > render_error > logger_error. A new
    contract, W10, states the path computation as a portable function, and the hand-written
    mismatchPathVectors group (57 vectors: the documented rows, D13's absent sides, own members named
    like Object.prototype members, the depth cap, the array rule, the root cases, every cap, the
    uncut count and the number rule) is replayed by every port. SPEC.md says the same.
  • Quint (D12). The shadow profile, now version 4, captures the value and diff leaves at
    admission and predicts each warning's fields, null fields, scalar paths and entries, age and
    outcome. It has starting and runtime choices for every level, the shorthand rows and written off
    leaves, three failure inputs (a failing hook, a failing logger, a source value with no JSON
    rendering), six new invariants and 21 new exported regressions. Admission, dark-layers,
    shadow-layers and shadow-read-deadlines keep their versions and agree with main on every history.
  • Mutants and challenges. M08, M09 and M65 move to the leaf code in every port. M66 to M76 plant
    the array rule, a root kind change, an absent side as render_error, a hook failure falling back to
    the walk, the hook at diff: "off", an ignored written off leaf, both precedence swaps, a warning
    with logging off, entries logged at "paths" and an invalid diff leaf read as values, in
    TypeScript and Go, with Rust twins M14 to M25. Eleven C60 model-fault challenges map to them. The
    Go and Rust mutation matrices grow to 15 and 4 shards so the larger catalogs fit their timeouts.
    Python has no mutation catalog, so its evidence for these rules is the shared replay, the vectors
    and its native tests. The walk's own-member rule has no mutant, because Go maps have no inherited
    members; the vectors pin it in every port.

Where the code lives

  • typescript/src/internal/shadow-log-json.ts: rendering, the built-in differ, its caps and the hook
    runner.
  • typescript/src/internal/runtime-config.ts: the shorthand expansion and the leaf-wise merge.
  • typescript/src/config.ts: the config types and disabled().
  • typescript/src/dialcache.ts: the shadowMismatchDiff option and the warning.
  • typescript/src/metrics.ts, datadog.ts, prometheus.ts: the outcome metric.
  • typescript/src/internal/thenable.ts: one helper that consumes a promise-like returned by a
    logger, metrics adapter, mismatch hook, comparator or stale recovery predicate, so a rejection is
    never left unhandled. It replaces main's settleUnexpectedThenable at the comparator and
    predicate sites; the comparator now waits one more microtask before its deadline check when it
    returns a value that is neither a boolean nor a promise, which no caller can observe.
  • Go: go/policy.go (leaves and merge), go/shadow_log.go (rendering, walk, hook), go/shadow.go
    (the warning), go/metrics*.go (the outcome metric).
  • Rust: rust/src/policy.rs, rust/src/protocol/mismatch.rs (walk), rust/src/shadow_log.rs
    (warning content on the bounded CPU executor), rust/src/operation.rs (hook and renderer).
  • Python: python/dialcache/config.py, python/dialcache/_shadow_log.py, python/dialcache/cache.py.
  • Formal: formal/models/kernel/shadow.qnt, formal/models/dialcache-shadow-conformance.qnt,
    formal/replay/features.mjs, formal/catalogs/protocol-vectors.json, and every port's replay
    driver.

Ports

Port Status
TypeScript Done
Go Done
Rust Done
Python Done

The temporary bridges the TypeScript-first stage needed are gone: every driver records the whole
warning and the shadowMismatchLogging event, the corpus exercises the new leaves, the vectors and
W10 are in, and the Go parity ledger maps the feature to its Go bindings. Each port documents its
native differences (B03, X01) in docs/shadow-validation.md and the record's Deviations: Go's and
Rust's JSON parsers fail very deep nesting as hook_error, diffJson member order follows each
port's maps, and the walk's cost grows with value size and member count in every port.

Validation

Head: 934d5b8.

  • TypeScript review: multi-lane review rounds (correctness, tests, simplicity,
    architecture, contracts, security, reliability, performance, then two holistic passes each), with
    a plan critic before each set of fixes.

    • Round 1 (84dcfbd) led to 6cd9fe3 to c289ed3: one clamp rule for built-in and hook
      entries, a registration TypeError for a non-function hook, three simplifications, and the docs
      and record notes above.
    • Round 2 (c289ed3) led to D13 (0e6fb37, 32d59d9, 3239e49).
    • Round 3 (6684ac7) led to a919d50: the per-call shadow merge carries the raw groups and
      expands the shorthand once at admission, checked against expanding each layer first across
      8,064 combinations.
    • The maintainer then kept main's config reads (D14): unknown-field reporting dropped (ed80bbe,
      646d5d9) and shadow read as ordinary properties again (377526c, 964ade2). With it gone,
      a local-hit getOrLoad and a cached() local hit with a provider and a shadow group measure
      within noise of main, and a disabled getOrLoad with an inline shadow config about 8% over.
    • A second review run on 45cc3d2 led to 658d6d2: the merge expands each layer's shorthand first
      again, as the record states the rule, at main's per-call cost; its second round led to
      70b5e58, an independent oracle for the merge test. The maintainer then made an asynchronous
      logger failure a logger_error (D15, efa6b33, aad2ae7).
    • Its third and fourth rounds (e508c68, d79e4a7) pinned the walk to own members (f6a8aba,
      e1537c1) and D15's timing: a hook or render failure is recorded at once even while the logger's
      promise is pending, and a promise that never settles records nothing (48b3dc0). A returned
      promise is now observed through await (2cbca04), so a global Promise replaced after load
      cannot miscount a delivered log or leak a rejection.
    • Its fifth round (a9c47d9) led to d8a6412: one internal helper, isPromiseLike and
      fulfills, now consumes a promise returned by a logger, metrics adapter, hook, comparator or
      stale recovery predicate, replacing four per-site copies; and 7cdef0f, which states the walk's
      memory and sorting cost.
    • Its sixth round (842e5f9) was clean in every lane and in both holistic passes, with no fix
      applied: the TypeScript reference's clean confirmation round.
    • Four non-material items it raised were applied with the port integration and reviewed there:
      the thenable.ts comment and a test name were narrowed (a native promise whose own
      constructor is also replaced is adopted through its own then, as on main), the go-parity
      rationale names fulfills, a test pins that the hook receives the raw undefined for an absent
      side, and the upgrade note's 40 KiB figure says field bytes.
    • Every fix-shaped test above was checked against a planted bug that it must catch.
  • Ports and Plan step 4: each piece was built on its own branch, reviewed by fresh reviewers until a
    round came back clean, and merged here.

    • Go, Rust and Python libraries: a parity reviewer (probes running the vectors and documented rows
      through each port against TypeScript) and a quality reviewer per port. Round 1 found the same
      defect in all three: the walk copied the whole pointer prefix for every member, quadratic where
      TypeScript's rope strings are linear; each now builds pointers only for kept entries, pinned by an
      allocation test. Rust's review also found that a redacting preview no longer bounded the
      difference fields; it now fails closed. Each port planted 20 to 29 bugs against its new tests.
    • The shared formal layer and TypeScript driver (clean on its first round), the three port drivers
      and vector runners, the formal additions (written off leaves) and the shared docs (three rounds,
      each fixing one accuracy slip), then the mutants and challenges (two rounds: the larger catalog
      needed more Go and Rust mutation shards, and the Rust driver had dropped diffJson at
      "paths", which hid a mutant).
  • Whole-PR review round 1 (0532f83): every port with all eight lanes, the formal layer with six, the docs
    and record with two, and the TypeScript delta with three, then a holistic pass. It accepted 21 findings, fixed
    in a424f87 to 80d41dd and each verified clean:

    • check-models already ran 143–148 of its 150 minutes on main, and the 11 new challenges push it over (Full
      run 40245022083 confirmed it: every other lane passed and check-models was cancelled at its limit). Its
      timeout is now 210 minutes, and formal-validation.test.ts pins its capacity, as it does the mutation lanes.
    • Go's default logger escapes control characters, so a data-derived member name cannot forge log lines.
      Kept pointers are clamped while they are built in Go and TypeScript, which ends a 64× memory amplification
      from one huge member name. Go builds warning content while the shadow job still holds its capacity.
    • Python drives a logger's awaitable for every warning through one boundary and compares numbers as doubles,
      the documented rule, which five new vectors now pin in every port. Rust keeps one value renderer, and the
      stale vector copies are gone.
    • Nine witness label families gained negative controls, the hook-over-render witness needs a rendered value,
      a redundant invariant was deleted, and the docs say when an outcome waits for an async logger.
    • Rejected with reasons: an error-message sentinel for Rust's test-only absent marker, and exact 64-bit number
      comparison, which would need lossless JSON parsing in TypeScript (see Follow-ups).
  • Whole-PR review round 2 (80d41dd), the same lanes again: 24 of 35 clean; 11 findings, none high, fixed in
    f16f75f to 85febcc and each verified clean, plus round 2b from its holistic challenge pass:

    • Go's default logger writes the warning details as JSON, so a root difference ([""]) no longer prints like no
      difference ([]), and hook sides honor a redacting MarshalJSON at every depth (the codec's stored encoding
      is unchanged).
    • Python checks a logger's return value inside its isolation boundary, shares one native JSON rendering with
      storage, and compares integers beyond the double range as doubles too (the maintainer confirmed the double
      rule as W10's in session, 2026-10-25: "Confirm doubles").
    • A vector pins the pointer cap below a parent of exactly 512 bytes (56 vectors); the Rust cost test now fails
      when every reported difference builds a pointer; check-rust lints the published Rust build without
      test-util.
    • The mismatch-logging decoder moved to its own Quint kernel module, imported only by the shadow profile, so
      its five challenges no longer run against four profiles that cannot reach it.
    • The record's Deviations now carry the number rule, Go's slot hold, the deleted invariant, each port's
      parse-failure rule and the Rust executor coupling. The walk's cost stays as documented, by the maintainer's
      decision on 2026-10-25, with a faster implementation tracked in Shadow mismatch paths: make the built-in walk faster, TypeScript first #248.
  • A final review of the delta since 80d41dd (85febcc), with lanes sized to each area and only medium-or-higher
    findings blocking: Rust, Python, formal and docs clean. Three Go lanes and the holistic pass found one medium
    issue: a hook side marshalled a struct from a copy, so a field's pointer-receiver MarshalJSON did not run and
    a redacted value reached diffJson. a8fd864 marshals an addressable struct through its pointer, as
    encoding/json does. An independent check matched encoding/json on 38 value shapes, found the stored
    encoding byte-identical, and confirmed the new test fails on the old code. make check, make formal-go,
    make smoke and make integration-go passed at ef9618f.

  • Local validation at 85febcc, each a separate command, all passed: make formal-generate (fresh corpus),
    make fixtures-check, make check (TypeScript 3,944 tests; Go, Rust and Python native suites; docs;
    make audit), make smoke, make model-check, make formal-ts (7,920 tests), make formal-go,
    make formal-rust and make formal-python (7,837 required cases each), and make integration (every port
    against Redis, Valkey and Cluster, and the wire vectors).

  • TypeScript review on this pull request (posted 2026-10-26 at 80d41dd, folded in at the maintainer's request): the
    defensive code is cut to the real boundaries, config parsing and the diff generator. a74b2a4 merges it:

    • the unreachable non-object branch in expandMismatchLogging and the catch around the built-in walk are gone;
    • the one catch around warning construction now logs every requested field as null with render_error
      instead of dropping them (a Deviation records it);
    • the walk builds a pointer only for an entry the cut keeps, with identical output;
    • the tests that pinned engine promise-adoption rules, and the duplicate hook-failure cases, are removed.

    typescript/src shrank by 10 lines and typescript/test by 106. Behavior a caller, logger or metrics adapter can
    see is unchanged: all 57 vectors and every warning-shape test pass. A verifier tried to make each deleted guard
    matter and could not. It did catch one regression in the first cut (a deep pointer rebuilt from the whole
    stack, 14–100× slower), fixed in 8f7a264.

  • Validation for that TypeScript-only delta, by the maintainer's decision, is the TypeScript lanes: at d756793,
    make check-ts, make formal-ts, make docs and make audit; at dcb7784 (the same TypeScript source),
    make mutations-ts MUTATION_ONLY=M66,M67,M68,M69,M70,M72,M75, with every required detection held. No Go, Rust,
    Python, model or catalog source changed after ef9618f.

  • Custom differ entries are no longer validated (99880fe), at the maintainer's decision: "if a custom differ
    returns non correct results its fine". A hook that throws, returns a promise-like or returns a non-array still
    gives hook_error and never falls back to the walk; the first 64 entries are logged as returned, with paths in
    string form. The record carries it as a Deviation of D6. Validated with make check-ts, make formal-ts,
    make check-python, make formal-python, make docs and make audit at 4515217.

  • The same defensive-code trim in Go, Rust and Python (d2f5960), under the maintainer's rule to remove defensive
    code that no real boundary needs: Go drops re-validation of the already-validated static policy and an encoder
    fallback that could not fire; Rust reads a mismatch verdict's values as given and drops dead branches; Python
    drops the catch around the built-in walk and duplicate checks. 96 lines added, 199 removed, no observable
    change for any supported input. Each port's native checks (make check-go, make check-rust,
    make check-python, ruff) passed and every mutant anchor still matches; PR CI covers the replay and the rest.

  • Full run 40274292379 at ef9618f: passed, every lane.

  • Full run 40259905276 at 80d41dd: passed, every lane, check-models within its new 210-minute limit.

  • Full run 40245022083 at 0532f83: every parity and mutation lane passed; check-models timed out at its old
    150-minute limit.

Full run 40272657294 at 85febcc was cancelled for the Go fix; nothing is running.

Follow-ups

  • A faster built-in walk (Shadow mismatch paths: make the built-in walk faster, TypeScript first #248): the measured no-dependency speed-up, TypeScript first.

  • Faster validation for portable-behavior changes (Formal: cut wall time for portable-behavior changes without weakening the evidence #247): shard the model-fault campaign, scope mutants to the
    cohorts that can catch them, cache Full lanes by input closure.

  • Go hook sides and encoding.TextMarshaler. Outside a struct, a hook side honors json.Marshaler but not
    encoding.TextMarshaler, which encoding/json also calls; the docs promise only json.Marshaler.

  • A comparator that returns a promise that never settles (pre-existing on main, unchanged
    here). The shadow operation awaits a comparator's accidental promise-like so the flight is retained
    until it settles, by design. If it never settles, the flight is never released. With the default
    shadowMaxInFlight of 1, every later shadow job on that instance is then dropped, for every key.
    The hook and the stale recovery predicate consume such a value without awaiting it. Whether the
    comparator should do the same, or release the flight at the deadline, is a separate decision.

  • A JSON renderer for Rust with_codec operations (Rust: let with_codec operations supply a JSON renderer for mismatch paths #245): today they get difference fields only
    through a mismatch_diff hook.

  • Go's member sort allocates on every comparison (utf16Less), which dominates rendering a very
    wide map; the cost is linear-logarithmic and documented, so it is a performance follow-up.

  • Exact comparison of 64-bit integers in the walk. Numbers compare as doubles in every port, so two ids
    that differ beyond a double's precision log no path. Exact comparison would need lossless JSON parsing in
    TypeScript (for example JSON.parse with source access).

  • A Python mutation catalog, so the Python port's tests are measured like the others.

  • Strict runtime config validation (D14): a runtime config that fails validation is rejected
    as a whole, records an error metric and runs the call uncached, as one invariant in every port.
    It gets its own design record and pull request.

lan17 added 2 commits October 24, 2026 13:04
…rnings

Add the design record for a mismatchLogging content group with paths, value
and diff leaves, RFC 6901 pointers for the differing fields, a built-in JSON
differ with hand-authored protocol vectors, and a per-use-case paths hook.
Every decision is Open; the decisions reviewed on #138 are carried.
@codecov-commenter

codecov-commenter commented Oct 24, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.13200% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.67%. Comparing base (b4e4fba) to head (934d5b8).

Files with missing lines Patch % Lines
rust/src/shadow_log.rs 88.04% 22 Missing ⚠️
rust/src/policy.rs 98.85% 5 Missing ⚠️
go/shadow.go 97.36% 1 Missing ⚠️
rust/src/observe.rs 95.23% 1 Missing ⚠️
rust/src/protocol/mismatch.rs 99.42% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #242      +/-   ##
==========================================
+ Coverage   93.73%   94.67%   +0.94%     
==========================================
  Files          78       84       +6     
  Lines       11357    13026    +1669     
  Branches      656      805     +149     
==========================================
+ Hits        10645    12332    +1687     
+ Misses        642      628      -14     
+ Partials       70       66       -4     
Flag Coverage Δ
go 93.25% <99.70%> (+2.13%) ⬆️
python 93.60% <100.00%> (+1.52%) ⬆️
rust 94.78% <96.89%> (+0.48%) ⬆️
typescript 97.21% <ø> (+0.65%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

lan17 added 4 commits October 24, 2026 13:33
…ts one at a time

Add D10: logMismatches stays permanently and expands to every mismatchLogging
leaf before the leaf-wise merge, with explicit leaves winning. The shared
corpus therefore stays valid, so TypeScript lands first in #242, each port
follows in its own pull request, and the vector group and contract row land
with the last port.
…paths proposal

shadowMismatchPaths is the only hook. paths is the value-free tier and diff
narrows values to the differing leaves, so the projector moves to Out of
scope with the evidence that would reopen it.
…e list

shadowMismatchDiff(cached, source) returns { path, cached?, source? }
entries; mismatchPaths and diffJson are two projections of that list, so no
pointer is ever resolved against a rendering. The built-in walk attaches both
values as it emits each entry. Dropping diff and a textual diff are recorded
as rejected and out of scope.
@lan17 lan17 changed the title feat(shadow)!: log the paths of mismatching fields in confirmed-mismatch warnings feat(shadow): log the paths of mismatching fields in confirmed-mismatch warnings Oct 24, 2026
lan17 added 6 commits October 24, 2026 14:29
mismatchLogging is now { value: boolean, diff: "off" | "paths" | "values" }.
The two diff levels are projections of one entry list and read as the
exposure ladder they are; logMismatches: true expands to the maximal setting.
The three-boolean shape moves to Rejected with the reason.
…failure handling

D11: a dedicated metric with outcome logged, hook_error, render_error or
logger_error, one event per warning attempt and nothing when logging is off;
the warning carries loggingOutcome when it is not logged. The Proposal now
states that warning work is detached and never reaches the caller, that the
verdict is decided before it starts, and that a failure logs less, never more.
The shorthand sets a baseline for the group and leaves beside it are the
exceptions, in both directions; a table gives the same-object and cross-layer
cases, and a dominant false is recorded as rejected under D10.
D3 and D5 speak of entries, D6 of mismatchPaths, D7 is titled by diff level,
the projector and values-level alternatives use the current leaf names, and
the TypeScript step of the Plan lists the shadowMismatchLogging metric.
The maintainer confirmed D0 through D11 in session on 2026-10-24; every
decision records that, and the status moves from Proposed to Accepted.
lan17 added 15 commits October 24, 2026 16:15
…smatch warnings

Add the shadow.mismatchLogging group with the value and diff leaves, keep
logMismatches as the shorthand that expands to them before the leaf-wise
merge, and validate each leaf independently. Report unknown key-config
fields as config_unknown_field.

Add the built-in JSON differ (RFC 6901 pointers, UTF-16 member order, the
array-length rule, root and depth handling, and the entry, pointer, side and
diffJson caps), the shadowMismatchDiff hook with fail-closed handling, the
new warning fields, and the shadowMismatchLogging outcome metric with its
Datadog and Prometheus exporters.

The replay driver records only the mismatchWarning fields every port emits
(#243). Re-anchor mutants M08 and M09 on the new resolution code and add
M65 for a true shorthand that leaves logging off.
…come metric

Reword C60 and the SPEC mismatch-logging rows for the value and diff
leaves, and describe the leaves, the shorthand expansion, the warning
fields, the shadowMismatchDiff hook, the shadowMismatchLogging metric and
config_unknown_field in the shadow validation, observability,
configuration, API, upgrading and TypeScript pages. Reference #243 from the
design record.
Re-pin the reviewed docs, guides and tests in the source audit with
contract dispositions for the new tests, the feature-coverage test hashes,
and the Go parity source rows, whose mapping reviews now name the
TypeScript-only parts tracked by #243. Update the audit counts lesson 13
quotes.
The diff derives from the same renderings as the value dumps, so it logs
nothing outside them, but the walk reads each whole rendering while the
dumps stop at 8 KiB, so for a larger value it shows differing fields past
that cut. Reword the Proposal, the hook paragraph and D10's rejected
{ value: true } expansion accordingly; the maintainer kept D10.
…quest

The maintainer chose to implement Go, Rust and Python in #242 once the
TypeScript reference is settled, with the vector group, the contract row,
the corpus re-encoding and the differ and hook mutants in every port, and
to ship without a breaking-change marker. Record both in the Plan and the
Release impact.
…them

Walk through a GetUser record that differs at four fields, with the logger
call at diff "paths" and the diffJson at diff "values", ahead of the
computation rules, and add a table of smaller payload pairs with the paths
each one produces. Pin every example: the pairs as path vectors, the Date
row with its own values, and both GetUser warnings end to end, field order
included.
…fig fields

A disabled call is pass-through and observes no config, so only an enabled
getOrLoad call reports config_unknown_field for its default config. Make the
fetchKeyConfig unknown-field callback optional so mutant M29's edit, which
calls it with two arguments, compiles again.
…ow profile

Add D12, Open: the shadow profile predicts each confirmed-mismatch
warning's fields, null fields, scalar paths and entries, age and
shadowMismatchLogging outcome, which every port's replay checks, with
three failure inputs and a group of shared regression runs beside the
path vectors. Record the companion changes it implies in the contract,
Quint, observation and native bullets, Plan steps 1 and 4 and Impact,
state the outcome precedence as a fixed order, and return the record to
Proposed; only Plan step 4 waits for D12.
Static defaults and runtime resolution validated the diff level with two
unlinked lists, neither checked against ShadowMismatchDiffLevel. Both now use
one predicate over a record that fails to compile when the union grows, and
membership uses Object.hasOwn so Object.prototype member names stay invalid.
Every DialCacheKeyConfig construction scanned its input for unknown fields,
including the default snapshot built on each getOrLoad() and the merged
config built on each provider call, and a clean constructed config was
scanned again at every read. Constructed configs now keep a definite answer,
configs DialCache assembles from known fields skip the scan, and objects
construction never saw are still scanned.

A provider result that is the key's own default snapshot is treated like
null: the snapshot was observed at registration or when the call began, and
merging it with itself resolves the same policy. Reporting no longer depends
on whether the unknown field sits in a layer map or elsewhere.
Mismatch details were set only when logging was on, yet the verdict step
re-derived that from a separately threaded plan with a default constant and a
helper. Admission now resolves the plan to null when both leaves are off, the
details carry it, and the warning runs exactly when details exist.
The mismatch warning reached the raw logger through a second wrapper only to
learn whether warn threw. The shared logger wrapper now reports that for
every method, and the observer helper returns it directly.
…tured

A null or non-function hook took the hook path and turned every confirmed
mismatch warning into hook_error. cached() and getOrLoad() now throw a
TypeError for it, as they do for shouldAttemptStaleRecovery, before the use
case registers.
lan17 added 7 commits October 25, 2026 23:26
…n DialCache

fulfills is try { await value } catch, and its doc comment now says only
that it never rejects. The removed comment and tests covered how await
adopts a native promise whose own then was replaced and a global Promise
swapped mid-microtask, which no supported environment does: the
replaced-then logger row and hook, the two foreign-global-Promise logger
cases with their ForeignPromise class, and the bare-thenable iteration
of the hook rejection test, which could never report an unhandled
rejection. The rejecting-promise and rejecting-thenable logger rows and
the rejecting-promise hook keep the consumption covered.
The fifteen cases exercised five branches of runShadowMismatchHook. One
case each now covers the catch, the promise-like, the non-array result,
the non-object member and the non-string path, beside the invalid member
past the 64-entry cut that the contract still rules on and the throwing
side getter that shows the try covers member reads.
Re-key the three changed test files in the source audit, filling the
retitled pointer-bound and construction-net tests and the new
failed-fields test, and dropping the removed walk-throw and
foreign-Promise entries. Re-pin the shadow-log-json test in feature
coverage, the five changed typescript/src files in Go parity with one
more declaration, and the source-audit test count the course quotes.
Joining every reference token for each kept entry made a deep value cost
up to 32 escaped 512-code-unit names per pointer, 64 times per warning:
about 13 ms on a 31-level chain of 512-character "~" names, against
0.13 ms before the token stack. keptPointer now stops appending once the
pointer passes SHADOW_MISMATCH_PATH_MAX_BYTES code units, as childPointer
did, so each kept pointer again materializes at most three times the cap
and that chain takes 0.5 ms. The clamped pointers are unchanged, and the
bound test returns to 3 * 512 + 1, which the unbounded join fails at 1543.
Re-key the shadow-log-json test in the source audit with the restored
pointer-bound title and its W10 and X01 dispositions, re-pin that test in
feature coverage, and re-pin shadow-log-json.ts in Go parity; the
declaration count is unchanged.
…tate the warning-construction fallback per port
@lan17
lan17 marked this pull request as ready for review October 26, 2026 07:54
lan17 added 20 commits October 26, 2026 13:15
At the maintainer's decision, DialCache no longer validates the entries a shadowMismatchDiff hook returns. A hook that throws, returns a promise-like or returns a non-array still fails with hook_error and never falls back to the built-in walk. Only the first 64 entries are read, inside the hook's catch, and each path is logged in string form, so a hook that returns wrong entries logs wrong paths. Go's and Rust's types already require a string path. The record carries it as a Deviation of D6.
The static shadow group is written from typed fields: its shorthand is a
bool and its mismatchLogging is always an object, so expanding it cannot
fail and never marks the group invalid. Drop the unreachable error return
and the inherited invalid flag; the runtime group keeps both.
Only the library calls the default logger, and the one map it logs, the
mismatch warning, holds strings, finite numbers, string lists and nulls,
which encoding/json always encodes. Drop the fmt.Sprint fallback for a map
that fails to encode and the test that fed it NaN.
The library logs mismatchPaths only as a []string or a nil, and the driver
already records nil as null, so encoding the list cannot fail.
Since Go 1.21 panic(nil) is recovered as *runtime.PanicNilError, so the row
tested the runtime rather than DialCache; the panics row covers the same
branch.
…given

The only Mismatch verdict carries both, so the warning no longer keeps a
metadata-only fallback for one without them.
The walk is our own code and does not panic: a rendering too deep for the
parser is an error, which still fails the walk with hook_error. The CPU
job's own net stays.
Bounded entries hold a string pointer and JSON value sides, which always
serialize, so the warning no longer tracks a render_error for it.
…ration's type

The comparator accepted both values as T before any Mismatch verdict, so
the erased hook returns its entries directly and the JSON renderer keeps
no fallback for another type. A failed downcast would still land in each
one's panic net. Drops the test that pinned only the impossible case.
…the merge

The merge now checks the runtime group with the same policy_object call and
message static parsing uses, so expanding the leaves cannot fail and
resolution keeps no fallback for a shape the merge already rejected.
The walk only parses renderings that already succeeded and compares them
iteratively under the 32-token depth cap, so it cannot raise for a value
that rendered. The warning's outer net still catches anything unforeseen.
Drop the try, the two tests that pinned it by patching _walk, and the
docstring's mention of the walk as a hook_error source.
Each bounded entry holds a str path and sides that are either parsed from a
rendering of at most 1024 bytes or clamped text, so the list always renders.
Render it directly instead of checking for a None rendering.
Every shadow group reaching expand_mismatch_logging belongs to a constructed
Policy, whose construction already raises ConfigError for a group that is
not a mapping. Drop the second check, the merge's guard that only that
branch needed, and the test rows that fed raw non-mapping groups.
…ey outside the net

A mismatch outcome within budget always follows capturing the compared pair
when a plan exists, so the membership check never fails. Clamping our own
logical key cannot raise, so it moves into the warning literal and the net
covers only rendering application values; its comment now names the case
it exists for. The fallback test loses its clamp_utf8 row.
Six rows reached the hook's catch or its list check the same way another
row already does. Keep raises, raises-cancelled, coroutine, tuple (the only
row the list check decides), member-without-path and failing-side.
@lan17
lan17 merged commit e33f27c into main Oct 27, 2026
22 checks passed
@lan17
lan17 deleted the claude/dialcache-shadow-reporting-1d5357 branch October 27, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shadow mismatch paths: port to Go, Rust and Python and land the shared observations, vectors and mutants

2 participants