Skip to content

feat(events): persist events to disk so they survive the process (tier 3) - #538

Draft
abelonogov-ld wants to merge 105 commits into
v11from
andrey/event-durability-tier3-persistence
Draft

abelonogov-ld wants to merge 105 commits into
v11from
andrey/event-durability-tier3-persistence

Conversation

@abelonogov-ld

@abelonogov-ld abelonogov-ld commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Third in the event-durability stack, on top of #536. Recorded events go to an on-disk log rather than an in-memory buffer, so an application that dies moments after recording one still reports it: the next run delivers what the last one left behind.

What changes

EventStore is a new on-disk log, with capacity, framing, interrupted-log recovery, and multi-process safety. NullEventStore stands in where the platform gave the SDK nowhere to write, which is the behaviour the SDK had before.

Recording no longer encodes on the caller's thread. An evaluation appends to a held run and updates a counter; a commit turns the whole run into bytes at once. Recording a custom or identify event is a commit point, because an application reporting something it chose to report is saying it matters more than the microseconds it costs. Evaluations commit once pendingCommitThreshold of them accumulate, when a delivery starts, or on flush.

LDConfig.eventPersistence lets the application choose whether a commit point runs on the caller's thread (.immediate), off it (.deferred), or not at all (.disabled).

Delivery sends batches oldest first and stops at the first retryable failure, so the rest keep their place in the log instead of the SDK spending the session's requests on a service that is refusing them.

Reconciling with tier 2

Tier 2 replaced the delivery gate this branch was built on, so the merge had to pick one design rather than keep both.

Coalescing comes from tier 2: flushes arriving before a delivery starts join it, one already started is never joined, and nothing holds the drain back while a retry runs. What tier 2 did not need is a way to keep an in-flight batch from being sent twice — it empties its buffer before publishing, so a second delivery finds nothing. Here a batch stays on disk until its response arrives, which is what isDelivering guarded.

batchesInFlight takes its place. A delivery claims the batches it lists and releases them once it has an answer, so a later delivery still runs and still sends what it closed for itself; only the batches already on their way are left out. A caller whose own events are all that remain unsent is answered true, which is exactly the contract tier 2 documents: an answer is about the batch the caller's own delivery took, never one an earlier delivery lost.

The loss flag is now taken once per delivery rather than once per caller, so everyone joined to it hears about a loss instead of only whoever was answered first, and a scheduled delivery with no waiters leaves it for the caller who is owed it. This branch had also fixed the duplicate event reporter independently; that now comes from #537, so the local copy is gone.

Verification

  • 824 tests pass on iOS; macOS and tvOS build. No new SwiftLint violations.
  • The in-flight claim is load-bearing: removing the filter fails 5 tests, including does not send a batch again when a delivery is asked for while one is in flight.
  • DiagnosticReporterSpec.testRetries flaked once under load and passed on re-run; it has a 0.1 s timeout and is unrelated to these changes.

Note

Overview
Adds Tier 3 event durability: analytics events can be appended to an on-disk log (EventStore with framed EventLogFormat, recovery of interrupted logs, and macOS file locking) so they survive process death and are delivered on the next launch. LDConfig.eventPersistence (.disabled / .deferred / .immediate) controls whether commits run off-thread, on the caller’s thread at track/identify, or not at all; default eventCapacity rises from 100 to 1000.

Flush behavior moves to LDClientFlush and grows: fire-and-forget flush(), flush(completion:) and flushAndWait(timeout:) (plus Obj-C equivalents) aggregate all environment clients and report delivery success via flushReportingOutcome. On iOS/tvOS background, the client commitAtCommitPoint() then flushes under a BackgroundActivity assertion so events are durable and a best-effort send runs before suspension.

Encoding path changes: EventJSONWriter, JSONWriter, ContextEncodingCache, and LDContextJSONWriter replace context copying for anonymous redaction—redaction is driven per event kind via Event.redactsAnonymousAttributes. ContextSummarizer keys contexts by full equality (with caps when too many distinct contexts) instead of contextHash() alone.

Supporting tweaks: DiagnosticCache uses UnfairLock instead of a serial queue; DarklyService.publishEventData invokes completion when the mobile key is missing so reporters do not stall.

Reviewed by Cursor Bugbot for commit 81a7856. Bugbot is set up for automated code reviews on this repo. Configure here.

abelonogov-ld and others added 30 commits September 18, 2026 17:31
…igest

ContextSummarizer keyed its per-context trackers by LDContext.contextHash(),
which encodes the context to JSON and takes a SHA-256 of the result. That is the
right price for the flag cache, which persists the digest and compares it across
launches, but a bucketing key does not need to cost it — and an application that
reads a flag on every redraw paid it on every redraw. Measured on the recording
benchmark, an evaluation that only summarizes falls from 24.92 µs to 3.29 µs.

The key is now the context itself: it hashes fullyQualifiedKey(), a string the
context already holds, and lets Equatable break ties. Contexts sharing a
canonical key but differing elsewhere collide in the hash and separate in ==, so
the Hashable contract holds.

It also clears redactAnonymousAttributes when building the key. That flag says
how a context is encoded for output rather than which context it is, but Swift
synthesizes == across every stored property, so leaving it set would let an
encoding concern split one context into two summary buckets. The Java SDK's
LDContext.equals has no such field, so clearing it is also what keeps Apple and
Android agreeing on which evaluations belong to the same summary.

This is more correct than what it replaces. Keying by a digest meant a collision
would silently merge two contexts' summaries; comparing the contexts themselves
cannot.

Co-authored-by: Cursor <cursoragent@cursor.com>
The event path is the one place the SDK encodes often enough for Codable's
reflection to show up in a profile. This writes the wire form directly into a
byte buffer instead, which is what the Android SDK already does, and caches the
encoded context so that a run of evaluations -- nearly always the same context
over and over -- encodes it once.

Getting there needed one change in the model. A feature event used to redact its
context by deep-copying LDContext and setting a flag on the copy, which cost an
allocation per event and defeated the buffer-identity comparison that makes a
context-keyed cache affordable. Redaction is a property of the event rather than
of the context -- one context redacts on a feature event and does not on the
debug event beside it -- so it now lives on Event, and the context is held as the
caller gave it.

JSONEncoder is kept as the oracle: EventJSONWriterTests asserts the two produce
identical bytes for every event shape, and where they disagree Codable is right.
The reporter's encoder is also built once rather than per delivery, and one
JSONWriter covers a whole run so its buffer is reused across the events in it.

Co-authored-by: Cursor <cursoragent@cursor.com>
The redaction test built a fresh event per encoding and compared the whole
JSON, so two events either side of a millisecond boundary disagreed on
creationDate and the assertion failed on a field it has nothing to say about.

Co-authored-by: Cursor <cursoragent@cursor.com>
A bounded flush spends up to a caller-supplied budget trying to deliver, then
reports whether the events left the SDK's hands. Android already had this; Apple
now has LDClient.flushAndWait(timeout:), spelled in seconds to match identify
and start.

EventReporting.flushReportingOutcome is the seam: true when the events were
delivered, refused for good, or there were none; false when the SDK is offline
or a retryable failure means they are still waiting. Saying that needed the
response classification to be a decision rather than a Bool, so
processEventResponse now returns settled, retryable, or dropped -- the three
outcomes that were already implied by its two callers.

TimeoutExecutor bounds the wait. Completions run on a queue that is neither main
nor the reporter's delivery queue, so a lifecycle caller on the main thread
cannot deadlock waiting for itself.

Backgrounding now spends its last moment trying to deliver, inside a
ProcessInfo activity assertion so the system does not suspend the process out
from under a request that has only just reached the network.

This is not a crash-time API. The SDK has no crash hook of its own on Apple, and
a Mach exception handler is not a supported caller.

Spec: Event Durability §10, §11, §17.3.
Co-authored-by: Cursor <cursoragent@cursor.com>
Tiers 1 and 2 both need the process to survive. When it does not, everything
recorded since the last delivery goes with it -- including the crash report an
application was recording when it died. Tier 3 is the answer to that: events are
written to a log under Application Support, and a later launch delivers whatever
the previous run left behind.

The write is a plain append into the page cache, never an fsync, so a commit
point costs tens of microseconds rather than milliseconds. track and identify
are commit points: at .immediate the write happens on the calling thread, so a
process that dies the instant after track returns still reports that event.
Evaluations are not -- they are held as objects and encoded in runs, so the
hottest path in the SDK pays a counter bump and nothing else.

This is off by default. Writing events changes what a track costs and leaves a
directory of the application's data on the device, and a minor version cannot
decide either on an application's behalf. LDConfig.eventPersistence has three
levels: disabled, which is what Tiers 1 and 2 describe and is the default;
deferred, which writes off the calling thread; and immediate, which is the whole
of the above.

Delivery is held for the round trip rather than just its start, because a batch
now stays on disk until its response is handled and a second delivery beginning
inside that window would send it again under the same payload ID.

Spec: Event Durability §13, §14; Page-Cache Event Persistence.
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the reverse-digit integer path; String(value).utf8 is 3× slower.

Co-authored-by: Cursor <cursoragent@cursor.com>
The counterpart of the Android benchmark of the same name, and deliberately so:
same context shapes, same privacy settings, same run length, same
warm-then-fastest-of-five aggregation. Run this in the Simulator and that one in
an emulator and both execute on the host Mac's cores, which is the only way the
two platforms' encoders have been compared without a difference in silicon
sitting in the middle of the answer.

It also prints the byte count beside every row, because only the shapes whose
output matches on both platforms can be quoted across them, and that is worth
checking rather than assuming.

Opt-in, and awkwardly so: the environment variable has to come from the scheme's
Test action, which xcodebuild's own environment does not reach. The doc comment
says how, rather than leaving it set in the shared scheme where it would add
twenty-five seconds to every CI run.

Co-authored-by: Cursor <cursoragent@cursor.com>
Only the benchmark on andrey/event-store-sqlite reads them. On this branch
they are dead state updated under the cache's lock on every lookup.

Co-authored-by: Cursor <cursoragent@cursor.com>
A single unserializable event -- a custom event with a non-finite metric
value, say -- made the whole flush return nil, so every event queued with it
was lost and the next flush hit the same wall. Encode event by event and skip
the ones that fail, logging each at error level.

Failures are dropped rather than put back: the writer cannot fail
transiently, so a retried event would fail every later flush too.
The cache was a long-lived object behind a configuration flag and its own
lock. None of those three earned their keep:

- Encoding happens in one place, EventReporter.encode(_:), which already
  builds a JSONWriter per batch. Giving the EventJSONWriter the same lifetime
  means the cache is reached from one thread by construction, so the
  UnfairLock goes. It was also not buying what it looked like: publish runs on
  a global queue, so two flushes in flight shared one two-entry cache and
  evicted each other.
- A batch is where the reuse is -- a run of evaluations on one context -- so
  nothing is lost by not outliving one.
- With no lifetime to choose, there is no mode to choose either, so
  cachingContexts and Encoding.handWrittenCachingContext go with it.

Redaction now writes _meta.redactedAttributes from Reference.canonical()
rather than the spelling the caller used. Reference equality is on parsed
components, so Reference("name") == Reference("/name"); writing the spelling
out meant two contexts could be equal and still encode differently, which the
cache had to close with a separate spellings comparison. Writing one spelling
closes it at the source instead, and both spellings are explicitly allowed in
redactedAttributes by the contract test harness.

The benchmark that measures the cache moves to andrey/event-store-sqlite with
the other benchmarks.
…-durability-tier2-bounded-flush

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
Tier 3 stages encoded bytes into the store instead of holding events in
memory, so the assertion that nothing was buffered has to ask the store.
The writer and the JSONWriter it wrote into had to be created, scoped and kept
off other threads together, but the caller was the one keeping them in step.
Now that EventJSONWriter has a lifetime of its own -- a batch -- the buffer
has the same one, so it lives inside and encode(_:) is the only entry point.
encode(_:into:) goes, and write(_:into:) becomes private.
…y/event-durability-tier3-persistence

# Conflicts:
#	LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence

Also renames handWrittenEncoder to eventJSONWriter, after its type.
The private helpers took the JSONWriter as a parameter from when the caller
supplied it. It is a stored property now, so the parameter only shadowed it.
Above 2^53 JSONEncoder switches to exponent form; the writer kept plain
integers up to Int64 range, which parsed to the same value but broke the
equal-JSON comparison from 1e17 up. Tests now assert equal JSON rather than
identical bytes, and pin the known byte-level differences (/ and -0).

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…-durability-tier2-bounded-flush

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly.xcodeproj/project.pbxproj
…y/event-durability-tier3-persistence

Co-authored-by: Cursor <cursoragent@cursor.com>

# Conflicts:
#	LaunchDarkly.xcodeproj/project.pbxproj
Co-authored-by: Cursor <cursoragent@cursor.com>
The completion contract promises a background queue, but NullEventReporter
answered inline, as did flush(completion:) on a closed client. The threading a
caller saw then depended on whether events were turned on, and one taking a
lock its completion also takes would deadlock only with them off.

flushAndWait(timeout: 0) is documented not to wait, and so to report false, but
it waited against an expired deadline, which succeeds where an answer already
landed. It now gives up explicitly, after starting the deliveries: at
.immediate those put the events on disk before returning, which is what makes a
zero timeout worth calling.

Co-authored-by: Cursor <cursoragent@cursor.com>
abelonogov-ld and others added 7 commits October 9, 2026 19:19
Every process shared one open log, named current. A second process starting
up recovered it as if a previous run had left it, renaming it into a batch
while the first was still appending through its open descriptor, so the
first's later events landed in a batch already sent or deleted.

The open log is now open-<process name>-<digest>, as on Android, and recovery
only touches this process's own. Closed batches stay shared, so a process that
never runs again still has its events delivered by another.

Co-authored-by: Cursor <cursoragent@cursor.com>
A Mac runs a second instance of an application under the same process
name, so a log named for the process alone would still be shared. Where
that can happen (macOS, and iPhone or iPad applications running on a
Mac), the open log is held under a non-blocking flock. An instance that
finds the name's log locked opens one named for its process ID instead,
and recovery closes off any open log no live process holds, whatever it
is named. The kernel drops the lock when the process dies, so a crash
never leaves a stale one.

iOS, tvOS and watchOS do not lock: a suspended application holding a
lock on a file in a shared container is killed (0xdead10cc), and there a
process name is one live process.

Co-authored-by: Cursor <cursoragent@cursor.com>
…e is off

Co-authored-by: Cursor <cursoragent@cursor.com>
A commit only stages events; whichever delivery runs next is what closes
them into a batch. So a flush could find everything claimed, conclude it
had nothing of its own left to send, and report success for events that
were about to fail with somebody else's delivery.

batchesInFlight becomes a map to a Claim holding the batch's outcome and
the deliveries waiting to hear it, and a DeliveryAnswer gathers this
delivery's own outcome with every claim it skipped. Only the answer
waits; the batches it closed for itself go out straight away.

The loss flag becomes a counter for the same reason. Each delivery takes
the count as it closes its batch and answers for the losses up to that
mark, so a loss reaches the caller whose delivery it happened before
rather than whichever response arrived first.

Claims also move off stateLock onto a claimLock of their own, so an
evaluating thread reading lastEventResponseDate never waits on a listing
of the store.

Co-authored-by: Cursor <cursoragent@cursor.com>
@abelonogov-ld
abelonogov-ld changed the base branch from andrey/event-durability-tier2-bounded-flush to v11 October 11, 2026 23:47
abelonogov-ld and others added 3 commits October 11, 2026 17:29
… sent twice

A write that fails partway can leave whole frames in the log while the store
keeps every byte of it in memory to deliver from there. The landed frames were
then sent twice: beside the memory copy when the log was closed this session,
or by recovery on the next launch. The log is now cut back to where the write
started before persistence is given up on.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ile it writes through

DiskEventLog reads apart from diagnostic logging; the store's property becomes
diskLog. File names on disk are unchanged, so a log left by an earlier version
is still recovered.

EventLogFile was a namespace of functions over a raw descriptor, with -1
standing for closed. It is now a value owning the descriptor, with writing,
sizing, truncating and closing as its methods, and DiskEventLog holds it as an
optional. The write hook tests use takes the file rather than a descriptor.

Co-authored-by: Cursor <cursoragent@cursor.com>
@abelonogov-ld
abelonogov-ld marked this pull request as ready for review October 12, 2026 02:01

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 81a7856. Configure here.


bufferLock.withLock { closedEvents = batches.reduce(0) { $0 + $1.eventCount } }

return batches

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closed event count races deliveries

Medium Severity

pendingBatches replaces closedEvents with a snapshot taken before it acquires bufferLock, while remove adjusts that counter from the network callback. Overlapping deliveries, which this change allows, can leave the counter too high and shed new events, or too low and store more than eventCapacity.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 81a7856. Configure here.

abelonogov-ld and others added 4 commits October 11, 2026 19:36
…oked up once

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@abelonogov-ld
abelonogov-ld marked this pull request as draft October 12, 2026 03:56
abelonogov-ld and others added 10 commits October 11, 2026 21:17
…lainly

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…th the protocol and null object in files of their own

Co-authored-by: Cursor <cursoragent@cursor.com>
…es from

Co-authored-by: Cursor <cursoragent@cursor.com>
…s it

Co-authored-by: Cursor <cursoragent@cursor.com>
…er, as Android has it

Co-authored-by: Cursor <cursoragent@cursor.com>
… cannot open room under capacity

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Moves the reporting and writing methods into same-file extensions to keep the type bodies under the limit, moves the LDValue non-finite check to LDCommon so LDContext stays under the file limit, and clears redundant nil coalescing, explicit nil initialization and void ternaries. No behavior change.

Co-authored-by: Cursor <cursoragent@cursor.com>
On macOS processAssertion runs the work inline and never asks the system, so the specs that wait for the system to end the assertion timed out there and failed test-swiftpm. macOS gets a spec for its own behavior instead.

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant