Repository navigation
feat(events): persist events to disk so they survive the process (tier 3) - #538
Draft
abelonogov-ld wants to merge 105 commits into
Draft
abelonogov-ld wants to merge 105 commits into
abelonogov-ld wants to merge 105 commits into
Conversation
…igest ContextSummarizer keyed its per-context trackers by LDContext.contextHash(), which encodes the context to JSON and takes a SHA-256 of the result. That is the right price for the flag cache, which persists the digest and compares it across launches, but a bucketing key does not need to cost it — and an application that reads a flag on every redraw paid it on every redraw. Measured on the recording benchmark, an evaluation that only summarizes falls from 24.92 µs to 3.29 µs. The key is now the context itself: it hashes fullyQualifiedKey(), a string the context already holds, and lets Equatable break ties. Contexts sharing a canonical key but differing elsewhere collide in the hash and separate in ==, so the Hashable contract holds. It also clears redactAnonymousAttributes when building the key. That flag says how a context is encoded for output rather than which context it is, but Swift synthesizes == across every stored property, so leaving it set would let an encoding concern split one context into two summary buckets. The Java SDK's LDContext.equals has no such field, so clearing it is also what keeps Apple and Android agreeing on which evaluations belong to the same summary. This is more correct than what it replaces. Keying by a digest meant a collision would silently merge two contexts' summaries; comparing the contexts themselves cannot. Co-authored-by: Cursor <cursoragent@cursor.com>
The event path is the one place the SDK encodes often enough for Codable's reflection to show up in a profile. This writes the wire form directly into a byte buffer instead, which is what the Android SDK already does, and caches the encoded context so that a run of evaluations -- nearly always the same context over and over -- encodes it once. Getting there needed one change in the model. A feature event used to redact its context by deep-copying LDContext and setting a flag on the copy, which cost an allocation per event and defeated the buffer-identity comparison that makes a context-keyed cache affordable. Redaction is a property of the event rather than of the context -- one context redacts on a feature event and does not on the debug event beside it -- so it now lives on Event, and the context is held as the caller gave it. JSONEncoder is kept as the oracle: EventJSONWriterTests asserts the two produce identical bytes for every event shape, and where they disagree Codable is right. The reporter's encoder is also built once rather than per delivery, and one JSONWriter covers a whole run so its buffer is reused across the events in it. Co-authored-by: Cursor <cursoragent@cursor.com>
The redaction test built a fresh event per encoding and compared the whole JSON, so two events either side of a millisecond boundary disagreed on creationDate and the assertion failed on a field it has nothing to say about. Co-authored-by: Cursor <cursoragent@cursor.com>
A bounded flush spends up to a caller-supplied budget trying to deliver, then reports whether the events left the SDK's hands. Android already had this; Apple now has LDClient.flushAndWait(timeout:), spelled in seconds to match identify and start. EventReporting.flushReportingOutcome is the seam: true when the events were delivered, refused for good, or there were none; false when the SDK is offline or a retryable failure means they are still waiting. Saying that needed the response classification to be a decision rather than a Bool, so processEventResponse now returns settled, retryable, or dropped -- the three outcomes that were already implied by its two callers. TimeoutExecutor bounds the wait. Completions run on a queue that is neither main nor the reporter's delivery queue, so a lifecycle caller on the main thread cannot deadlock waiting for itself. Backgrounding now spends its last moment trying to deliver, inside a ProcessInfo activity assertion so the system does not suspend the process out from under a request that has only just reached the network. This is not a crash-time API. The SDK has no crash hook of its own on Apple, and a Mach exception handler is not a supported caller. Spec: Event Durability §10, §11, §17.3. Co-authored-by: Cursor <cursoragent@cursor.com>
Tiers 1 and 2 both need the process to survive. When it does not, everything recorded since the last delivery goes with it -- including the crash report an application was recording when it died. Tier 3 is the answer to that: events are written to a log under Application Support, and a later launch delivers whatever the previous run left behind. The write is a plain append into the page cache, never an fsync, so a commit point costs tens of microseconds rather than milliseconds. track and identify are commit points: at .immediate the write happens on the calling thread, so a process that dies the instant after track returns still reports that event. Evaluations are not -- they are held as objects and encoded in runs, so the hottest path in the SDK pays a counter bump and nothing else. This is off by default. Writing events changes what a track costs and leaves a directory of the application's data on the device, and a minor version cannot decide either on an application's behalf. LDConfig.eventPersistence has three levels: disabled, which is what Tiers 1 and 2 describe and is the default; deferred, which writes off the calling thread; and immediate, which is the whole of the above. Delivery is held for the round trip rather than just its start, because a batch now stays on disk until its response is handled and a second delivery beginning inside that window would send it again under the same payload ID. Spec: Event Durability §13, §14; Page-Cache Event Persistence. Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the reverse-digit integer path; String(value).utf8 is 3× slower. Co-authored-by: Cursor <cursoragent@cursor.com>
The counterpart of the Android benchmark of the same name, and deliberately so: same context shapes, same privacy settings, same run length, same warm-then-fastest-of-five aggregation. Run this in the Simulator and that one in an emulator and both execute on the host Mac's cores, which is the only way the two platforms' encoders have been compared without a difference in silicon sitting in the middle of the answer. It also prints the byte count beside every row, because only the shapes whose output matches on both platforms can be quoted across them, and that is worth checking rather than assuming. Opt-in, and awkwardly so: the environment variable has to come from the scheme's Test action, which xcodebuild's own environment does not reach. The doc comment says how, rather than leaving it set in the shared scheme where it would add twenty-five seconds to every CI run. Co-authored-by: Cursor <cursoragent@cursor.com>
Only the benchmark on andrey/event-store-sqlite reads them. On this branch they are dead state updated under the cache's lock on every lookup. Co-authored-by: Cursor <cursoragent@cursor.com>
A single unserializable event -- a custom event with a non-finite metric value, say -- made the whole flush return nil, so every event queued with it was lost and the next flush hit the same wall. Encode event by event and skip the ones that fail, logging each at error level. Failures are dropped rather than put back: the writer cannot fail transiently, so a retried event would fail every later flush too.
The cache was a long-lived object behind a configuration flag and its own
lock. None of those three earned their keep:
- Encoding happens in one place, EventReporter.encode(_:), which already
builds a JSONWriter per batch. Giving the EventJSONWriter the same lifetime
means the cache is reached from one thread by construction, so the
UnfairLock goes. It was also not buying what it looked like: publish runs on
a global queue, so two flushes in flight shared one two-entry cache and
evicted each other.
- A batch is where the reuse is -- a run of evaluations on one context -- so
nothing is lost by not outliving one.
- With no lifetime to choose, there is no mode to choose either, so
cachingContexts and Encoding.handWrittenCachingContext go with it.
Redaction now writes _meta.redactedAttributes from Reference.canonical()
rather than the spelling the caller used. Reference equality is on parsed
components, so Reference("name") == Reference("/name"); writing the spelling
out meant two contexts could be equal and still encode differently, which the
cache had to close with a separate spellings comparison. Writing one spelling
closes it at the source instead, and both spellings are explicitly allowed in
redactedAttributes by the contract test harness.
The benchmark that measures the cache moves to andrey/event-store-sqlite with
the other benchmarks.
…-durability-tier2-bounded-flush # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
Tier 3 stages encoded bytes into the store instead of holding events in memory, so the assertion that nothing was buffered has to ask the store.
The writer and the JSONWriter it wrote into had to be created, scoped and kept off other threads together, but the caller was the one keeping them in step. Now that EventJSONWriter has a lifetime of its own -- a batch -- the buffer has the same one, so it lives inside and encode(_:) is the only entry point. encode(_:into:) goes, and write(_:into:) becomes private.
…-durability-tier2-bounded-flush
…y/event-durability-tier3-persistence # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…-durability-tier2-bounded-flush
…y/event-durability-tier3-persistence Also renames handWrittenEncoder to eventJSONWriter, after its type.
The private helpers took the JSONWriter as a parameter from when the caller supplied it. It is a stored property now, so the parameter only shadowed it.
…-durability-tier2-bounded-flush
…y/event-durability-tier3-persistence
Above 2^53 JSONEncoder switches to exponent form; the writer kept plain integers up to Int64 range, which parsed to the same value but broke the equal-JSON comparison from 1e17 up. Tests now assert equal JSON rather than identical bytes, and pin the known byte-level differences (/ and -0). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…-durability-tier2-bounded-flush Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly.xcodeproj/project.pbxproj
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly.xcodeproj/project.pbxproj
Co-authored-by: Cursor <cursoragent@cursor.com>
The completion contract promises a background queue, but NullEventReporter answered inline, as did flush(completion:) on a closed client. The threading a caller saw then depended on whether events were turned on, and one taking a lock its completion also takes would deadlock only with them off. flushAndWait(timeout: 0) is documented not to wait, and so to report false, but it waited against an expired deadline, which succeeds where an answer already landed. It now gives up explicitly, after starting the deliveries: at .immediate those put the events on disk before returning, which is what makes a zero timeout worth calling. Co-authored-by: Cursor <cursoragent@cursor.com>
2 of 3 tasks
Every process shared one open log, named current. A second process starting up recovered it as if a previous run had left it, renaming it into a batch while the first was still appending through its open descriptor, so the first's later events landed in a batch already sent or deleted. The open log is now open-<process name>-<digest>, as on Android, and recovery only touches this process's own. Closed batches stay shared, so a process that never runs again still has its events delivered by another. Co-authored-by: Cursor <cursoragent@cursor.com>
A Mac runs a second instance of an application under the same process name, so a log named for the process alone would still be shared. Where that can happen (macOS, and iPhone or iPad applications running on a Mac), the open log is held under a non-blocking flock. An instance that finds the name's log locked opens one named for its process ID instead, and recovery closes off any open log no live process holds, whatever it is named. The kernel drops the lock when the process dies, so a crash never leaves a stale one. iOS, tvOS and watchOS do not lock: a suspended application holding a lock on a file in a shared container is killed (0xdead10cc), and there a process name is one live process. Co-authored-by: Cursor <cursoragent@cursor.com>
…e is off Co-authored-by: Cursor <cursoragent@cursor.com>
A commit only stages events; whichever delivery runs next is what closes them into a batch. So a flush could find everything claimed, conclude it had nothing of its own left to send, and report success for events that were about to fail with somebody else's delivery. batchesInFlight becomes a map to a Claim holding the batch's outcome and the deliveries waiting to hear it, and a DeliveryAnswer gathers this delivery's own outcome with every claim it skipped. Only the answer waits; the batches it closed for itself go out straight away. The loss flag becomes a counter for the same reason. Each delivery takes the count as it closes its batch and answers for the losses up to that mark, so a loss reaches the caller whose delivery it happened before rather than whichever response arrived first. Claims also move off stateLock onto a claimLock of their own, so an evaluating thread reading lastEventResponseDate never waits on a listing of the store. Co-authored-by: Cursor <cursoragent@cursor.com>
abelonogov-ld
changed the base branch from
andrey/event-durability-tier2-bounded-flush
to
v11
October 11, 2026 23:47
… sent twice A write that fails partway can leave whole frames in the log while the store keeps every byte of it in memory to deliver from there. The landed frames were then sent twice: beside the memory copy when the log was closed this session, or by recovery on the next launch. The log is now cut back to where the write started before persistence is given up on. Co-authored-by: Cursor <cursoragent@cursor.com>
…ile it writes through DiskEventLog reads apart from diagnostic logging; the store's property becomes diskLog. File names on disk are unchanged, so a log left by an earlier version is still recovered. EventLogFile was a namespace of functions over a raw descriptor, with -1 standing for closed. It is now a value owning the descriptor, with writing, sizing, truncating and closing as its methods, and DiskEventLog holds it as an optional. The write hook tests use takes the file rather than a descriptor. Co-authored-by: Cursor <cursoragent@cursor.com>
abelonogov-ld
marked this pull request as ready for review
October 12, 2026 02:01
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 81a7856. Configure here.
|
|
||
| bufferLock.withLock { closedEvents = batches.reduce(0) { $0 + $1.eventCount } } | ||
|
|
||
| return batches |
There was a problem hiding this comment.
Closed event count races deliveries
Medium Severity
pendingBatches replaces closedEvents with a snapshot taken before it acquires bufferLock, while remove adjusts that counter from the network callback. Overlapping deliveries, which this change allows, can leave the counter too high and shed new events, or too low and store more than eventCapacity.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 81a7856. Configure here.
…oked up once Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
abelonogov-ld
marked this pull request as draft
October 12, 2026 03:56
…lainly Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…th the protocol and null object in files of their own Co-authored-by: Cursor <cursoragent@cursor.com>
…es from Co-authored-by: Cursor <cursoragent@cursor.com>
…s it Co-authored-by: Cursor <cursoragent@cursor.com>
…er, as Android has it Co-authored-by: Cursor <cursoragent@cursor.com>
… cannot open room under capacity Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Moves the reporting and writing methods into same-file extensions to keep the type bodies under the limit, moves the LDValue non-finite check to LDCommon so LDContext stays under the file limit, and clears redundant nil coalescing, explicit nil initialization and void ternaries. No behavior change. Co-authored-by: Cursor <cursoragent@cursor.com>
On macOS processAssertion runs the work inline and never asks the system, so the specs that wait for the system to end the assertion timed out there and failed test-swiftpm. macOS gets a spec for its own behavior instead. Co-authored-by: Cursor <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Third in the event-durability stack, on top of #536. Recorded events go to an on-disk log rather than an in-memory buffer, so an application that dies moments after recording one still reports it: the next run delivers what the last one left behind.
What changes
EventStoreis a new on-disk log, with capacity, framing, interrupted-log recovery, and multi-process safety.NullEventStorestands in where the platform gave the SDK nowhere to write, which is the behaviour the SDK had before.Recording no longer encodes on the caller's thread. An evaluation appends to a held run and updates a counter; a commit turns the whole run into bytes at once. Recording a custom or identify event is a commit point, because an application reporting something it chose to report is saying it matters more than the microseconds it costs. Evaluations commit once
pendingCommitThresholdof them accumulate, when a delivery starts, or on flush.LDConfig.eventPersistencelets the application choose whether a commit point runs on the caller's thread (.immediate), off it (.deferred), or not at all (.disabled).Delivery sends batches oldest first and stops at the first retryable failure, so the rest keep their place in the log instead of the SDK spending the session's requests on a service that is refusing them.
Reconciling with tier 2
Tier 2 replaced the delivery gate this branch was built on, so the merge had to pick one design rather than keep both.
Coalescing comes from tier 2: flushes arriving before a delivery starts join it, one already started is never joined, and nothing holds the drain back while a retry runs. What tier 2 did not need is a way to keep an in-flight batch from being sent twice — it empties its buffer before publishing, so a second delivery finds nothing. Here a batch stays on disk until its response arrives, which is what
isDeliveringguarded.batchesInFlighttakes its place. A delivery claims the batches it lists and releases them once it has an answer, so a later delivery still runs and still sends what it closed for itself; only the batches already on their way are left out. A caller whose own events are all that remain unsent is answeredtrue, which is exactly the contract tier 2 documents: an answer is about the batch the caller's own delivery took, never one an earlier delivery lost.The loss flag is now taken once per delivery rather than once per caller, so everyone joined to it hears about a loss instead of only whoever was answered first, and a scheduled delivery with no waiters leaves it for the caller who is owed it. This branch had also fixed the duplicate event reporter independently; that now comes from #537, so the local copy is gone.
Verification
does not send a batch again when a delivery is asked for while one is in flight.DiagnosticReporterSpec.testRetriesflaked once under load and passed on re-run; it has a 0.1 s timeout and is unrelated to these changes.Note
Overview
Adds Tier 3 event durability: analytics events can be appended to an on-disk log (
EventStorewith framedEventLogFormat, recovery of interrupted logs, and macOS file locking) so they survive process death and are delivered on the next launch.LDConfig.eventPersistence(.disabled/.deferred/.immediate) controls whether commits run off-thread, on the caller’s thread attrack/identify, or not at all; defaulteventCapacityrises from 100 to 1000.Flush behavior moves to
LDClientFlushand grows: fire-and-forgetflush(),flush(completion:)andflushAndWait(timeout:)(plus Obj-C equivalents) aggregate all environment clients and report delivery success viaflushReportingOutcome. On iOS/tvOS background, the clientcommitAtCommitPoint()then flushes under aBackgroundActivityassertion so events are durable and a best-effort send runs before suspension.Encoding path changes:
EventJSONWriter,JSONWriter,ContextEncodingCache, andLDContextJSONWriterreplace context copying for anonymous redaction—redaction is driven per event kind viaEvent.redactsAnonymousAttributes.ContextSummarizerkeys contexts by full equality (with caps when too many distinct contexts) instead ofcontextHash()alone.Supporting tweaks:
DiagnosticCacheusesUnfairLockinstead of a serial queue;DarklyService.publishEventDatainvokes completion when the mobile key is missing so reporters do not stall.Reviewed by Cursor Bugbot for commit 81a7856. Bugbot is set up for automated code reviews on this repo. Configure here.