Skip to content

fix: Use monotonic clocks for intervals, durations, and deadlines - #459

Merged
tanderson-ld merged 1 commit into
mainfrom
tanderson/sdk-3104-monotonic-clock
Oct 6, 2026
Merged

tanderson-ld merged 1 commit into
mainfrom
tanderson/sdk-3104-monotonic-clock

Conversation

@tanderson-ld

@tanderson-ld tanderson-ld commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replaces wall-clock time with monotonic time everywhere the SDK measures an interval, duration, or deadline. Wall clocks (Time.now, epoch millis) remain only for values that leave the process or are compared against externally supplied absolute times. Implements SDK-3104.

A wall-clock step (NTP correction, manual clock set, container/VM time resync, the time correction when a frozen environment thaws) previously could cause:

  • RepeatingTask (polling, event flush, big-segment status polling, store-status polling): an immediate re-run on a forward step, or up to interval-plus-step of silence on a backward step.
  • ExpiringCache (big-segments membership cache, persistent-store read-through cache): entries served past their TTL after a backward step; whole-cache eviction (thundering herd against Redis/DynamoDB) on a forward step. A backward step could also strand genuinely expired entries behind the eviction scan's early exit until capacity pressure removed them.
  • Migration op latency: inflated values on a forward step; silently dropped measurements on a backward step (the tracker discards negatives), skewing old/new latency comparisons.
  • FDv2 fallback/recovery verdicts (60s/10s/300s windows): a step inside the window could trigger or suppress synchronizer failover.
  • Stream-init diagnostics: FDv1 sent negative durationMillis after a backward step; FDv2 clamped negatives but still inflated forward steps.

Changes

  • Impl::Util.monotonic_seconds (Process.clock_gettime(CLOCK_MONOTONIC, :float_second)) is the single monotonic seam — now the only clock read in lib/ besides Util.current_time_millis.
  • RepeatingTask measures each run monotonically; fixed-rate semantics unchanged.
  • ExpiringCache stamps and evicts on an injectable monotonic clock (clock: kwarg). This also makes the eviction scan's "stop at first unexpired entry" early exit a true invariant (stamps are now non-decreasing in insertion order).
  • Migrations::Executor measures operation latency monotonically.
  • StatusProviderV2 keeps a monotonic twin of state_since, stamped under the same write lock on state change, exposed via the atomic status_and_seconds_in_state (one read-lock acquisition returns [status, seconds], so a verdict can never pair a stale state with a fresh duration). Public status.state_since remains a wall-clock Time. Also fixed here: a nil state from a custom synchronizer previously fell into the change path and could permanently disable FDv2 fallback/recovery; it is now ignored, matching the FDv1 sink.
  • FDv2 fallback_condition / recovery_condition are pure functions of (status, seconds_in_state), with a truth-table spec pinning the 60/10/300s thresholds and the strict-> boundary semantics.
  • FDv1/FDv2 stream-init durationMillis is measured monotonically (integer ms); the reported timestamp stays wall-clock. The monotonic stamp is also the "attempt in flight" sentinel, so lifecycle and measurement share one typed value.

Deliberately unchanged (wall clock is correct): analytics event creationDate, diagnostic creationDate/dataSinceDate, Status#state_since, ErrorInfo#time, debugEventsUntilDate comparisons (server-supplied absolute dates), and big-segments staleness — last_up_to_date is written by another process, so only a wall clock is comparable across hosts; it remains sensitive to host-to-host skew, which no local clock choice can fix.

Behavioral notes

Not a breaking change. All modified classes are Impl-private; additions are optional kwargs and a @private accessor; behavior under a stable clock is identical. Three nuances:

  1. Suspend semantics. Linux CLOCK_MONOTONIC does not advance during system suspend, so TTLs and the FDv2 state windows now measure process-visible time. We considered CLOCK_BOOTTIME and chose CLOCK_MONOTONIC everywhere for cross-platform consistency (macOS/JRuby lack BOOTTIME, and macOS's monotonic clock already counts sleep). Notably, this costs nothing in AWS Lambda: a frozen execution environment pauses the entire microVM with no guest-visible suspend, so BOOTTIME and MONOTONIC behave identically there. The practical consequence in warm Lambdas is that TTLs become cumulative active time — persistent-store/big-segments cache entries can live longer in wall time than the configured TTL under sporadic short invocations (the old wall-clock behavior instead re-fetched everything at thaw, since the thaw time-sync is a forward step). If that ever matters in practice, dual-stamping entries (wall + monotonic, expire on whichever elapses first) is the known escape hatch.
  2. Application test suites that time-travel. Apps using Timecop (or similar) to drive SDK-internal timing — e.g. jumping Time.now to expire the SDK's caches — will find the SDK no longer follows the wall clock. Timecop's opt-in Timecop.mock_process_clock = true with the offset forms (Timecop.travel(3600)) still works, since it mocks Process.clock_gettime. Our own ExpiringCache specs needed exactly this rewrite.
  3. Measurement/report split. Where a wall timestamp and a monotonic duration appear side by side (stream inits), timestamp + durationMillis will not equal the wall end time if a step occurred mid-measurement. The duration is the trustworthy value.

One correction to the originating audit's framing: a Ruby Time is an absolute epoch instant (the zone affects display only), so Time subtraction was never DST-affected — Ruby's actual exposure was NTP/manual/VM clock steps. The fixes are identical either way.

Review

A three-agent review (general / security / adversarial) ran against these changes: no Critical or High findings. All seven adversarial attack vectors failed against production code — including an exhaustive 4096-sequence brute force showing the wall/monotonic stamps cannot diverge through any update_status input sequence, and a 3.1M-sample race probe of the status/duration pairing (zero violations, zero negative durations). Each new regression test was verified to fail against the pre-fix implementation. Review-driven hardening is included here (atomic accessor, nil-state guard, condition truth table, spec Timecop hygiene).

Two pre-existing issues surfaced by the review are intentionally out of scope and will be ticketed separately: no minimum gap between RepeatingTask runs when a task overruns its interval (reachable via an unvalidated status_poll_interval), and interval type validation.

Relationship to #448

The RepeatingTask rewrite in #448 removes the elapsed-time measurement entirely (fixed-delay semantics), which supersedes this PR's repeating_task.rb change. The fix here is deliberate insurance on main's current code: whichever PR lands second should resolve the conflict in #448's favor for that one file. No other files overlap.

Precedent

Same change-class as the .NET fixes in launchdarkly/dotnet-core#339 (propagated by launchdarkly/dotnet-core#343).

Testing

Full suite passes (1118 examples; the one failure, launchdarkly-server-sdk_spec.rb's Bundler.require test, is pre-existing and environment-dependent — it shells out to a bare ruby outside the bundle — and touches nothing in this change). New regression tests simulate wall-clock steps for the scheduler, cache, migrator, status provider, and both stream-init paths. rubocop clean on all changed files.


Note

Overview
Wall-clock steps (NTP, manual set, VM resync) could skew polling intervals, cache TTLs, migration latency, FDv2 synchronizer failover/recovery, and stream-init diagnostics. This PR routes all internal interval and duration measurement through new Impl::Util.monotonic_seconds, while keeping wall time for outward-facing timestamps like Status#state_since.

StatusProviderV2 tracks a monotonic twin of state entry time, exposes atomic status_and_seconds_in_state, and ignores nil state updates. FDv2 uses that duration for the 60s / 10s / 300s fallback and recovery thresholds instead of Time.now - state_since. FDv1 and FDv2 streaming compute diagnostic durationMillis from monotonic stamps; reported init timestamps stay wall-clock. ExpiringCache, RepeatingTask, and migration Executor latency also switch to monotonic measurement (cache accepts an optional injectable clock: for tests).

Regression specs cover wall-clock manipulation via Timecop, FDv2 condition boundaries, and global Timecop.return after each example.

Reviewed by Cursor Bugbot for commit 49aa78a. Bugbot is set up for automated code reviews on this repo. Configure here.

Wall-clock steps (NTP corrections, manual clock sets, VM/container
time resync) could cause immediate re-polls or hour-long silences in
RepeatingTask, stale or mass-evicted ExpiringCache entries, skewed or
silently dropped migration latencies, wrong FDv2 fallback/recovery
verdicts, and negative stream-init durations. Every interval, duration,
and deadline now reads Util.monotonic_seconds; wall clocks remain only
for values reported outside the process or compared against
server-supplied absolute dates.

Note for applications that use Timecop to drive SDK-internal timing:
these measurements no longer follow Time.now. Timecop's opt-in
mock_process_clock with the offset forms (e.g. Timecop.travel(3600))
still works.

SDK-3104
@tanderson-ld
tanderson-ld requested a review from a team as a code owner October 2, 2026 20:33
@tanderson-ld
tanderson-ld requested a review from keelerm84 October 2, 2026 20:33
@tanderson-ld
tanderson-ld merged commit 5016129 into main Oct 6, 2026
12 checks passed
@tanderson-ld
tanderson-ld deleted the tanderson/sdk-3104-monotonic-clock branch October 6, 2026 20:47
tanderson-ld added a commit that referenced this pull request Oct 7, 2026
🤖 I have created a release *beep* *boop*
---


##
[8.18.2](8.18.1...8.18.2)
(2026-10-06)


### Bug Fixes

* Use monotonic clocks for intervals, durations, and deadlines
([#459](#459))
([5016129](5016129))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Release 8.18.2** — bumps the SDK version from `8.18.1` to `8.18.2`
in `lib/ldclient-rb/version.rb`, `.release-please-manifest.json`, and
`PROVENANCE.md`, and adds the corresponding `CHANGELOG.md` section.
> 
> The changelog records one shipped fix: **intervals, durations, and
deadlines now use monotonic clocks**
([#459](#459)), so
timing for things like data-source status duration is not skewed by
wall-clock adjustments (NTP, DST, manual changes). This PR contains no
runtime code changes—only release bookkeeping from Release Please.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
40431a6. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
tanderson-ld added a commit to launchdarkly/python-server-sdk that referenced this pull request Oct 12, 2026
…ndows (#537)

## Summary

Replaces wall-clock time with monotonic time in the two remaining places
this SDK measures a duration or decides behavior from one: stream-init
diagnostics and the FDv2 fallback/recovery windows. Wall clocks remain
only for values reported outside the process or compared against
externally supplied times. Implements SDK-3105.

A wall-clock step (NTP correction, manual clock set, container/VM time
resync, the time correction when a frozen environment thaws) previously
could cause:

- **Stream-init diagnostics** (`datasource/streaming.py`,
`datasource/async_streaming.py`): a forward step inflated the reported
`durationMillis` by the step size; a backward step hit the `elapsed >= 0
else 0` clamp and silently reported zero — a lost measurement either
way.
- **FDv2 fallback/recovery verdicts** (`datasystem/fdv2_common.py`,
60s/10s/300s windows): a forward step could make a just-interrupted
source look an hour degraded and trigger an immediate synchronizer
fallback; a backward step could suppress a legitimate one.

Two audit findings in this story's original scope — the wall-clock
schedulers in `repeating_task.py` and `aio/concurrency.py` — were
already resolved on main by the `RepeatingTask`/`DelaySource` rework
that shipped in 9.18.0 (the wait is event-based with no clock
arithmetic), so they appear here as this note rather than as code.

## Changes

- `Impl::util.monotonic_seconds()` (`time.monotonic()`) is the single
monotonic seam.
- **Stream-init**: the connection-attempt stamp is now monotonic and
serves as both the duration base and the "attempt in flight" sentinel
(explicit `is not None`, which also fixes a latent float-truthiness wart
in the old guard). The reported `timestamp` stays wall-clock; the
negative-value clamp is removed because a negative duration is now
impossible.
- **FDv2 windows**: the condition-check loop in each coordinator owns a
small `_StateAgeTracker` — created per synchronizer session, so windows
reset naturally on failover — which measures time-in-state monotonically
from first observation. Its identity key is `(state, since)`: a real
transition changes `since` even when the state repeats (an
interrupted→valid→interrupted flap between two 10-second checks resets
the window), while a same-state error update preserves it (a stream of
error reports cannot keep resetting the clock). `since` is used only for
equality, never arithmetic. `fallback_condition`/`recovery_condition`
take the measured seconds as a parameter and are pure, table-testable
predicates. This observer-owned shape matches the Rust server SDK's FDv2
implementation, which builds its failover deadlines from
`Instant::now()` inside the coordinator.
- The status provider and the public `DataSourceStatus` are untouched —
zero diff against main.

**Deliberately unchanged** (wall clock is correct):
`DataSourceStatus.since`, `DataSourceErrorInfo.time`, event
`creationDate`s, diagnostic payload timestamps, `debugEventsUntilDate`
comparisons, and big-segments staleness (cross-process stamp).

## Behavioral notes

**Not a breaking change** — every modified component is `Impl`-internal;
the public status surface is untouched; behavior under a stable clock is
identical, with one bounded exception:

1. **Observation latency.** Time-in-state is measured from when the
check loop first *sees* a state, not from the transition itself, so a
verdict can fire up to one 10-second check period later than before
(material only for the 10-second INITIALIZING window, where worst-case
detection moves from ~20s to ~30s). Always in the safe direction —
defer, never spuriously trigger.
2. **Suspend semantics.** `time.monotonic()` on Linux does not advance
during system suspend, so these measurements now count process-visible
time. Notably this costs nothing in AWS Lambda: a frozen execution
environment pauses the entire microVM with no guest-visible suspend, so
every in-process clock behaves identically there — while the thaw-time
wall correction is exactly the kind of step this change defends against.
3. **Test suites that freeze time.** Applications using freezegun (or
similar) to drive SDK-internal timing should note these measurements no
longer follow `time.time()`; freezegun does not patch `time.monotonic()`
by default.

## Known sibling, deliberately out of scope

`ldclient/impl/datasourcev2/streaming.py` and `async_streaming.py` carry
the same wall-clock stream-init pattern (including a wall-clock
future-start stamp from `classify_stream_error`). Those files postdate
the originating audit; they'll be handled under a separate decision
rather than folded in here.

## Relationship to the Ruby change

Same change-class as
launchdarkly/ruby-server-sdk#459. The FDv2
window fix intentionally uses a different shape than Ruby's
(observer-owned tracker here vs. a provider-held stamp there); the Rust
FDv2 implementation is the shared precedent for the approach used here.

## Testing

Full suite passes (1766 passed, 371 skipped — database integrations).
New `test_fdv2_conditions.py` covers the threshold truth table
(strict-`>` boundaries pinned), the tracker's
first-observation/flap/error-spam semantics, and a regression test
proving a +1h wall step can no longer trigger a spurious fallback.
Wall-step regression tests added to both streaming test suites use a
*forward* step deliberately: the old code clamped negatives to zero, so
only the forward direction distinguishes old from new behavior. mypy,
isort, and pycodestyle clean.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Replaces wall-clock timing with **monotonic** measurements for two
internal SDK behaviors so NTP or manual clock jumps cannot skew
diagnostics or failover decisions.
> 
> **Stream-init diagnostics** (sync and async streaming processors):
connection-attempt duration is computed from `monotonic_seconds()`
instead of `time.time()`, while reported timestamps stay wall-clock. The
old negative-duration clamp is removed; retry backoff is excluded from
measured latency by stamping the attempt start after the wait.
> 
> **FDv2 fallback/recovery** (`fdv2_common`, wired in sync/async
coordinators): a per-session `_StateAgeTracker` measures how long the
current data-source state has been observed on the monotonic timeline
(keyed by `(state, since)` so error spam does not reset the window).
`fallback_condition` / `recovery_condition` now take that measured
`seconds_in_state` instead of `time.time() - status.since`.
> 
> Adds `monotonic_seconds()` in `impl/util.py` and regression tests for
wall-clock steps and FDv2 thresholds.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
ee290ec. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants