Skip to content

feat: Add file loading code for flag overrides - #449

Draft
kinyoklion wants to merge 11 commits into
feat/overridesfrom
rlamb/overrides-ruby-filedata
Draft

kinyoklion wants to merge 11 commits into
feat/overridesfrom
rlamb/overrides-ruby-filedata

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Summary

This is the first of a series that adds flag overrides, as defined by the OVERRIDE specification, to the Ruby server SDK. The existing FDv1 and FDv2 file data sources keep their current behavior: this PR does not change lib/ldclient-rb/impl/integrations/file_data_source.rb, file_data_source_v2.rb, or their specs, and the override feature is additive throughout the series.

The file-based override source must tolerate a missing file, a file that is being written, and a burst of change notifications, and it must keep the last good data when a reload fails. This change adds LaunchDarkly::Impl::FileData, the file reading, parsing, merging, and reloading code that source needs:

  • Document parsing for JSON and YAML. The parser checks that the document and its flags, flagValues, and segments members are objects.
  • An ordered merge of several documents with duplicate keys handling of fail or ignore, expansion of each flagValues entry into a flag that is on and serves the value by fallthrough (so an evaluation reports the FALLTHROUGH reason kind, as the specification describes), deserialization into the data model classes, and per-document entry counts.
  • A reloader that serializes reloads, debounces change signals, keeps the last good result when a reload fails, retries after a bounded delay, reports each distinct failure once, and can skip a reload whose content is byte-identical to the last applied content. A configured file that does not exist can be treated as a file with no content. An exception raised by the consumer's apply is treated as a failed reload: nothing from that load is remembered, the failure is reported, and the reload is retried.
  • A stat-based poller that compares existence, modification time, and size. A file that appears or disappears is a change.
  • A watcher that uses rb-inotify on Linux (a dependency of the optional listen gem) to watch the directory of each file without descending into subdirectories, and listen elsewhere. It picks up a file that does not exist yet when it appears, and it watches each directory on its own: a directory that does not exist is retried once per second while the others stay watched, and one change is signaled when a retry sets up a watch. Non-recursive watching matters because listen scans the whole tree under a watched directory and fails when a subdirectory cannot be read, as happens for a file in the system temporary directory. On Linux, a watched directory that is deleted is watched again once it exists, and one change is signaled so the reload picks up what was written meanwhile.

A new spec, spec/integrations/file_data_source_compatibility_spec.rb, pins the behaviors of the existing file data sources that their other specs did not assert (per-file version numbering, keying by the entry's own key, the on-plus-fallthrough expansion of flagValues, no retry of a failed load, log messages, modification-time-only polling, no reaction to a deleted file when polling, the FDv1 poller's reload cadence after a change, and use of listen when it is present), so that they stay as they are while this code exists beside them.

Verification: unit specs for the document parser, merge, poller, reloader (including debounce, retry, last-good retention, sequential reloads, and stop), and watcher (including a directory with an unreadable subdirectory and the inotify thread lifecycle), the new compatibility pins, and the unchanged existing file data source specs. Full suite and RuboCop are clean. Each new spec was checked against a deliberate defect in the code it covers, and each compatibility pin against a deliberate change to the existing source it pins.

SDK-3249


Note

Overview
Introduces LaunchDarkly::Impl::FileData, shared infrastructure for upcoming file-based flag overrides. It does not change the existing FDv1/FDv2 integrations in this PR.

Document handling: JSON/YAML parsing into flags, flagValues, and segments, with ReadError for per-file failures and helpers to expand flagValues into on/fallthrough flags.

Multi-file merge: Ordered combination with fail or ignore duplicate-key policy, model deserialization, and per-document counts.

Reload pipeline: Reloader serializes loads, debounces change signals, retries failures, keeps the last good data on error, optionally skips byte-identical reloads, and can treat missing files as empty. Poller detects changes via existence/mtime/size; Watcher uses rb-inotify (Linux, non-recursive) or optional listen elsewhere, with retries for missing or deleted watch directories.

Tests: Unit specs for each component plus file_data_source_compatibility_spec.rb to lock current FDv1/FDv2 behavior (version numbering, polling semantics, logging, etc.) while the new code lives alongside them.

Reviewed by Cursor Bugbot for commit 73afbaa. Bugbot is set up for automated code reviews on this repo. Configure here.

Adds `LaunchDarkly::Impl::FileData`, the file reading, parsing, merging,
and reloading code that the file-based override source defined by the
OVERRIDE specification needs. That source must tolerate a missing file,
a file that is being written, and a burst of change notifications, and
it must keep the last good data when a reload fails. The module provides:

- Document parsing for JSON and YAML with validation that the document
  and its flags, flagValues, and segments members are objects.
- An ordered merge of several documents with duplicate keys handling of
  fail or ignore, flagValues expansion into off flags that serve the
  value, model deserialization, and per-document entry counts.
- A reloader that serializes reloads, debounces change signals, retains
  the last good result when a reload fails, retries after a bounded
  delay, reports each distinct failure once, and can skip byte-identical
  reloads. A configured file that does not exist can be treated as a
  file with no content.
- A stat-based poller that compares existence, modification time, and
  size, so a file that appears or disappears is a change.
- A watcher that uses rb-inotify on Linux to watch the directory of each
  file without descending into subdirectories, and the listen gem
  elsewhere, so a file that does not exist yet is picked up when it
  appears. It retries when a directory does not exist.

The existing FDv1 and FDv2 file data sources are not changed. They keep
their own implementation and behavior. A new spec pins the behaviors of
those sources that their other specs did not assert, so that they stay
as they are while this code exists beside them.
@kinyoklion
kinyoklion force-pushed the rlamb/overrides-ruby-filedata branch from eb26dd7 to 72d5776 Compare September 28, 2026 20:36
@kinyoklion kinyoklion changed the title feat: Make file data loading reliable and shared feat: Add file loading code for flag overrides Sep 28, 2026
The result is not remembered as applied, on_error receives the exception, the reload is retried, and the worker thread continues.
The inotify watcher tears its watches down when a watched directory is deleted or moved, logs it, retries setup once per second, and signals one change when the watches are in place again.
Compare absolute paths through File.absolute_path, close temp files before deleting them, and backdate modification times instead of sleeping before a write that the pollers must see.
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

value.map { |v| symbolize_keys(v) }
else
value
end

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YAML on key silently disables flags

Medium Severity

YAML.safe_load treats unquoted on as a boolean, and symbolize_keys then stores that field as :true. Full flag definitions never see :on, so FeatureFlag treats them as off. Natural YAML like on: true therefore loads as a disabled flag instead of failing or preserving the on field.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 2e31db3. Configure here.

…pop timeouts

Queue#pop accepts a timeout only from Ruby 3.2 on. The sync examples also wait for the poller thread or the listen call before they act, because the source starts change detection after it yields the initial data.
The watcher set up its watches all or nothing. If any directory of the configured paths did not exist, it logged the problem, watched nothing, and retried once per second until every directory existed. A configured file in a directory that never exists is a valid steady state for the override source, which treats a missing file as contributing nothing, so one permanently missing directory meant that changes to files in the directories that do exist were never detected.

Each directory is now watched on its own. At start, the directories that exist are watched at once and the others are remembered as missing and retried once per second. A retry adds the watches for the directories that exist by then, and the callback runs once when it added at least one, whether or not other directories are still missing. The directories that are still missing, and any watch that could not be set up, are logged together, so that an unchanged situation repeats at debug level as before. When a watched directory is lost, only its watch is removed and it goes back to the missing set; the other directories keep their watches. With rb-inotify, one notifier and one thread take the watches as they are added, and a lost directory's watch is closed. With the listen gem, there is one listener per directory, and stop ends each of them.

The spec that lost a directory as soon as its watches were set up waited for the inotify thread to end as its sign that the loss had been handled. The thread outlives one directory now, so it waits for the warning in the log instead. New specs cover a directory missing at start beside one that exists, a lost directory watched again while another is still missing, and stop while a directory is missing.
@kinyoklion

Copy link
Copy Markdown
Member Author

bugbot review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Fix All in Cursor

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 73afbaa. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant