Skip to content

feat: Durable materialization-interval history for every registry backend - #377

Merged
piket merged 1 commit into
masterfrom
feat/materialization-records
Aug 21, 2026
Merged

feat: Durable materialization-interval history for every registry backend#377
piket merged 1 commit into
masterfrom
feat/materialization-records

Conversation

@piket

@piket piket commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Adds a full, uncapped materialization-interval history, retained in the registry itself (not delegated to an external system), compatible with every registry backend -- file-based, SQL, Snowflake, HTTP, and gRPC -- plus a configurable cap on the existing rolling-window FeatureView.materialization_intervals list.

Cap configurability:

  • RegistryConfig.materialization_intervals_max_len (default 10, matching the prior hardcoded MATERIALIZATION_INTERVALS_MAX_LEN constant) lets deployments tune the cap via config instead of a code change -- e.g. start permissive and lower it in steps. Wired into SqlRegistry.apply_materialization and its _apply_object hydration path.

Durable interval history:

  • New MaterializationIntervalHistoryEntry proto (protos/feast/core/FeatureView.proto) and a new top-level Registry.materialization_interval_history field for the file-based backend's "separate node" (protos/feast/core/Registry.proto).
  • FeatureView.add_materialization_interval/update_materialization_intervals/ _cap_materialization_intervals now return the dropped intervals instead of None, so callers can archive them -- purely additive, existing callers ignoring the return value are unaffected.
  • New BaseRegistry.get_materialization_interval_history abstract method, implemented for real by every concrete backend (not duck-typed/backend-gated):
    • SqlRegistry/SqlFallbackRegistry: new materialization_interval_history table (auto-created via the existing metadata.create_all(), inherited by SqlFallbackRegistry for free), idempotent inserts keyed on (feature_view_name, project, start_time, end_time) so archiving both a newly-added interval and, separately, a later-dropped one never duplicates a row.
    • File-based Registry: same idempotent semantics against the new top-level proto field.
    • SnowflakeRegistry: new MATERIALIZATION_INTERVAL_HISTORY table via snowflake_table_creation.sql, bespoke inserts (bypassing the upsert-by-key _apply_object/_get_object helpers, which don't fit a genuinely multi-row-per-key table).
    • RemoteRegistry/gRPC: new GetMaterializationIntervalHistory RPC (RegistryServer.proto + registry_server.py handler + client method), following the ListEntities pagination template. The existing ApplyMaterialization RPC needed zero changes -- it already delegates straight to the proxied registry's apply_materialization, so once every backend archives internally, gRPC callers get correct behavior for free.
    • HttpRegistry: now calls a dedicated endpoint instead of the old client-side-append + whole-object PUT (see below).

Also fixes two real, pre-existing bugs surfaced while designing this:

  1. File-based Registry and SnowflakeRegistry never enforced materialization_intervals_max_len at all (silent divergence from SqlRegistry) -- both now route through add_materialization_interval/ update_materialization_intervals like every other backend.
  2. HttpRegistry.apply_materialization appended the new interval client-side and PUT the whole FeatureView object to the generic feature_views endpoint, bypassing the server's cap/archive logic entirely and risking a silent clobber of previously-stored intervals if the in-memory object wasn't fully hydrated. It now calls a dedicated materialization endpoint (paired with a matching addition in eg-feature-store-registry) so the server fetches the canonical stored feature view itself.

New tests across feature_view.py, SqlRegistry, the file-based Registry, SnowflakeRegistry (mocked -- no live Snowflake available), and the new gRPC RPC/RemoteRegistry client. Full sdk/python/tests/unit suite: 1183 passed, 22 skipped, the same 9 pre-existing/unrelated failures as before (sqlite disk I/O + one docling error), no new failures.

What this PR does / why we need it:

Which issue(s) this PR fixes:

Misc

@piket
piket force-pushed the feat/materialization-records branch from a082bff to 78ab839 Compare August 20, 2026 16:12
self.materialization_intervals.append((start_date, end_date))
return self._cap_materialization_intervals(max_intervals)

def _cap_materialization_intervals(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think _cap_materialization_intervals silently disables capping when max_intervals == 0. Is that intended?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was leftover from a previous iteration, removed.

)

Index(
"idx_materialization_interval_history_fv_project",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like two concurrent calls (e.g. a retry racing the original) can both pass the check before either commits, producing duplicate history rows. Could you please check this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some checks to avoid this.

…kend

Adds a full, uncapped materialization-interval history, retained in the
registry itself (not delegated to an external system), compatible with
every registry backend -- file-based, SQL, Snowflake, HTTP, and gRPC --
plus a configurable cap on the existing rolling-window
FeatureView.materialization_intervals list.

Cap configurability:
- RegistryConfig.materialization_intervals_max_len (default 10, matching
  the prior hardcoded MATERIALIZATION_INTERVALS_MAX_LEN constant) lets
  deployments tune the cap via config instead of a code change -- e.g.
  start permissive and lower it in steps. Wired into
  SqlRegistry.apply_materialization and its _apply_object hydration path.

Durable interval history:
- New MaterializationIntervalHistoryEntry proto (protos/feast/core/FeatureView.proto)
  and a new top-level Registry.materialization_interval_history field for
  the file-based backend's "separate node" (protos/feast/core/Registry.proto).
- FeatureView.add_materialization_interval/update_materialization_intervals/
  _cap_materialization_intervals now return the dropped intervals instead of
  None, so callers can archive them -- purely additive, existing callers
  ignoring the return value are unaffected.
- New BaseRegistry.get_materialization_interval_history abstract method,
  implemented for real by every concrete backend (not duck-typed/backend-gated):
  - SqlRegistry/SqlFallbackRegistry: new materialization_interval_history
    table (auto-created via the existing metadata.create_all(), inherited
    by SqlFallbackRegistry for free), idempotent inserts keyed on
    (feature_view_name, project, start_time, end_time) so archiving both a
    newly-added interval and, separately, a later-dropped one never
    duplicates a row.
  - File-based Registry: same idempotent semantics against the new
    top-level proto field.
  - SnowflakeRegistry: new MATERIALIZATION_INTERVAL_HISTORY table via
    snowflake_table_creation.sql, bespoke inserts (bypassing the
    upsert-by-key _apply_object/_get_object helpers, which don't fit a
    genuinely multi-row-per-key table).
  - RemoteRegistry/gRPC: new GetMaterializationIntervalHistory RPC
    (RegistryServer.proto + registry_server.py handler + client method),
    following the ListEntities pagination template. The existing
    ApplyMaterialization RPC needed zero changes -- it already delegates
    straight to the proxied registry's apply_materialization, so once every
    backend archives internally, gRPC callers get correct behavior for free.
  - HttpRegistry: now calls a dedicated endpoint instead of the old
    client-side-append + whole-object PUT (see below).

Also fixes two real, pre-existing bugs surfaced while designing this:
1. File-based Registry and SnowflakeRegistry never enforced
   materialization_intervals_max_len at all (silent divergence from
   SqlRegistry) -- both now route through add_materialization_interval/
   update_materialization_intervals like every other backend.
2. HttpRegistry.apply_materialization appended the new interval
   client-side and PUT the whole FeatureView object to the generic
   feature_views endpoint, bypassing the server's cap/archive logic
   entirely and risking a silent clobber of previously-stored intervals if
   the in-memory object wasn't fully hydrated. It now calls a dedicated
   materialization endpoint (paired with a matching addition in
   eg-feature-store-registry) so the server fetches the canonical stored
   feature view itself.

New tests across feature_view.py, SqlRegistry, the file-based Registry,
SnowflakeRegistry (mocked -- no live Snowflake available), and the new
gRPC RPC/RemoteRegistry client. Full sdk/python/tests/unit suite: 1183
passed, 22 skipped, the same 9 pre-existing/unrelated failures as before
(sqlite disk I/O + one docling error), no new failures.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@piket
piket force-pushed the feat/materialization-records branch from 78ab839 to 0bd67ae Compare August 21, 2026 20:17

@vanitabhagwat vanitabhagwat left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit tests are failing. Rest looks good to me.

@piket
piket merged commit f145e58 into master Aug 21, 2026
38 of 39 checks passed
@piket
piket deleted the feat/materialization-records branch August 21, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants