Skip to content

fix: Keep a zero watermark delay threshold on a Kafka source - #6874

Open
Rodrigo-Palma wants to merge 1 commit into
feast-dev:masterfrom
Rodrigo-Palma:fix/kafka-zero-watermark-delay
Open

Rodrigo-Palma wants to merge 1 commit into
feast-dev:masterfrom
Rodrigo-Palma:fix/kafka-zero-watermark-delay

Conversation

@Rodrigo-Palma

Copy link
Copy Markdown
Contributor

KafkaOptions.__init__ stored watermark_delay_threshold or None (data_source.py:50), and timedelta(0) is falsy, so a delay of zero was read as "not given":

from datetime import timedelta
from feast.data_source import KafkaOptions

o = KafkaOptions("localhost:9092", fmt, "t", watermark_delay_threshold=timedelta(0))
o.watermark_delay_threshold                             # None
o.to_proto().HasField("watermark_delay_threshold")      # False

The value never reached the proto, so it was lost on apply() and on every registry round-trip, while timedelta(seconds=30) came back intact. Zero is a meaningful setting for a stream source: it asks for no watermark delay at all, which is not the same as leaving the option out.

Two things were keeping this hidden.

The branch that was meant to preserve a zero could not return one. Both from_proto sites mapped an explicit zero Duration back to timedelta(days=0), and then handed it to the constructor above, which dropped it. Feeding KafkaOptions.from_proto a proto that does carry a zero Duration returned None.

KafkaSource.from_proto tested the message for truth instead of for presence (data_source.py:555). A protobuf message object is always truthy, so an absent delay entered the branch too and came out as timedelta(days=0); only the or None in the constructor hid that. Fixing the constructor alone would have turned this into a visible regression, which is why both are in this PR. KafkaOptions.from_proto, its sibling in the same file, already reads the field with HasField, and both now do.

After the change: an absent delay stays None and is not written to the proto, a zero delay stays timedelta(0) and survives a round-trip, and a non-zero delay is unchanged.

Test run: sdk/python/tests/unit/test_data_sources.py, 14 passed; the new test fails on master with assert None == datetime.timedelta(0). ruff check and ruff format --check clean on both files with the pinned 0.16.4. (test_stream_feature_view.py fails 3 tests here for a missing pyspark, the same way on master.)

KafkaOptions.__init__ stored 'watermark_delay_threshold or None', and
timedelta(0) is falsy, so a delay of zero was read as 'not given':

    KafkaOptions(..., watermark_delay_threshold=timedelta(0))
    .watermark_delay_threshold            -> None
    to_proto().HasField(...)              -> False

The value never reached the proto, so it was lost on apply and on every
registry round-trip, while timedelta(seconds=30) survived.

Both from_proto sites carried a branch mapping an explicit zero Duration
back to timedelta(days=0). That branch could not return anything: the
constructor it hands the value to dropped it. Feeding a proto that does
carry a zero Duration returned None.

KafkaSource.from_proto also tested the Duration message for truth
instead of presence. A protobuf message is always truthy, so an absent
delay entered the branch and came out as timedelta(days=0); only the
'or None' above hid it. KafkaOptions.from_proto, its sibling in this
file, already uses HasField, which is what both now use.

An absent delay stays None; a zero delay stays zero.

Signed-off-by: Rodrigo-Palma <email.rodrigopalma@gmail.com>
@Rodrigo-Palma
Rodrigo-Palma requested a review from a team as a code owner September 26, 2026 21:27
@codecov-commenter

codecov-commenter commented Sep 26, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 47.66%. Comparing base (2000379) to head (25bcc6e).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #6874   +/-   ##
=======================================
  Coverage   47.66%   47.66%           
=======================================
  Files         422      422           
  Lines       52396    52396           
  Branches     7606     7606           
=======================================
  Hits        24973    24973           
  Misses      25632    25632           
  Partials     1791     1791           
Flag Coverage Δ
go-feature-server 30.58% <ø> (ø)
python-unit 49.00% <100.00%> (ø)
Files with missing lines Coverage Δ
sdk/python/feast/data_source.py 81.43% <100.00%> (+0.32%) ⬆️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2000379...25bcc6e. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants