fix: Keep a zero watermark delay threshold on a Kafka source - #6874
Open
Rodrigo-Palma wants to merge 1 commit into
Open
Rodrigo-Palma wants to merge 1 commit into
Rodrigo-Palma wants to merge 1 commit into
Conversation
KafkaOptions.__init__ stored 'watermark_delay_threshold or None', and
timedelta(0) is falsy, so a delay of zero was read as 'not given':
KafkaOptions(..., watermark_delay_threshold=timedelta(0))
.watermark_delay_threshold -> None
to_proto().HasField(...) -> False
The value never reached the proto, so it was lost on apply and on every
registry round-trip, while timedelta(seconds=30) survived.
Both from_proto sites carried a branch mapping an explicit zero Duration
back to timedelta(days=0). That branch could not return anything: the
constructor it hands the value to dropped it. Feeding a proto that does
carry a zero Duration returned None.
KafkaSource.from_proto also tested the Duration message for truth
instead of presence. A protobuf message is always truthy, so an absent
delay entered the branch and came out as timedelta(days=0); only the
'or None' above hid it. KafkaOptions.from_proto, its sibling in this
file, already uses HasField, which is what both now use.
An absent delay stays None; a zero delay stays zero.
Signed-off-by: Rodrigo-Palma <email.rodrigopalma@gmail.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6874 +/- ##
=======================================
Coverage 47.66% 47.66%
=======================================
Files 422 422
Lines 52396 52396
Branches 7606 7606
=======================================
Hits 24973 24973
Misses 25632 25632
Partials 1791 1791
... and 1 file with indirect coverage changes Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KafkaOptions.__init__storedwatermark_delay_threshold or None(data_source.py:50), andtimedelta(0)is falsy, so a delay of zero was read as "not given":The value never reached the proto, so it was lost on
apply()and on every registry round-trip, whiletimedelta(seconds=30)came back intact. Zero is a meaningful setting for a stream source: it asks for no watermark delay at all, which is not the same as leaving the option out.Two things were keeping this hidden.
The branch that was meant to preserve a zero could not return one. Both
from_protosites mapped an explicit zeroDurationback totimedelta(days=0), and then handed it to the constructor above, which dropped it. FeedingKafkaOptions.from_protoa proto that does carry a zeroDurationreturnedNone.KafkaSource.from_prototested the message for truth instead of for presence (data_source.py:555). A protobuf message object is always truthy, so an absent delay entered the branch too and came out astimedelta(days=0); only theor Nonein the constructor hid that. Fixing the constructor alone would have turned this into a visible regression, which is why both are in this PR.KafkaOptions.from_proto, its sibling in the same file, already reads the field withHasField, and both now do.After the change: an absent delay stays
Noneand is not written to the proto, a zero delay staystimedelta(0)and survives a round-trip, and a non-zero delay is unchanged.Test run:
sdk/python/tests/unit/test_data_sources.py, 14 passed; the new test fails onmasterwithassert None == datetime.timedelta(0).ruff checkandruff format --checkclean on both files with the pinned 0.16.4. (test_stream_feature_view.pyfails 3 tests here for a missingpyspark, the same way onmaster.)