[GLUTEN-12597][CORE] Migrate TopNRel to Expression count + FetchMode (Substrait 0.98) - #12728
Open
nielspardon wants to merge 1 commit into
Open
[GLUTEN-12597][CORE] Migrate TopNRel to Expression count + FetchMode (Substrait 0.98)#12728nielspardon wants to merge 1 commit into
nielspardon wants to merge 1 commit into
Conversation
…(Substrait 0.98) Substrait 0.98 redesigned TopNRel: the scalar `int64 n = 3` became `Expression count = 5`, `sorts` moved to field 3, and it gained `Expression offset = 4` and `FetchMode mode = 6`. This migrates Gluten's vendored proto, the JVM producer, and the Velox consumer/validator in lockstep. Gluten's only TopNRel producer (TopNTransformer, Spark's TakeOrderedAndProject) always supplies a literal Long limit with no offset and no ties, so the producer wraps the limit into an i64-literal Expression and sets FETCH_MODE_ROWS_ONLY explicitly (FETCH_MODE_UNSPECIFIED is the proto3 default and not a valid producer choice). TopNRel stays at Rel-oneof field 19, so there is no oneof renumbering. - Proto: replace `int64 n=3`/`sorts=4`/`advanced_extension=10` with the 0.98 body (`sorts=3`, `offset=4`, `count=5`, `mode=6`, `advanced_extension=7`) and add the top-level FetchMode enum (vendored verbatim from 0.98). - Producer: TopNNode wraps the limit via ExpressionBuilder.makeLongLiteral, sets FETCH_MODE_ROWS_ONLY, and now attaches RelCommon (it was built but never set). - Velox: SubstraitToVeloxPlan and SubstraitToVeloxPlanValidator extract the count via a new SubstraitParser::getRowCount helper (rejects a count that is not a positive i64 literal in [1, INT32_MAX]) and reject an offset or a mode other than ROWS_ONLY; the validator also rejects an empty sort list, which core::TopNNode asserts against. - Add TopNRelProtoSuite pinning the producer contract and the 0.98 field numbers. Part of apache#12597 Generated-by: Claude Code (Claude Opus 4.8)
|
Run Gluten Clickhouse CI on x86 |
nielspardon
marked this pull request as ready for review
August 10, 2026 11:29
19 tasks
Contributor
|
@nielspardon LGTM. Thanks for your work. Could you help to remove the |
Contributor
Author
Do you prefer to update the docs file on each PR or should do we one consolidation round at the end? |
Contributor
@nielspardon It is ok for me to combine these changes into single PR at the end. |
JkSelf
approved these changes
Aug 12, 2026
JkSelf
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Thanks for your work
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Substrait 0.98 redesigned
TopNRel: the scalarint64 n = 3becameExpression count = 5,sortsmoved to field 3, and it gainedExpression offset = 4andFetchMode mode = 6. This migrates Gluten's vendored proto, the JVM producer, and the Velox consumer/validator in lockstep, as one step of the Substraitv0.23.0→0.98.0proto rebase (#12597).Gluten's only
TopNRelproducer (TopNTransformer, Spark'sTakeOrderedAndProject) always supplies a literalLonglimit with no offset and no ties, so the producer wraps the limit into an i64-literalExpressionand setsFETCH_MODE_ROWS_ONLYexplicitly (FETCH_MODE_UNSPECIFIEDis merely the proto3 default and not a valid producer choice).TopNRelstays at Rel-oneof field 19, so there is no oneof renumbering.int64 n = 3/sorts = 4/advanced_extension = 10with the 0.98 body (sorts = 3,offset = 4,count = 5,mode = 6,advanced_extension = 7) and add the top-levelFetchModeenum (vendored verbatim from 0.98).TopNNodewraps the limit viaExpressionBuilder.makeLongLiteral, setsFETCH_MODE_ROWS_ONLY, and now attachesRelCommon(previously it was built but never set).SubstraitToVeloxPlanandSubstraitToVeloxPlanValidatorextract the count via a newSubstraitParser::getRowCounthelper — which rejects a count that is not a positive i64 literal in[1, INT32_MAX]— and reject anoffsetor anymodeother thanFETCH_MODE_ROWS_ONLY. The converter mirrors the validator so a plan reaching it with native validation disabled (or via the JSON-plan/benchmark paths) fails loudly instead of silently dropping OFFSET / WITH TIES semantics. The validator also rejects an empty sort list, whichcore::TopNNodeasserts against.ClickHouse has no
TopNRelparser, and the Velox → Substrait reverse converter maps a VeloxTopNNodeto aFetchRel(not aTopNRel), so neither is affected.How was this patch tested?
TopNRelProtoSuitepins the producer contract (the count lands as an i64-literal expression,FETCH_MODE_ROWS_ONLY, no offset,RelCommondirect) and asserts the 0.98 field numbers on the descriptor (a same-schema round trip cannot catch a renumber).protocdup-field check;gluten-substraitbuilds and the new suite passes (mvn -Pspark-3.5); the Velox native backend compiles and links (consumer, validator, and the newgetRowCounthelper);clang-format(v15) clean on all changed C++ files.TakeOrderedAndProjectsuites exercise the consumer extraction on Velox.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
🤖 Generated with AI