Skip to content

[GLUTEN-12597][CORE] Migrate TopNRel to Expression count + FetchMode (Substrait 0.98) - #12728

Open
nielspardon wants to merge 1 commit into
apache:mainfrom
nielspardon:feat/substrait-0.98-topnrel
Open

[GLUTEN-12597][CORE] Migrate TopNRel to Expression count + FetchMode (Substrait 0.98)#12728
nielspardon wants to merge 1 commit into
apache:mainfrom
nielspardon:feat/substrait-0.98-topnrel

Conversation

@nielspardon

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Substrait 0.98 redesigned TopNRel: the scalar int64 n = 3 became Expression count = 5, sorts moved to field 3, and it gained Expression offset = 4 and FetchMode mode = 6. This migrates Gluten's vendored proto, the JVM producer, and the Velox consumer/validator in lockstep, as one step of the Substrait v0.23.00.98.0 proto rebase (#12597).

Gluten's only TopNRel producer (TopNTransformer, Spark's TakeOrderedAndProject) always supplies a literal Long limit with no offset and no ties, so the producer wraps the limit into an i64-literal Expression and sets FETCH_MODE_ROWS_ONLY explicitly (FETCH_MODE_UNSPECIFIED is merely the proto3 default and not a valid producer choice). TopNRel stays at Rel-oneof field 19, so there is no oneof renumbering.

  • Proto: replace int64 n = 3 / sorts = 4 / advanced_extension = 10 with the 0.98 body (sorts = 3, offset = 4, count = 5, mode = 6, advanced_extension = 7) and add the top-level FetchMode enum (vendored verbatim from 0.98).
  • Producer: TopNNode wraps the limit via ExpressionBuilder.makeLongLiteral, sets FETCH_MODE_ROWS_ONLY, and now attaches RelCommon (previously it was built but never set).
  • Velox: SubstraitToVeloxPlan and SubstraitToVeloxPlanValidator extract the count via a new SubstraitParser::getRowCount helper — which rejects a count that is not a positive i64 literal in [1, INT32_MAX] — and reject an offset or any mode other than FETCH_MODE_ROWS_ONLY. The converter mirrors the validator so a plan reaching it with native validation disabled (or via the JSON-plan/benchmark paths) fails loudly instead of silently dropping OFFSET / WITH TIES semantics. The validator also rejects an empty sort list, which core::TopNNode asserts against.

ClickHouse has no TopNRel parser, and the Velox → Substrait reverse converter maps a Velox TopNNode to a FetchRel (not a TopNRel), so neither is affected.

How was this patch tested?

  • New TopNRelProtoSuite pins the producer contract (the count lands as an i64-literal expression, FETCH_MODE_ROWS_ONLY, no offset, RelCommon direct) and asserts the 0.98 field numbers on the descriptor (a same-schema round trip cannot catch a renumber).
  • Locally verified: protoc dup-field check; gluten-substrait builds and the new suite passes (mvn -Pspark-3.5); the Velox native backend compiles and links (consumer, validator, and the new getRowCount helper); clang-format (v15) clean on all changed C++ files.
  • Existing end-to-end TakeOrderedAndProject suites exercise the consumer extraction on Velox.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

🤖 Generated with AI

…(Substrait 0.98)

Substrait 0.98 redesigned TopNRel: the scalar `int64 n = 3` became
`Expression count = 5`, `sorts` moved to field 3, and it gained
`Expression offset = 4` and `FetchMode mode = 6`. This migrates Gluten's
vendored proto, the JVM producer, and the Velox consumer/validator in
lockstep.

Gluten's only TopNRel producer (TopNTransformer, Spark's
TakeOrderedAndProject) always supplies a literal Long limit with no offset
and no ties, so the producer wraps the limit into an i64-literal Expression
and sets FETCH_MODE_ROWS_ONLY explicitly (FETCH_MODE_UNSPECIFIED is the
proto3 default and not a valid producer choice). TopNRel stays at Rel-oneof
field 19, so there is no oneof renumbering.

- Proto: replace `int64 n=3`/`sorts=4`/`advanced_extension=10` with the 0.98
  body (`sorts=3`, `offset=4`, `count=5`, `mode=6`, `advanced_extension=7`)
  and add the top-level FetchMode enum (vendored verbatim from 0.98).
- Producer: TopNNode wraps the limit via ExpressionBuilder.makeLongLiteral,
  sets FETCH_MODE_ROWS_ONLY, and now attaches RelCommon (it was built but
  never set).
- Velox: SubstraitToVeloxPlan and SubstraitToVeloxPlanValidator extract the
  count via a new SubstraitParser::getRowCount helper (rejects a count that
  is not a positive i64 literal in [1, INT32_MAX]) and reject an offset or a
  mode other than ROWS_ONLY; the validator also rejects an empty sort list,
  which core::TopNNode asserts against.
- Add TopNRelProtoSuite pinning the producer contract and the 0.98 field
  numbers.

Part of apache#12597

Generated-by: Claude Code (Claude Opus 4.8)
@github-actions github-actions Bot added CORE works for Gluten Core VELOX labels Aug 7, 2026
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@nielspardon
nielspardon marked this pull request as ready for review August 10, 2026 11:29
@JkSelf

JkSelf commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

@nielspardon LGTM. Thanks for your work. Could you help to remove the TopNRel changes in SubstraitModifications.md ?

@nielspardon

Copy link
Copy Markdown
Contributor Author

Could you help to remove the TopNRel changes in SubstraitModifications.md ?

Do you prefer to update the docs file on each PR or should do we one consolidation round at the end?

@JkSelf

JkSelf commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Could you help to remove the TopNRel changes in SubstraitModifications.md ?

Do you prefer to update the docs file on each PR or should do we one consolidation round at the end?

@nielspardon It is ok for me to combine these changes into single PR at the end.

@JkSelf JkSelf left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for your work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants