[Abhishek-Mohan]

Which issue does this PR close?

Part of #4419. Operator renames are left for a separate issue, because they change plan output and test files (see my comment on #4419).

Rationale for this change

#4419 proposes a shared vocabulary for Comet's docs, mainly replacing vague uses of "native" with specific terms. The README and several user guides already use it (#4428, #4650, #4651, #4652), but the rules exist only in the issue, and several pages still use the old wording.

This PR writes the rules down as a style guide and applies them to the remaining pages that #4419 lists. Each page is its own commit, so you can review one commit at a time, or I can split the PR if you'd prefer.

How this PR covers #4419

#4419 asks for	Where
Three-axis vocabulary (pipeline, language, data format)	Style guide, T1-T10
Rule for "native", with the allowed compounds	Style guide, N1-N6
"Vectorized" not used as a synonym for columnar	T10, and gluten_comparison.md
Value prop on README.md	Already done in #4428
Value prop on docs/source/index.md	The landing page's opening line
scala_java_udfs.md	Already done on main
understanding-comet-plans.md	Leftover wording. The three node kinds stay (see below)
plugin_overview.md: "Rust-implemented operators", and how JVM Arrow expressions fit into the pipeline	Wording, plus the new codegen dispatch paragraph
native_shuffle.md and jvm_shuffle.md	Wording. Their titles stay, as #4419 suggests
gluten_comparison.md: the JVM-on-Arrow path as a differentiator	Its existing Codegen Dispatch section already covers this. Wording updated
Migration plan, step 1: the vocabulary first, with no code or plan changes	This PR

Deliberate departures, explained in my comment on #4419:

• Hybrid keeps its shipped meaning.
• Expressions are not plan nodes, so there's no fourth "Arrow-native JVM expression" node kind. Codegen dispatch is described as an expression kind instead.
• Operator renames, and the open questions about them, are left for a separate issue.

What changes are included in this PR?

1. Style guide (docs/source/contributor-guide/style_guide.md), linked from the Reference section of the contributor guide and from AGENTS.md:

• Vocabulary (T1-T10). The three questions from Establish nomenclature style guide and audit operator names for clarity #4419: does Comet handle the work, which language runs it, and what format is the data in. Also covers codegen dispatch, the expression kinds, Arrow IPC, Spark rows, and "vectorized".
• The rule for "native" (N1-N6). Which uses are vague, which are allowed ("native" followed by a component, such as native shuffle or native Parquet reader), and the contributor-guide forms such as "native code".
• Operators, plan names, and expressions (O1-O4). Use the names plan output prints, link to the existing operator reference, and don't describe renames that haven't happened.
• Page conventions (P1-P18). The structure, accuracy, and hygiene checks from Add a docs-review Claude skill to enforce style guide and documentation quality #4420, plus which content is generated, a redirect rule for moved pages, and attribution of third-party content.

Every rule has a stable ID so reviews and the #4420 skill (#7153) can cite it.

Where #4419 and the current code differ, the guide follows the code:

• Hybrid keeps its meaning from expressions.md: an expression with both a Rust path and a codegen-dispatch path.
• Expressions are kept separate from plan nodes.
• "Vectorized" is used for proper names and batch kernels, not as a synonym for columnar.
• "Arrow-native" describes data format. It doesn't claim that no per-row work happens.

2. Page cleanups, one commit per page:

Page	Changes
contributor-guide/plugin_overview.md	Wording, plus factual corrections, listed below
contributor-guide/native_shuffle.md	"Comet native operators" → "Comet operators", because native shuffle accepts any CometPlan child. "native" used as a noun → the component it means. Also: the revert exception for converted shuffle input, and what happens when direct read can't serve a stage
contributor-guide/jvm_shuffle.md	Says which parts run on the JVM and which in Rust. Corrects the spark.comet.shuffle.directRead.enabled description
about/gluten_comparison.md	"native" → "Comet pipeline", "Rust", or "Velox"; "vectorized" → "columnar". Corrects the claim that Comet has no Delta Lake support: there is an experimental, opt-in contrib scan (#5365). Points the two Gluten limitations links, which return 404, at Gluten's v1.7.0 sources
user-guide/latest/understanding-comet-plans.md	Leftover uses after #4650. Adds CometTakeOrderedAndProject as another exception to single-block fusion. Also: the explain section says seven configs and adds the missing spark.comet.explain.planOnly.enabled and spark.comet.explain.rules rows, and the CometCollectLimit row says it runs Rust limit plans unless Spark collects it at the plan root
index.md (landing page)	"the Apache DataFusion native engine" ×4. The opening line now uses the README's Arrow framing. Removes "powerful", and makes the benchmarking link relative

Corrections in plugin_overview.md, which need the closest review:

• Rule registration. CometSparkSessionExtensions also registers post-transition rules, AQE stage optimizer rules on Spark 3.5 and later, and IcebergWriteStrategy.
• Partial stage conversion. An unsupported operator stays in Spark, not the whole plan. After Spark inserts transitions, RevertNativeForTransitionHeavyStages runs first. It can revert a transition-heavy stage when spark.comet.exec.transitionRevert.enabled is set, unless reverting it is unsafe. Then EliminateRedundantTransitions turns the remaining transitions into CometColumnarToRowExec, or into the Rust CometNativeColumnarToRowExec when that opt-in config is set.
• Plan leaves.
  • ScanExec takes Arrow batches from the JVM through the C Stream.
  • A direct shuffle read is a ShuffleScanExec.
  • The Parquet, Iceberg, and CSV scans read their files in Rust.
• Codegen dispatch (new paragraph). Establish nomenclature style guide and audit operator names for clarity #4419 asks this page to explain how JVM code that works on Arrow batches fits into the pipeline. The new paragraph covers expressions routed through the JVM codegen dispatcher: JvmScalarUdfExpr → CometUdfBridge → CometScalaUDFCodegen, over Arrow through JNI. Scala and Java UDFs take the same path, and the operator stays Rust-implemented.
• Block formation. Blocks are formed from consecutive CometNativeExec operators, except that the write operators start a new block for their child. The native shuffle writer takes in a CometNativeExec child.

Config keys, class names, and quoted explain output are unchanged on every page.

Overlap with open PRs:

• docs: add a scan contributor guide and a scan PR review skill #6052 (scan guide) rewrites the Phase 1 scan section of plugin_overview.md, so this PR leaves that section as it is.
• docs: publish the newest release at user-guide/latest and main at user-guide/dev #7112 renames user-guide/latest to user-guide/dev. The guide's links use latest today, and I'll update them if docs: publish the newest release at user-guide/latest and main at user-guide/dev #7112 lands first.
• feat: txt file support #6784 adds a row to the Scans table in understanding-comet-plans.md, which this PR realigns, so the two conflict.
• fix: stop gating columnar shuffle on native-serde checks it never uses #6110, perf: read shuffle blocks directly in native Iceberg writes #6926 and docs: add AI-assisted contributions page to the contributor guide #6734 touch nearby lines in the shuffle guides, the plans page, AGENTS.md, and the contributor index.

I'll rebase whichever lands first.

Not included:

• Operator renames.
• Problems the review turned up that need new content rather than rewording:
  • the out-of-date from_unixtime example;
  • operators missing from the operator reference;
  • the CometBatchScan row;
  • two H1s in plugin_overview.md.
• A link from contributing.md, which I'll add after docs: add AI-assisted contributions page to the contributor guide #6734 merges.

How are these changes tested?

Docs only.

• prettier --check passes on every changed file.
• I built the docs locally with Sphinx. Every page renders, the new page appears under Reference, and every changed link and anchor resolves. The build gave no new warnings.
• I ran the Add a docs-review Claude skill to enforce style guide and documentation quality #4420 skill (docs: add a docs-review skill #7153) on this PR with five models. It found real errors, such as a seventh explain config, conditions missing from the native columnar-to-row and direct-read sentences, and the wrong sources in the Generated Content table. Those are fixed here.
• I checked each changed claim against main. Five independent AI reviews (GPT-6 Astra, Grok 4.7, Gemini 4 Argon, Claude Opus 5.5, GPT-6.1 Sol) checked them against the code. Over two rounds, each finding was either fixed or listed under Not included.

Council review

Each round, seven models ran dry runs: they followed the style guide literally as an agent would, rewrote pages with it, and ran the review skill on this branch. Findings were checked against the code before fixing. Models already at 10/10 on both PRs were not rerun in round 4.

Rounds: 4

Delegate	Round 1	Round 2	Round 3	Round 4
Claude Opus 5.5	8/10	8/10	9/10	10/10
Gemini 4 Argon	9/10	10/10	10/10	not run
Grok 4.7	7/10	8/10	9/10	10/10
GPT-6.1 Sol	9/10	9/10	10/10	not run
GPT-6 Astra	9/10	9/10	10/10	10/10
Claude Sonnet 5.5	8/10	8/10	8/10	9/10
GPT-6 Luna	8/10	8/10	9/10	10/10

AI assistance: OpenCode helped draft the guide and the edits and checked them against the code. The reviews above were run with OpenCode subagents.

[sunchao]

Reviewed all seven commits and nine changed files at 56396e38e2dd066e01256201b6b3dc4772dec970 against base 2b1c6a3938cbe1c4b2d0aa6383e64360569f877a. The full PR diff contains documentation changes only. I verified four new P2 findings. The PR has no existing reviews, conversation comments, or review threads.

Summary

• Prior state and problem
  The guides used “native” for both work handled by Comet and work executed in Rust. The plugin overview also said a query stage could convert only as a whole.
• Design approach
  The style guide distinguishes which engine handles an operation, which language executes it, and which format carries its data. The revised pages apply those distinctions to plans, shuffles, and expression dispatch.
• Correctness
  A non-adaptive final aggregate can read a Comet shuffle through ScanExec with direct reads enabled. Separately, a stage can revert to Spark without appearing in the spark.comet.explain.rules log. Both cases contradict new documentation.
• Compatibility analysis
  Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0 sources confirm the documented transition ordering and collect-limit behavior. The AQE stage optimizer hook is present from Spark 3.5. Celeborn adds a child-class restriction missing from the revised shuffle eligibility rule.
• Key design decisions
  The guide preserves printed operator names and the shipped Hybrid expression classification. A JVM-dispatched expression can remain inside a Rust-implemented operator.
• Implementation sketch
  AGENTS.md and the contributor index link the new guide. The plugin overview, both shuffle guides, plan reference, Gluten comparison, and landing page adopt its terminology.
• Performance
  The patch adds no query execution overhead. Its JVM shuffle diagram does overstate the cost for a row-forwarding bridge. The probe consumed two rows with zero Arrow batches produced by the bridge.
• Design
  Linking to the existing operator reference and identifying the inputs to GenerateDocs gives contributors one place to verify plan names and generated content.
• Abstraction & complexity
  CometPlan identifies membership in the Comet pipeline. Predicting shuffle behavior also requires the manager’s selection rule and the child’s doExecute() implementation.
• Behavioral changes worth calling out
  Compared with branch-1.2, the PR changes contributor and user guidance. It documents partial-stage conversion, the separate final Rust plan for multi-partition top-K, and the experimental opt-in Delta scan.
• Suggested improvements
  Qualify direct reads by the serialized ShuffleScan leaf, distinguish Celeborn’s child requirement, show the JVM bridge’s row path, and narrow the rule-logging guarantee.

Routed skills. review-comet-pr, review-comet-shuffle-pr, review-comet-ffi-pr, review-comet-expression-pr, review-comet-memory-pr, and review-comet-iceberg-write-pr.

Exact-head CI. Label pull requests passed. Comet CI, CodeQL, and Check PR Title each reported action_required with zero jobs executed. Comet CI run

Validation. Focused Spark 4.1.3 probes exercised shuffle serialization, manager selection, JVM row consumption, and rule logging. Sphinx 7.4.7 built the head and merge base with the same 80 warnings. All eight changed documentation pages rendered, and all 52 local article links checked resolved. Validation excluded full Spark/native suites, live Celeborn I/O, release-document assembly, and Mermaid rendering.

[Abhishek-Mohan]

Note

🤖 Claude Opus 5.5 responding on behalf of @Abhishek-Mohan

@sunchao all four findings are fixed in 3c982c8, and main is merged in (459c6fe). Ready for another look.

[andygrove]

Changing this link format may affect how the site gets published?

[Abhishek-Mohan]

Note

🤖 Claude Opus 5.5 responding on behalf of @Abhishek-Mohan

No effect on publishing. Sphinx resolves /user-guide/latest/compatibility/index.md to the relative link ../user-guide/latest/compatibility/index.html in the built page, the same form main already uses for four other links on this page. A base-vs-head docs build shows no new warnings, and the build now warns if the target page moves.

[andygrove]

This looks great @Abhishek-Mohan. I've been reviewing the changes manually. Will finish reviewing tomorrow!

[comphead]

I checked the rewritten paragraphs against the code at this head, and the transition rules, the write-block exception, the explain configs and the renamed operators all match. Three leftovers outside the diff, so no inline comments. In plugin_overview.md the Shuffle Reads paragraph (lines 202 to 204) still says the compressed blocks go to the plan whenever a native plan consumes the shuffle and spark.comet.shuffle.directRead.enabled is on, while the rewritten paragraph at lines 156 to 163 now requires a ShuffleScan leaf, which needs an AQE shuffle query stage. Could it use the same condition, as the jvm_shuffle.md sentence already does? The overview figure comet-overview.svg still says "Apache DataFusion native engine" and "Native plan", which the new alt text and the guide's own "Before" example move away from. Line 81 of index.md still says "seamlessly", which P10 lists as a marketing word.

[Abhishek-Mohan]

Note

🤖 Claude Opus 5.5 responding on behalf of @Abhishek-Mohan

@comphead all three fixed in 60af6cb: the Shuffle Reads paragraph uses the ShuffleScan condition, comet-overview.svg and index.md no longer say native or seamlessly. bc3ffc2 corrects two of the figure labels.