diff --git a/.ai/skills/review-comet-docs/SKILL.md b/.ai/skills/review-comet-docs/SKILL.md new file mode 100644 index 00000000000..95c0b12688e --- /dev/null +++ b/.ai/skills/review-comet-docs/SKILL.md @@ -0,0 +1,424 @@ +--- +name: review-comet-docs +description: Use when reviewing the documentation in a DataFusion Comet pull request or branch, or when auditing an existing page or directory under docs/source or README.md. Checks wording against the documentation style guide and checks that config keys, operator names, links, images and generated content match the code. Advisory only, it never edits files or posts comments. Load alongside review-comet-pr when a PR changes docs. +argument-hint: +--- + + + +Review Comet docs: $ARGUMENTS + +Pick the mode from the argument: + +- **PR mode**: a number. Review the docs a pull request changes. +- **Audit mode**: a file or directory that exists in the working tree. Review it as it is now. +- **Branch mode**: a branch, tag, or commit that `git rev-parse --verify` accepts. Review what it + changes compared with `main`, for example before opening a PR. Otherwise, ask which one is meant. + +In PR mode, **REQUIRED BACKGROUND:** use `review-comet-pr` for its steps 2 to 4 (PR metadata, +existing comments, and the diff) and its CI check. This skill only covers documentation. + +This skill gives advice to a human. Never edit files, commit, push, or post comments. + +**Shell notes.** Run every command from the repository root. Each tool call may start a new shell, +so save the variables you set in a state file named for this run. Use a different name for each run +and mode, and keep it the same in every call of that run: + +```bash +STATE="$(git rev-parse --git-dir)/review-comet-docs-$(printf %s "" | tr / -).env" +{ printf 'R=%q\n' "$R"; printf 'HEAD_SHA=%q\n' "$HEAD_SHA"; printf 'BASE=%q\n' "$BASE"; + printf 'TARGET=%q\n' "$TARGET"; printf 'PATHS=%q\n' "$PATHS"; printf 'ADDED=%q\n' "$ADDED"; } \ + > "$STATE" +``` + +Start every later call by loading it: + +```bash +STATE="$(git rev-parse --git-dir)/review-comet-docs-$(printf %s "" | tr / -).env"; . "$STATE" +``` + +In zsh, `$HEAD_SHA:spark/...` is read as a history modifier, so always write `"${HEAD_SHA}:"`. +Save the state again after step 3 sets `PATHS` and step 4 sets `ADDED`. Stop and report the review as +incomplete if a command in steps 1 to 4 reports an error. A search that finds nothing, which makes +`git grep` exit with status 1, is not an error. Where a step gives a fallback, such as the guide +lookup in step 2 or the docs build in step 4, use it and keep going. + +## 1. Select the Revisions + +Find the remote that points at `apache/datafusion-comet`, because its name differs between clones: + +```bash +R=$(git remote -v | awk '/apache\/datafusion-comet(\.git)? \(fetch\)/ {print $1; exit}') +``` + +If `R` is empty, use `https://github.com/apache/datafusion-comet.git` in its place. + +**PR mode.** Read the PR's base branch and head commit, then fetch both without touching the working +tree. The leading `+` lets the fetch replace a ref after a force push. + +```bash +gh pr view --repo apache/datafusion-comet --json baseRefName,headRefOid +git fetch "$R" "+refs/heads/:refs/review/base-" "+pull//head:refs/review/pr-" +TARGET=refs/review/base-; HEAD_SHA=$(git rev-parse refs/review/pr-) +``` + +Check that `HEAD_SHA` equals `headRefOid`. If it does not, the PR changed while you fetched it: run +both commands again, at most twice. From here on, read everything at `HEAD_SHA`, not at the ref, +which a later fetch can move. + +**Branch mode.** Fetch `main` the same way. `TARGET=refs/review/main` and +`HEAD_SHA=$(git rev-parse --verify )`: + +```bash +git fetch "$R" "+refs/heads/main:refs/review/main" +``` + +**Both modes.** Find the merge base. It fails in a shallow clone, so deepen the history and try again, +or report the comparison as incomplete: + +```bash +BASE=$(git merge-base "$TARGET" "$HEAD_SHA") || + { git fetch --deepen=1000 "$R" && BASE=$(git merge-base "$TARGET" "$HEAD_SHA"); } || + { echo "no merge base: report the comparison as incomplete" >&2; false; } +``` + +If `git rev-list --count "$BASE..$TARGET"` is not zero, the target branch has moved since the change +branched. Check the claims against `$TARGET` too, and report what has changed there. + +**Audit mode.** There is no base. List the files, including new files that are not committed yet, and +save the list as `PATHS`, as step 3 does in the other modes. The mechanical checks run on every file +in it; only the judgment checks in step 5 are capped: + +```bash +PATHS="${STATE%.env}.paths" +git ls-files --cached --others --exclude-standard -- > "$PATHS" +``` + +## 2. Read the Style Guide + +Read `docs/source/contributor-guide/style_guide.md` in full on every run, from the revision under +review: `git show "${HEAD_SHA}:docs/source/contributor-guide/style_guide.md"` in PR and branch mode, +or the working tree in audit mode. If it does not exist there, as for a change made before the guide +was added, read it from `$TARGET`. If it does not exist there either, read the working tree's copy and say in +the report which copy you used. Stop and ask the user for a revision only when there is no copy at +all. + +The guide is the rulebook, and its first section says what it covers. Cite its rule IDs (`T3`, `N1`, +`P16`, and so on), and do not restate or reinterpret its rules here. If the guide and this skill +disagree, the guide wins. + +## 3. Select the Files + +In PR and branch mode, list the changed files from Git, which is complete even for large PRs: + +```bash +git -c core.quotePath=false diff --name-only "$BASE" "$HEAD_SHA" +``` + +From that list, `` below means: + +- `.md` and `.rst` files under `docs/source`, and `README.md`; +- files that change `.doc(...)` strings in `CometConf.scala` or in contrib config classes; +- files that change doc comments (Scaladoc, Javadoc, and Rust `///` comments); +- `AGENTS.md` and `.ai/skills/**/SKILL.md`; +- `.svg` files under `docs/source`, whose ``, `<desc>`, and `<text>` elements are text readers + see. + +Save that list to a file, one path per line, and pass it to Git as `-- $(cat "$PATHS")` wherever a +command says `-- <paths>`. Code files are all kept here; step 4 keeps only their doc comments: + +```bash +PATHS="${STATE%.env}.paths" +git -c core.quotePath=false diff --name-only "$BASE" "$HEAD_SHA" \ + | grep -E '^(docs/source/.*\.(md|rst|svg)|README\.md|AGENTS\.md|\.ai/skills/.*/SKILL\.md|.*\.(scala|java|rs))$' \ + | grep -v '/src/test/' > "$PATHS" +``` + +For doc comments, `AGENTS.md`, and skills, check the vocabulary and accuracy rules the guide names +for them, not the page conventions. Check the changed lines and the paragraphs around them, not the +whole page. Report a problem in unchanged text only when the change makes it wrong, or list it +separately as already on the page. Also read the code diff for the behavior the changed docs +describe, because docs are usually only as accurate as the code change beside them. + +In every mode: + +- Skip the content the guide's **Generated Content** section lists as generated or as records. + Review the hand-written text around the generated markers. On `main` the text between the markers + is empty, because the docs build fills it in, so any text a change adds between them is a hand + edit, and a finding (`P5`). +- Do the judgment checks in step 5 for at most 10 files per run, counting each file that has lines + in `$ADDED` after the code-file filter in step 4. In audit mode, count the files step 1 lists, in + that order. List any pages you skipped so the user can run the skill again on them. + +## 4. Mechanical Checks + +These checks produce candidates, not findings. Read each hit in context before reporting it. For the +wording checks, ignore hits inside fenced code blocks, inline code, quoted plan output or logs, and +text that quotes a phrase in order to discuss it (`N6`), such as the style guide's own examples. The +accuracy checks still apply to code font: config keys, operator names, and code examples are usually +written that way. For each check, record what it checked, the number of hits, and how many of them +are findings. + +The commands use `git grep -E` to find candidate lines and `perl` to match them. They do not need a +Git built with PCRE, and they avoid two traps: the system `grep` on macOS has no `-P`, and `\b` +misses Markdown italics such as `_native execution_`, because `_` counts as a word character. + +**Lines to check.** In PR and branch mode, list the added lines with their file and line number, and +run the line checks below on this list only. In audit mode, run them on the files with +`git grep -n --untracked`. + +```bash +ADDED=$(mktemp "$(git rev-parse --git-dir)/review-added.XXXXXX") +git -c core.quotePath=false diff --no-ext-diff --src-prefix=a/ --dst-prefix=b/ -U0 \ + "$BASE" "$HEAD_SHA" -- <paths> \ + | awk '/^--- /{h=1; next} + h && /^\+\+\+ b\//{f=substr($0,7); sub(/\t$/,"",f); h=0; next} + /^@@/{h=0; split($3,a,/[+,]/); n=a[2]; next} + /^\+/{print f":"n": "substr($0,2); n++}' > "$ADDED" +``` + +In a code file, only doc comments and `.doc(...)` strings are text to check. Before the wording checks, +leave out its other added lines: + +```bash +perl -i -ne 'print if !/\.(scala|java|rs):\d+: / || /:\d+: \s*(\/\*\*|\*|\/\/[\/!]|\.doc\(|")/' "$ADDED" +``` + +**"Native" and "vectorized".** Check each hit against `N1`-`N6` and `T10`. Which page the hit is on +matters, because `N3` allows more in the contributor guide: + +```bash +P='(?:(?<![A-Za-z0-9])native(?:ly)?(?![A-Za-z0-9])|vectori[sz]ed)' +perl -ne "print if /^.+?:\d+: .*$P/i" "$ADDED" +git grep -n -i --untracked -E 'native|vectori[sz]ed' -- $(cat "$PATHS") | perl -ne "print if /^.+?:\d+:.*$P/i" # audit +``` + +**Config keys.** Every `spark.comet.*` key a page names must exist (`P13`). List the keys the text +names, then the keys the code defines at the revision under review. `<rev>` is `"$HEAD_SHA"`. In +audit mode, leave `<rev>` out entirely: an empty `"$HEAD_SHA"` is an empty revision, which Git +rejects. The second command reads definitions that span two lines, expands +`createExecEnabledConfig`, and leaves out deprecated alternative names and unexpanded templates: + +```bash +perl -ne 'print "$1\n" while /(spark\.comet\.[\w.<>]*[\w>])/g' "$ADDED" | sort -u +git grep -h --untracked -o -E 'spark\.comet\.[A-Za-z0-9_.<>]*[A-Za-z0-9_>]' -- $(cat "$PATHS") | sort -u # audit +git grep -h -A1 -E '(conf|ConfigBuilder|createExecEnabledConfig)\(' <rev> -- spark/src/main contrib \ + | perl -0777 -ne ' + while (/\b(?:conf|ConfigBuilder)\(\s*s?"([^"]+)"/g) { + ($k = $1) =~ s/\$\{?COMET_EXEC_CONFIG_PREFIX\}?/spark.comet.exec/; + $k =~ s/\$\{?COMET_PREFIX\}?/spark.comet/; + print "$k\n" if $k =~ /^spark\.comet\./ && $k !~ /\$/; + } + while (/\bcreateExecEnabledConfig\(\s*"([^"]+)"/g) { print "spark.comet.exec.$1.enabled\n" }' \ + | sort -u +``` + +If the second command prints nothing, it is broken: stop and report that, rather than flagging every +key. Some keys are built from a name at run time: + +- `spark.comet.expression.<Name>.enabled` and `.allowIncompatible`, also written with an uppercase + placeholder such as `EXPRNAME`: `<Name>` is the Spark expression + class name, as in the `QueryPlanSerde` serde maps, not a Comet or protobuf name. Some expressions + use their own opt-in key instead, set by `nativeOptInConfigKeyOverride`, such as + `spark.comet.caseConversion.enabled`. +- `spark.comet.operator.<Name>.allowIncompatible`: `<Name>` is the Spark operator class name, such as + `ProjectExec`. +- `spark.comet.datafusion.<key>`: not a Comet config. Comet passes it to DataFusion only when + `spark.comet.exec.respectDataFusionConfigs=true`, which is off by default. Check the key against + DataFusion. + +Before reporting a missing key, search the code for it, because a few keys are defined in other +ways, such as `spark.comet.version` in `Plugins.scala`. A deprecated key that a page still uses is a +finding (`P16`). + +**Operator names.** Check operator names against `O1`, class names against `O2`, and expressions +against `O4`. List the concrete operator classes and the classes that override their printed name: + +```bash +git grep -h -E 'case class (Comet|Iceberg)' <rev> -- spark/src/main contrib \ + | perl -ne 'print "$1\n" if /case class ((?:Comet|Iceberg)[A-Za-z]+Exec)\b/' | sort -u +git grep -n -E 'override (def|val|lazy val) (nodeName|simpleString)' <rev> -- spark/src/main contrib +``` + +`spark/src/main` includes the per-Spark-version source directories. If the first command prints +nothing, it is broken: stop and report that. A class's `nodeName` is its name without `Exec`, unless +it overrides `nodeName`. Compare each name in the text with these, as `O1` describes, not with a rule +about suffixes. When the page is an operator reference, also report concrete operators that it +leaves out. Leave out planning markers such as `CometScanExec`, which never appear in a final plan, +and contrib operators, which belong on their contrib page. A name the text calls a protobuf message, +such as `ToJson`, is not an operator: check it in `native/proto` instead. When you search the code +for a class name, `git grep -E` has no `\b`, so end the pattern with `([^A-Za-z0-9_]|$)`. + +**Docs build.** Links, anchors, images, toctrees, and page structure, in Markdown and +reStructuredText, are checked by building the docs the way the published site is built. This +catches what a reading of the changed lines cannot, such as a link on an unchanged page that a +renamed heading breaks, or a page that no toctree includes. Install the docs requirements once, in a +virtual environment that later calls find at the same path: + +```bash +VENV="$(git rev-parse --git-dir)/review-comet-docs-venv" +PY=$(for p in python3.13 python3.12 python3.11 python3.10 python3; do + command -v $p >/dev/null && $p -c 'import sys; sys.exit(sys.version_info < (3, 10))' && { echo $p; break; } +done) +[ -x "$VENV/bin/sphinx-build" ] || { [ -n "$PY" ] && "$PY" -m venv --clear "$VENV" && + "$VENV/bin/pip" install -q --disable-pip-version-check -r docs/requirements.txt; } || + { rm -rf "$VENV"; echo "docs build not checked: needs Python 3.10 or later" >&2; } +``` + +Then build the base and the head and compare their warnings. The function copies the expression +templates into the per-version pages, as `docs/build.sh` does, so links to those pages resolve. It +returns an error if any step fails, including Sphinx itself, so a failed build is never read as a +clean one: + +```bash +build() { # $1 = revision, or worktree in audit mode; $2 = output directory + VENV="$(git rev-parse --git-dir)/review-comet-docs-venv" + mkdir -p "$2/src/docs" || return 1 + if [ "$1" = worktree ]; then + cp -R docs/source "$2/src/docs/" && profiles=$(cat docs/build.sh) || return 1 + else + git archive "$1" docs/source | tar -x -C "$2/src" && profiles=$(git show "$1:docs/build.sh") || + return 1 + fi + T="$2/src/docs/source/user-guide/latest/compatibility/expressions" + for p in $(printf '%s\n' "$profiles" | sed -n 's/^SPARK_PROFILES=(\(.*\))/\1/p'); do + mkdir -p "$T/$p" && cp "$T/_category_template"/*.md "$T/$p/" || return 1 + done + rm -rf "$T/_category_template" + "$VENV/bin/sphinx-build" -q -b html -n -D mermaid_output_format=raw "$2/src/docs/source" \ + "$2/html" > "$2/log.txt" 2>&1 || { echo "sphinx-build failed: $2/log.txt" >&2; return 1; } + sed -e "s#$2/src/##" -e 's/:[0-9]*: /: /' "$2/log.txt" | sort > "$2/warnings.txt" +} +WORK=$(cd "$(mktemp -d)" && pwd -P) +build "$BASE" "$WORK/base" && build "$HEAD_SHA" "$WORK/head" && + comm -13 "$WORK/base/warnings.txt" "$WORK/head/warnings.txt" +build worktree "$WORK/head" && cat "$WORK/head/warnings.txt" # audit +``` + +Every warning that only the head build has is a finding: a broken link, anchor, or image, a page +missing from a toctree (`P6`), or a heading problem (`P2`). In audit mode, report the warnings for +the audited pages. If the docs requirements cannot be installed or a build fails, say that the build +was not checked, and check that each changed link's target file exists. + +Sphinx does not count H1s. Check that each new or changed Markdown page has one H1 with the parser +the docs build uses, which knows where code blocks start and end. It needs the same virtual +environment; without it, count the H1s by reading the page and say so: + +```bash +H1='import sys; from markdown_it import MarkdownIt +print(sum(t.type == "heading_open" and t.tag == "h1" for t in MarkdownIt("commonmark").parse(sys.stdin.read())))' +git show "${HEAD_SHA}:<file>" | "$(git rev-parse --git-dir)/review-comet-docs-venv/bin/python" -c "$H1" +"$(git rev-parse --git-dir)/review-comet-docs-venv/bin/python" -c "$H1" < <file> # audit +``` + +**Other structure checks.** A new file has the ASF header (`P1`). A link to Comet's published docs by +URL breaks `P4` on pages under `docs/source`, but not in `README.md`. List the pages the change +renames or deletes; each one needs an entry in `redirects` in `docs/source/conf.py` (`P17`). This +check needs a base, so audit mode skips it: + +```bash +git diff --name-status --diff-filter=DR "$BASE" "$HEAD_SHA" -- docs/source +``` + +**External links.** For external links on the lines under review (at most 20 in audit mode), check +the status with `curl -s -o /dev/null -w '%{http_code}' -I -L --max-time 10 <url>`. Report a 404 as +a finding. If the request fails or the network is unavailable, say the link was not checked. + +**Version claims and markers.** Find version claims to check against `P8`, and `TODO` or `XXX` +markers, which break `P7`: + +```bash +V='(?:(?:Spark|Iceberg) [0-9]\.[0-9x]+\+?|\b(?:TODO|XXX)\b)' +perl -ne "print if /^.+?:\d+: .*$V/" "$ADDED" +git grep -n --untracked -E 'Spark [0-9]|Iceberg [0-9]|TODO|XXX' -- $(cat "$PATHS") | perl -ne "print if /$V/" # audit +``` + +CI runs prettier and the Apache RAT license check on PRs, so do not repeat them. If CI has not run +them yet, for example on a draft, say they were not checked. Mention them otherwise only when CI has +failed on them. + +## 5. Judgment Checks + +- **Is it true?** For every sentence that names a class, rule, config, default, or file, read the + code it describes at `HEAD_SHA`, or in the working tree in audit mode. A sentence can follow + every wording rule and still be wrong: a scan that no longer exists, a config default that has + changed, or an operator described as Rust-implemented that runs on the JVM. Check numbers, versions, and names first, and + check that a condition in the text is the whole condition in the code. Pages that describe + architecture or test infrastructure go stale most often. A false or outdated claim breaks `P16`. + This is the most valuable check in this skill. +- **Does it agree with the rest of the docs?** A new sentence can contradict an older one on the + same page, or a count such as "five configs" can stop matching the list below it (`P16`). When a + change rewrites the description of a config or class, search `docs/source` for the same name and + check the other descriptions. +- **Which axis?** When you suggest a replacement for vague "native", choose the term the code + supports: _in the Comet pipeline_ (`T1`), _Rust-implemented_ (`T3`), or _JVM-side_ (`T4`). Never + suggest _Rust-implemented_ without checking the code. +- **Page conventions.** Check `P3`, `P7`, `P8`, `P9`, `P10`, `P11`, `P12`, `P14`, and `P18` by + reading the page. For plan output (`P12`, `P16`), check that the operator names exist, that the + tree shape is one Spark could print, and that messages in it match the strings in the code. Do not + ask for new plan output to be invented. +- **Same-change updates.** If the change adds an operator or expression, check the pages `P15` + lists. If it adds a config, check that its `.doc(...)` string follows the guide, since + `GenerateDocs` publishes it, and that any page that documents the config by hand mentions it. + +## 6. Severity + +- **Must-fix.** The page is wrong or broken: a false or outdated claim (`P16`), a config key, class, + or file that does not exist (`P13`), a broken link, anchor, or image, a hand edit to generated + content, a new file without the ASF header, or a new page missing from a toctree. When a sentence + is both false and badly worded, report it once, as must-fix, with a rewrite that fixes both. +- **Should-fix.** The page breaks a style guide rule, but what it says is true. Most `T`, `N`, `O`, + and `P` findings belong here. +- **Nit.** A one-off change that would make the text clearer, but that no rule requires. Nits carry + no rule ID. + +A problem that keeps coming up and that no rule covers is a **guide gap**, not a finding. Report it +so the guide can be extended. + +## 7. Output + +When this skill runs as part of a `review-comet-pr` review, put its findings into that skill's +**Findings** and **Suggested Review Comments**, and drop the nits. That skill's review bar has no +third tier, and its rules win for anything posted to a PR. On its own, use this format: + +1. **Scope.** The mode, the revisions reviewed (the base and head commits, and which guide version), + the files reviewed, and the files skipped, with the reason. +2. **Mechanical checks.** A table with each check, its hit count, and how many hits are findings. +3. **Findings**, grouped as must-fix, should-fix, and nit. For each one give the `path:line`, the + rule ID, the text quoted from the page, and a suggested rewrite. Keep rewrites short and plain, + and change only what the rule requires. +4. **Guide gaps.** + +If there are no findings, say so. + +## Checking This Skill + +After changing this skill or the style guide, run it on these cases and compare the results. The +expected results were checked on `main` in October 2026, so a later fix to a page can change them. +Each row lists results the run must report and false positives it must not. A run can also report +other real findings. + +| Run | Expected | +| ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| PR mode on #6786, which predates the guide, so step 2 reads it from `$TARGET` or the working tree | Must-fix: `datasources.md` and the new `.doc` string claim more than `shouldBridgeUnsupportedScan` does | +| Audit `docs/source/user-guide/latest/understanding-comet-plans.md` | No finding for Comet's _native shuffle_, `CometNativeScan`, `CometWindowExec`, or config keys | +| Audit `docs/source/contributor-guide/timezones.md` | No finding for _native code_ or _native plan_, or for `ToJson` and `ToCsv`, which are protobuf message names | +| Audit `docs/source/user-guide/latest/compatibility/expressions` | Lists all 16 files, reviews 10 of the templates and index pages, names the skipped ones, and skips only text between markers | +| Audit a scratch page containing `_native execution_` and "Spark's native shuffle" | `N1` and `N5` findings | diff --git a/.ai/skills/review-comet-expression-pr/SKILL.md b/.ai/skills/review-comet-expression-pr/SKILL.md index c1a80773961..a5a22fe8145 100644 --- a/.ai/skills/review-comet-expression-pr/SKILL.md +++ b/.ai/skills/review-comet-expression-pr/SKILL.md @@ -225,9 +225,10 @@ and the requirement to record a performance audit. Hold the PR to both. ### Generated, review the source instead -The compatibility pages under `docs/source/user-guide/latest/compatibility/expressions/` and -`docs/source/user-guide/latest/configs.md` are produced by `GenerateDocs` and regenerated by CI on -merge. Never ask the contributor to hand-edit them. Check what feeds them: +`GenerateDocs` fills in the expression compatibility tables and the config tables when the docs are +built, and CI rebuilds and publishes the docs when a merge changes their inputs. The **Generated Content** section of +`docs/source/contributor-guide/style_guide.md` says exactly which text is generated. Never ask the +contributor to hand-edit that text. Check what feeds it: - Expression marked `Incompatible(Some("..."))` but `getIncompatibleReasons()` is empty, so the compat page shows it as supported with no caveats. diff --git a/.ai/skills/review-comet-pr/SKILL.md b/.ai/skills/review-comet-pr/SKILL.md index ea1f57a9d1f..501ea2aaf1f 100644 --- a/.ai/skills/review-comet-pr/SKILL.md +++ b/.ai/skills/review-comet-pr/SKILL.md @@ -1,6 +1,6 @@ --- name: review-comet-pr -description: Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, and Iceberg writes. Provides guidance to a human reviewer rather than posting comments. +description: Use when reviewing a DataFusion Comet pull request. Covers the workflow that applies to every PR and routes to the area-specific review skills for expressions, FFI, memory management, shuffle, Iceberg writes, and documentation. Provides guidance to a human reviewer rather than posting comments. argument-hint: <pr-number> --- @@ -43,6 +43,7 @@ change. | `native/core/src/execution/memory_pools/`, `CometTaskMemoryManager`, `CometShuffleMemoryAllocator`, `MemoryConsumer` / `try_grow` / `reserve` call sites, pool configs | `review-comet-memory-pr` | | `native/shuffle/`, `spark/src/main/scala/org/apache/spark/sql/comet/execution/shuffle/`, `spark/src/main/java/org/apache/spark/shuffle/`, `CometShuffleExchangeExec` | `review-comet-shuffle-pr` | | `IcebergWriteStrategy`, `IcebergWriteExec`, `IcebergCommitExec`, `CometIcebergWriteExec`, `CometIcebergNativeWrite`, `iceberg_write.rs`, `iceberg_partition_path.rs`, the iceberg-rust pin | `review-comet-iceberg-write-pr` | +| `docs/source/`, `README.md`, `AGENTS.md`, `.ai/skills/**/SKILL.md`, `.doc(...)` strings in config classes, doc comments | `review-comet-docs` | For a new operator, read `docs/source/contributor-guide/adding_a_new_operator.md` alongside this skill. There is no dedicated operator review skill yet. @@ -52,8 +53,8 @@ For a PR that deals with timestamps or the session timezone, read cuts across the serde, the native kernels, the scans and the JVM/native boundary, so no area skill owns it. "Timestamps and timezones" in step 5 says what to check. -If the PR falls outside all of these, for example build, CI, docs only, or release tooling, this -skill on its own is the review. +If the PR falls outside all of these, for example build, CI, or release tooling, this skill on its +own is the review. ## 2. Gather PR Metadata @@ -95,7 +96,7 @@ Anything that reports `getSupportLevel()` must report it honestly: - `Compatible()`, matches Spark exactly - `Incompatible(Some("reason"))`, differs in documented ways and requires - `spark.comet.expr.allowIncompatible=true` + `spark.comet.expression.<Name>.allowIncompatible=true` - `Unsupported(Some("reason"))`, cannot be implemented A PR that quietly marks something `Compatible` while the diff shows a known divergence is the single @@ -264,7 +265,8 @@ For a change on a default path, look for tests with the inputs that broke earlie ### CI -`gh pr checks <pr> --repo apache/datafusion-comet`, and again with `--failed` for detail. +`gh pr checks <pr> --repo apache/datafusion-comet`. For only the failed checks, add +`--json name,state,bucket,link --jq '.[] | select(.bucket == "fail")'`. Summarize failures in the review. Do not compare against failures on `main`. @@ -299,10 +301,13 @@ same PR, because a follow-up that is not filed as an issue does not happen. Each area skill names the doc for its subsystem and the specific claims in it that go stale. When you load an area skill, read its doc and hold the diff against it. -**Do not ask contributors to hand-edit generated docs.** The compatibility guide pages under -`docs/source/user-guide/latest/compatibility/expressions/` and -`docs/source/user-guide/latest/configs.md` are produced by `GenerateDocs` and regenerated by CI on -every merge to `main`. Review the source that feeds them instead. +**Do not ask contributors to hand-edit generated docs.** `GenerateDocs` fills in the config tables +and the expression compatibility tables when the docs are built, and CI rebuilds and publishes the +docs when a merge to `main` changes their inputs. The +**Generated Content** section of `docs/source/contributor-guide/style_guide.md` says exactly which +text is generated. Review the source that feeds it instead. The prose around the generated tables, +the `_category_template/` files, and the `index.md` pages are written by hand and are reviewed +normally. ## Review Bar diff --git a/AGENTS.md b/AGENTS.md index b7a732268dc..57d10fbc0a9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,6 +30,9 @@ Relevant entry points: - [Adding a New Expression](docs/source/contributor-guide/adding_a_new_expression.md) / [Adding a New Operator](docs/source/contributor-guide/adding_a_new_operator.md). - [Debugging Guide](docs/source/contributor-guide/debugging.md). +- [Documentation Style Guide](docs/source/contributor-guide/style_guide.md): the terms to use for + how Comet runs a query, and the conventions for docs pages. Read it before editing `docs/source`, + `README.md`, or the `.doc(...)` strings in config classes. - [Backporting to Release Branches](docs/source/contributor-guide/backporting.md): a fix that goes to one release branch also goes to every newer one. Cherry-pick with `-x`, and never add an unrelated change to a backport pull request. diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.md index 53acfa1be69..1be327be39e 100644 --- a/docs/source/contributor-guide/index.md +++ b/docs/source/contributor-guide/index.md @@ -99,6 +99,7 @@ Expression Audits <expression-audits/index> Data Source Audits <data-source-audits/index> Supported Spark Configurations <spark_configs_support> Configuration Conventions <config_conventions> +Documentation Style Guide <style_guide> ``` ```{toctree} diff --git a/docs/source/contributor-guide/style_guide.md b/docs/source/contributor-guide/style_guide.md new file mode 100644 index 00000000000..d0921bc1004 --- /dev/null +++ b/docs/source/contributor-guide/style_guide.md @@ -0,0 +1,222 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Documentation Style Guide + +This guide is for anyone who writes or reviews Comet documentation: pages under `docs/source`, +`README.md`, and the user-facing `.doc(...)` strings in `CometConf.scala` and in contrib config +classes. It defines the words Comet's docs use for how a query runs, and the conventions every page +follows. The vocabulary rules (`T`, `N`, `O`) and the accuracy rules (`P13`, `P16`) also apply to doc +comments, `AGENTS.md`, and the agent skills under `.ai/skills`, with the contributor guide's +allowances. + +The sections cover how Comet runs a query, the vocabulary, the rule for "native", operator names, +page conventions, and generated content. + +Each rule has an ID, such as `N1`. Reviews cite the ID, so IDs are never renumbered or reused. +Where this guide and the code disagree, the code is right and the guide needs a fix. The +background for the vocabulary is in [#4419](https://github.com/apache/datafusion-comet/issues/4419). + +## How Comet Runs a Query + +The terms below describe this model. [Comet Plugin Architecture](plugin_overview.md) has the details. + +- Comet's planner rules replace the Spark operators Comet supports with Comet operators, most of + them named `Comet*`, and serialize their expressions to protobuf. An operator Comet does not + support stays a Spark operator. +- Most Comet operators run in Comet's Rust code, which is built on Apache DataFusion and is called + from the JVM through JNI. Some do their own work on the JVM, such as broadcast exchanges. +- Comet's query operators usually pass data to each other as Apache Arrow batches. Some expressions + run Spark's own generated JVM code over those batches. + +## Vocabulary + +Use the terms in the first column. They answer three separate questions: does Comet handle the +work, which language runs it, and what format is the data in. A term from one question says nothing +about the other two. + +| ID | Use | For | Instead of | +| --- | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------- | +| T1 | _Comet pipeline_, _runs in Comet_ | Work that a Comet operator handles, whatever the language | "the native path", "runs natively" | +| T2 | _falls back to Spark_, _Spark fallback_ | Work that stays in a Spark operator | — | +| T3 | _Rust-implemented_, _runs in Rust_, _native Rust_ | Work done by Comet's Rust code, usually through DataFusion | "native" on its own | +| T4 | _JVM-side_, _runs on the JVM_ | A Comet operator whose own work runs on the JVM, such as `CometBroadcastExchange` | "non-native Comet operator" | +| T5 | _codegen dispatch_ | Spark's generated JVM code for an expression, run over Arrow batches from inside a Rust-implemented plan | "falls back" (it does not fall back) | +| T6 | _Native_, _Codegen dispatch_, _Hybrid_ | The expression kinds in the [Implementation legend](../user-guide/latest/expressions.md#implementation-legend) | Any other meaning of "Hybrid" | +| T7 | _Arrow-native_, _Arrow batches_, _Arrow columnar_ | Claims about data format: the data is held in Arrow columnar format | "vectorized" as a synonym for columnar (T10) | +| T8 | _Arrow IPC_ | The format shuffle and broadcast write | — | +| T9 | _Spark rows_ (`UnsafeRow`), _Spark columnar_ | Spark's row format, or a Spark `ColumnarBatch` that is not backed by Arrow | "row format", "JVM format" without saying which | +| T10 | _vectorized_ | Proper names, such as Spark's vectorized Parquet reader, and batch-at-a-time kernels contrasted with row-by-row loops | "vectorized" as a synonym for columnar | + +Notes on these terms: + +- **Hybrid (T6)** describes an expression that has both a Rust path and a Spark-compatible path, + usually codegen dispatch. Each use of the expression in a plan takes one path. The term comes + from `GenerateDocs`, which writes the Implementation column of `expressions.md`. Do not use + "hybrid" for a plan that mixes Comet and Spark operators. That is a plan with fallback (T2). +- **Arrow-native (T7)** is about format. It does not mean that no per-row work happens: codegen + dispatch evaluates each row, and JVM shuffle buffers `UnsafeRow`s before encoding them to Arrow + IPC. +- **Spark rows (T9)** can appear inside the Comet pipeline. JVM shuffle, for example, buffers + `UnsafeRow`s. The format alone does not say whether Spark or Comet handles the work. +- **Expressions are not plan nodes.** An expression runs inside an operator such as `CometProject` + or `CometFilter`. A codegen-dispatched expression leaves its operator Rust-implemented. + +## The Rule for "Native" + +"Native" has several meanings in Comet's docs, so on its own it does not tell the reader which one +is meant. Name the meaning instead. + +- **N1.** Do not use _runs natively_, _natively in Comet_, _the native path_, _the native + pipeline_, _native execution_, _native operator_, or _native Comet operator_, in any tense or + form, such as _ran natively_, _go native_, _a native path_, or _Comet-native_. Write _in the Comet + pipeline_ (T1) or _Rust-implemented_ (T3). When "native" means "did not fall back to Spark", write + T1, because codegen dispatch (T5) is not a fallback. When the sentence is about Spark's own + behavior, N5 applies. +- **N2.** These compounds are allowed everywhere, because the second part fixes the meaning: + _native Rust_; _native shuffle_ and _the native shuffle path_ (the counterpart of _JVM shuffle_, + which need not appear beside it); "native", optionally followed by a format, followed by _scan_, + _reader_, _write_, or _writer_, only for Comet's Rust-implemented scans and writers (_native scan_, + _native Parquet reader_, _native Iceberg write_), not for a JVM-side scan such as + `CometInMemoryTableScanExec`; _native memory_ and _native library_ (the terms Comet's own log + lines use); _Arrow-native_ (T7); _Native_ as the expression kind (T6); and _native implementation_ + of an expression, meaning its Rust implementation. +- **N3.** In the contributor guide, "native" before a code noun is also allowed when it means the + crates under `native/` or the JNI boundary: _native code_, _native side_ (and _native-side logs_), + _native plan_, _native kernel_, and _JVM/native boundary_. The noun must name something under + `native/`, such as a crate, file, type, function, or plan, or the JNI boundary. Pages outside the + contributor guide, including `user-guide/`, `about/`, `index.md`, `README.md`, and `.doc(...)` + strings, allow only the N2 compounds and write _Rust_ instead. N1 applies everywhere. Any other use + of "native" is a finding: name the meaning with a term from the vocabulary above. +- **N4.** Do not use "native" as a noun. Write _the Rust side_ or name the component: "hands each + block to the Rust side", not "hands each block to native". +- **N5.** Do not call Spark's own behavior native. Write _Spark's built-in shuffle_, not _Spark's + native shuffle_. +- **N6.** Quote identifiers, config keys and values, API names, log lines, and plan output exactly + as they are, in code font. `CometNativeScan`, `CometNativeExec`, and + `spark.comet.shuffle.mode=native` are not prose and do not break N1. Text that quotes a phrase in + order to discuss it, such as the "Before" column below, does not break these rules either. + +Examples: + +| Before | After | +| ---------------------------------------------------------- | ------------------------------------------------------------- | +| Runs your Spark queries on the DataFusion native engine. | Speeds up your existing Spark queries with Apache DataFusion. | +| The surrounding native operators keep running natively. | The surrounding operators stay in the Comet pipeline. | +| Keeps evaluation inside the native pipeline. | Keeps evaluation inside the Comet pipeline. | +| Takes columnar input directly from Comet native operators. | Takes Arrow batches directly from Comet operators. | + +## Operators, Plan Names, and Expressions + +[Understanding Comet Plans](../user-guide/latest/understanding-comet-plans.md) describes the nodes +a plan can contain, grouped by role. Its +[Comet Operator Reference](../user-guide/latest/understanding-comet-plans.md#comet-operator-reference) +lists them. Link to it rather than copying it. + +Do not infer an operator's kind from its class hierarchy. For example, +`CometTakeOrderedAndProjectExec` runs Rust plans without extending `CometNativeExec`. Most Comet +operators print a `Comet` prefix, but not all: `IcebergWrite` and `IcebergCommit` are Comet +operators too. + +- **O1.** In prose about plans, use the name that plan output prints. Spark drops the `Exec` suffix + from most names (`CometProjectExec` prints as `CometProject`), but some operators keep it, such as + `CometWindowExec`, and `CometShuffleExchangeExec` prints as `CometExchange` or + `CometColumnarExchange`. This is the `nodeName`, which formatted explain output and the Spark UI + print. Some classes also override `simpleString`, which changes the line `df.explain()` prints: + `CometTakeOrderedAndProjectExec` prints as `CometTakeOrderedAndProjectExec(...)` there. Quote plan + output exactly as it was printed (`N6`), and check the operator reference when unsure. +- **O2.** Use the class name, with `Exec`, only when the text is about the code, including text + about how a planner rule rewrites a plan. +- **O3.** The current names are the correct names. Do not describe renames that have not happened. +- **O4.** Expressions are not operators. Describe an expression by its kind (T6), not as a node in + the plan. + +## Page Conventions + +- **P1.** Every new Markdown file starts with the ASF license header, as in this file. +- **P2.** A page has one H1, and heading levels are not skipped. +- **P3.** The first paragraph says who the page is for and what it covers. A long page then gives + a short overview of its sections. +- **P4.** On pages under `docs/source`, link to another page by its path, either relative to the + current page or starting with `/` for the `docs/source` root, not by the published docs URL, + because the site's address can change. `README.md` is rendered by GitHub, so it links to the + published docs. +- **P5.** Do not hand-edit generated content. Change the source it comes from instead. See + [Generated Content](#generated-content). +- **P6.** Every new page is added to a toctree. Pages for users go under `docs/source/user-guide/` + or `docs/source/about/`, and pages for contributors go under `docs/source/contributor-guide`. +- **P7.** The user guide describes what Comet does today, in the present tense. Plans go in the + [roadmap](roadmap.md). Do not leave `TODO` or `XXX` markers in published pages. +- **P8.** A claim that depends on a version, such as "Spark 3.5 and later", "Spark 3.5+", or "Spark + 4.0 only", matches the versions the code supports. The same applies to other dependencies, such + as Iceberg. +- **P9.** Prefer the active voice, and address the reader as "you". +- **P10.** Do not use marketing words such as "blazing fast", "seamless", or "powerful". State the + measured result, or leave the claim out. +- **P11.** Use a table to compare several items, such as operators, configs, or support levels. +- **P12.** Plan output in a page is real output from `explain`, in Spark's tree format with `+-` + connectors, and is shown in a code block. +- **P13.** Config keys, class names, and file paths named in a page exist in the code. Code + examples are valid in the language their block declares. Follow + [Configuration Conventions](config_conventions.md) for new configs. +- **P14.** A new user guide page includes at least one example a reader can run and links to + related pages. A new contributor guide page that describes a procedure says how to check that it + worked. +- **P15.** A change that adds an operator updates the + [Comet Operator Reference](../user-guide/latest/understanding-comet-plans.md#comet-operator-reference) + and [Supported Spark Operators](../user-guide/latest/operators.md) in the same pull request. A + change that adds an expression updates its row in + [Spark Expression Support](../user-guide/latest/expressions.md). +- **P16.** Describe the code as it is now. Remove text about code that has been removed, keep + sample output, such as explain output and messages, in step with the code that produces it, and + keep each page consistent with itself. +- **P17.** A change that moves, renames, or removes a page adds an entry to `redirects` in + `docs/source/conf.py`, so that existing links keep working. +- **P18.** Text, diagrams, and images taken from another project or website say where they came + from, and their license is compatible with the Apache License. See the ASF's + [third-party licensing policy](https://www.apache.org/legal/resolved.html). + +### Generated Content + +These parts of the docs are written by tools. Edit the source they come from instead. The rest of +each file, including the prose around the markers and the `index.md` pages, is written by hand. + +- **Config tables**, the text between `<!--BEGIN:CONFIG_TABLE[...]-->` and + `<!--END:CONFIG_TABLE-->`. `GenerateDocs` builds them from the config entries in `CometConf.scala` + and from the contrib config classes registered with `CometConfigProvider`. The `enable_expr` and + `enable_agg_expr` tables come from the expression serde maps in `QueryPlanSerde` instead. +- **Expression compatibility sections**, the text between `<!--BEGIN:EXPR_COMPAT[...]-->` and + `<!--END:EXPR_COMPAT-->` in `compatibility/expressions/_category_template/`. `GenerateDocs` builds + them from the expression serdes. +- **Cast tables**, the text between the `CAST_*_TABLE` markers. `GenerateDocs` builds them from + `CometCast.supportedTypes`, `CometCast.isSupported`, and Spark's `Cast.canCast`. +- **Per-version expression pages** built from `_category_template/`. On `main`, the docs build + creates them in a temporary tree. Release branches commit a generated copy, made with + `dev/generate-release-docs.sh`. +- **The Implementation column** in `expressions.md`, built by `GenerateDocs` from the expression + serdes in `QueryPlanSerde`. +- **`changelog/`**, written by the release process. + +The expression audits, data source audits, and benchmark results under `contributor-guide/` are +records of past work, not pages to restyle. + +## Changing This Guide + +Propose changes in a pull request. Add new rules with new IDs, and do not renumber or reuse an ID, +even when a rule is removed.