Repository navigation
Conversation
… by pipe SET The pipe SET operator is built as a star expansion that excludes the assigned column and appends a replacement of the same name, so the original attribute is dropped from the project list and a later qualified reference such as `t.a` has nothing to resolve to. This contradicts the documented behavior that table aliases keep referring to the original row values after an assignment. Retain the excluded attribute as hidden output on the Project that SET builds, via the existing Project.hiddenOutputTag mechanism that USING joins use for duplicated join keys. AddMetadataColumns splices it back only when referenced, so the output schema of SET and the plan for `|> SELECT *` are unchanged. The retention is flagged from the parser because pipe SET and regular `SELECT * EXCEPT` / `* REPLACE` produce an identical node at star-expansion time, and the latter must not be affected.
|
I’m not sure who would be the right reviewers for this PR. Adding @uros-b and @cloud-fan, could you please help out. |
|
@dongjoon-hyun Can you please take a look and guide |
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
This needs changes before it is ready. The new SET hidden-output path can silently reorder the original row returned by a qualified star, and installing its hidden-output tag before normal tag forwarding drops Spark Connect plan identity from a rewritten Project. The scalar qualified-reference coverage does not exercise either qualified-star ordering or a pre-tagged Connect relation.
Findings
2 total: 0 P0, 1 P1, 1 P2, 0 P3.
Blocking (P1)
- Preserve source column order in qualified-star expansion —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2107— see inline.
Non-blocking (P2)
- Preserve the original Project tags when adding hidden output —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2105— see inline.
| if (retain) { | ||
| val excepted = expanded.child.output.filterNot(expanded.outputSet.contains) | ||
| if (excepted.nonEmpty) { | ||
| expanded.setTagValue( |
There was a problem hiding this comment.
Non-blocking (P2): transformUpWithPruning normally forwards the original Project tags after this rewrite, but copyTagsFrom only does so when the destination tag map is empty. Setting hiddenOutputTag here first makes the map nonempty, so a Spark Connect relation rooted at pipe SET loses PLAN_ID_TAG; DataFrame columns bound to that relation can then fail with CANNOT_RESOLVE_DATAFRAME_COLUMN. Please preserve or merge the original project's tags when adding this tag.
There was a problem hiding this comment.
Thanks a lot @cloud-fan for catching both the issues. I've addressed them in d092103.
Moved copyTagsFrom to run before setting hiddenOutputTag. Previously, copying tags afterwards was overwriting PLAN_ID_TAG.
There was a problem hiding this comment.
Verified on the current commit: the original Project tags are copied before hiddenOutputTag is installed, and the focused analyzer test preserves both PLAN_ID_TAG and hiddenOutputTag. This issue is resolved.
| if (excepted.nonEmpty) { | ||
| expanded.setTagValue( | ||
| Project.hiddenOutputTag, | ||
| excepted.map(_.markAsQualifiedAccessOnly()) ++ expanded.child.metadataOutput) |
There was a problem hiding this comment.
Blocking (P1): UnresolvedStar expands qualified-only metadataOutput before visible output, so prepending the retained attributes here does not preserve the aliased row's column order. For example, VALUES (1, 2, 3) AS t(a, b, c) |> SET b = 20 |> SELECT t.* expands as (b, a, c) and returns (2, 1, 3) instead of (1, 2, 3). Please construct the retained output so t.* follows the original source order, including after repeated SETs.
There was a problem hiding this comment.
Done.
The SET Project now retains the full qualified source row, in its original order, as hidden output.
There was a problem hiding this comment.
Verified on the current commit: t.* now follows the original source-row order, including across repeated SET operations. This issue is resolved.
|
Thanks for the ping. I've reviewed the current SQL pipe SET patch and will leave the technical feedback on this PR. |
…T hidden output Address review comments: - Keep the original column order for a qualified star after pipe SET. The SET Project now retains the whole qualified source row, in order, as hidden output, and the qualified star expansion emits an attribute that is both hidden and visible once, at its hidden position. Previously `VALUES (1, 2, 3) AS t(a, b, c) |> SET b = 20 |> SELECT t.*` returned (2, 1, 3). DataFrame column resolution treats an attribute present in both output and metadataOutput as a regular match. - Copy the original Project tags before setting the hidden output tag, since tag forwarding only applies to a node without tags. This preserves the Spark Connect plan id of a relation rooted at pipe SET.
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
The SET source-row carrier needs to preserve the exact ordered row and remain durable until referenced attributes are materialized; fixing only star ordering or selected tag copies leaves that shared analyzer boundary unsound. The current revision resolves the earlier single-column ordering and immediate PLAN_ID_TAG overwrite issues, but it still collapses repeated row positions, depends on an auxiliary tag across analyzer iterations, and does not test the changed plan-ID visible/hidden classification. The repeated-position issue is blocking because it silently changes a valid query's schema and row shape.
Findings
3 total: 0 P0, 1 P1, 2 P2, 0 P3.
Blocking (P1)
- Preserve repeated source-row positions in qualified stars —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2115— see inline.
Non-blocking (P2)
- Make retained SET state durable until materialization —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2120— see inline. - Exercise plan-ID resolution for visible/hidden overlap —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:694— see inline.
Shared repair plans
Shared repair plan 1
Covered findings:
- Preserve repeated source-row positions in qualified stars —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2115 - Make retained SET state durable until materialization —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2120
Recommended change: Represent the retained SET source row as the exact ordered positional sequence and materialize any referenced source attributes within the same bounded analysis operation that resolves their parent references, so later rules no longer depend on an auxiliary tag for correctness. Add SQL analyzer coverage for repeated ExprIds and preservation through the full SET resolution path.
Why this works: Remove ExprId-based sequence deduplication and set-wide visible suppression in favor of occurrence-preserving matching. After parent expressions resolve through the retained row, perform narrowly scoped SET metadata materialization before ResolveReferences returns, leaving the hidden tag as discovery metadata rather than the sole cross-iteration correctness carrier.
Scope: Rework the Catalyst SET retention carrier and qualified-star merge, then extend pipe-operator SQL regression coverage for multiplicity and analyzer-lifetime behavior.
Compatibility: Pipe SET continues to expose assigned values through top-level names and original values only through pre-existing qualifiers, without widening its visible schema.
Risks: Eagerly materializing retained attributes too broadly could leak hidden columns into SET output or unqualified stars. Occurrence matching must preserve qualifiers and metadata while distinguishing repeated positions that share an ExprId.
Constraints: Ordinary SELECT * EXCEPT and * REPLACE must remain unchanged. USING/NATURAL and AS OF join hidden-output behavior must remain unchanged. The visible output schema of pipe SET and unqualified SELECT * must remain unchanged.
Success: Qualified t.* after SET returns every original aliased row position in order, including repeated positions with the same ExprId. Qualified scalar references to assigned source columns are backed by produced child attributes before subsequent analyzer rules run. Unqualified SET results and ordinary star-except behavior retain their existing schemas and values.
Re-review status
Prior AI findings: 2 addressed, 0 still present; additional unresolved findings in this review: 3.
New attribution: 2 newly introduced, 1 late catch, 0 previously raised, 0 unattributed.
Remaining prior AI findings
No prior AI findings remain.
| // then its output. Only qualified attributes are reachable through a qualified star. | ||
| val sourceRow = (child.metadataOutput.filter(_.qualifiedAccessOnly) ++ child.output) | ||
| .filter(_.qualifier.nonEmpty) | ||
| .distinctBy(_.exprId) |
There was a problem hiding this comment.
Blocking (P1): Project output is positional, so repeated projections can legitimately share an ExprId. Here distinctBy(_.exprId) drops one such position, and the later ExprId set removes every matching visible occurrence. For example, VALUES (1, 2) AS s(a, b) |> SELECT a, a, b |> AS t |> SET b = 3 |> SELECT t.* expands to [a, b] instead of [a, a, b], silently changing the schema and row shape. Please preserve source-row occurrences rather than using ExprId as positional identity.
See Shared repair plan 1 in the review body.
There was a problem hiding this comment.
@cloud-fan Done, Fixed in 262b18bbe7d
distinctBy(_.exprId) and the set-wide visible suppression are gone. t.* now returns (a, a, b) / 1 1 2
| // The rule forwards the original tags, such as the Spark Connect plan id, only onto a | ||
| // node without tags, so copy them here before adding the hidden output tag. | ||
| expanded.copyTagsFrom(original) | ||
| expanded.setTagValue( |
There was a problem hiding this comment.
Non-blocking (P2): This tag becomes correctness-critical across analyzer iterations: ResolveReferences can resolve t.a from metadata only after AddMetadataColumns has already run, so the attribute is materialized on a later iteration. Generic copyTagsFrom calls do not guarantee that hiddenOutputTag survives every intervening Project replacement. Please keep the retained source-row state at a durable owner, or materialize referenced source attributes within a bounded operation, so a documented qualified SET reference cannot lose its producing child.
See Shared repair plan 1 in the review body.
There was a problem hiding this comment.
The preference would be to retain hiddenOutputTag as the carrier, consistent with the existing analyzer lifecycle, added golden coverage across the relevant projection-rebuilding rules in 262b18bbe7d, and happy to add a case if any rule does not forward the tag correctly.
| // counts as a regular match. | ||
| val filtered = candidates.flatMap { c => | ||
| val hidden = c.hidden || c.expr.references.subsetOf(AttributeSet(p.metadataOutput)) | ||
| val hidden = c.hidden || (!c.expr.references.subsetOf(p.outputSet) && |
There was a problem hiding this comment.
Non-blocking (P2): The new test stops at tag coexistence and never constructs a plan-ID-bound column reference that reaches this visible/hidden overlap. Reverting this predicate would therefore leave the added assertions green while allowing the Spark Connect/DataFrame-column failure to return. Please add a focused analyzer case that invokes plan-ID resolution for an attribute present in both outputs and verifies that the visible candidate wins.
There was a problem hiding this comment.
Agreed, the earlier test was redundant. Replaced in 262b18bbe7d.
…T hidden output Address review comments: - Keep every position of the source row that pipe SET retains. A projection may repeat a column, and the repeated positions share an ExprId, so using the ExprId as positional identity dropped one of them: `VALUES (1, 2) AS s(a, b) |> SELECT a, a, b |> AS t |> SET b = 3 |> SELECT t.*` returned (a, b) instead of (a, a, b). Retention and qualified star expansion now match occurrences through the new UnresolvedStarBase.mergeHiddenAndVisibleOutput, where a hidden attribute consumes at most one visible occurrence and is emitted in its place. - Replace the AnalysisSuite case for DataFrame column resolution, which only asserted that the two Project tags coexist. It now resolves a plan id bound column across a visible and hidden overlap, which fails if the visible candidate is classified as hidden. - Add golden coverage for a qualified SET reference under the rules that rebuild the projection above it: filter, sort, aggregate, join, union, an enclosing subquery and a CTE.
|
@cloud-fan Can you please take a look |
|
@cloud-fan Addressed in 262b18b:
CI green, no conflicts. cc @dtenedor |
uros-b
left a comment
There was a problem hiding this comment.
Thank you @AMC-hawk and @cloud-fan for working on this!
|
Let's ping @cloud-fan for re-review, I think this looks good now |
|
@cloud-fan Requesting re-review |
cloud-fan
left a comment
There was a problem hiding this comment.
Review summary
The current revision addresses the prior repeated-position and plan-ID coverage issues. Two newly introduced issues remain: the shared hidden/visible merge can make qualified-star analysis quadratic for wide schemas, including pre-existing USING/NATURAL-join queries, and a new ScalaDoc sentence mixes counterfactual and past tense. The previously raised hidden-tag lifetime concern has no reachable current tag-loss path and is not retained.
Findings
2 total: 0 P0, 0 P1, 1 P2, 1 P3.
Non-blocking (P2)
- Avoid quadratic hidden/visible output merging —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala:653— see inline.
Nit (P3)
- Use consistent counterfactual tense in the ordering comment —
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2100— see inline.
Re-review status
Prior AI findings: 3 addressed, 0 still present; additional unresolved findings in this review: 2.
New attribution: 2 newly introduced, 0 late catch, 0 previously raised, 0 unattributed.
Remaining prior AI findings
No prior AI findings remain.
| * The hidden output holds the whole qualified source row, with every position in its original | ||
| * order, not just the excluded attributes, because a qualified star expands hidden output | ||
| * ahead of the visible output. Retaining only the excluded attributes would move them to the | ||
| * front, so that `VALUES (1, 2, 3) AS t(a, b, c) |> SET b = 20 |> SELECT t.*` returned |
There was a problem hiding this comment.
Nit (P3): This sentence starts a counterfactual with would move but completes it with returned, which makes the hypothetical (b, a, c) result read like observed current behavior. Please use would return (or make both clauses past-counterfactual) so the ordering rationale is unambiguous.
Verification:
- Inspection: Read the complete ordering comment and confirm its condition and consequence use consistent counterfactual grammar.
There was a problem hiding this comment.
Fixed in 5545d0d: it now reads "would return (b, a, c)".
| visibleOutput: Seq[Attribute]): Seq[Attribute] = { | ||
| val unconsumed = visibleOutput.to(mutable.ArrayBuffer) | ||
| val merged = hiddenOutput.map { hidden => | ||
| val visible = unconsumed.indexWhere(_.exprId == hidden.exprId) |
There was a problem hiding this comment.
Non-blocking (P2): Each hidden attribute scans the remaining visible buffer, and every overlap is removed from the middle. With wide USING/NATURAL joins this performs H * V searches; with pipe SET's full-width overlap, the removals repeatedly shift the remaining suffix. Qualified-star analysis therefore becomes quadratic before planning for both existing join queries and the new SET path.
Verification:
- Behavior: Compare merged output for disjoint, overlapping, repeated-ExprId, and repeated-SET inputs and assert exact order, multiplicity, qualifiers, and expression identities.
- Regression: Measure analysis across increasing schema widths for pipe SET plus a qualified star and for a wide USING/NATURAL join plus a qualified star, demonstrating linear rather than quadratic growth.
- Inspection: Confirm the merge performs one visible indexing pass, constant-time expected occurrence consumption, and one final visible-order emission pass.
There was a problem hiding this comment.
Fixed in 5545d0d.
-
Inspection: the merge now indexes visible positions by ExprId in one pass, consumes each in O(1) expected time, and emits the leftovers in one final pass.
-
Behavior: new AnalysisSuite cases cover disjoint, overlapping, repeated-ExprId and repeated-SET inputs, asserting exact instances. The new merge also matched the old one on 20k random inputs.
-
Regression: this also turned up AddMetadataColumns scanning the children's metadataOutput once per attribute, which made any wide SET quadratic. It now looks attributes up by ExprId. Two SETs + t.* at 16k columns went from 11.3 s to 0.33 s, now growing linearly. A qualified star over a wide USING/NATURAL join also grows linearly now.
|
I re-reviewed the current commit. The repeated-position, retained-state, and plan-ID changes are supported by the current analyzer paths and coverage. One issue remains: the new hidden/visible merge is quadratic for wide schemas, including existing qualified-star queries above wide USING/NATURAL joins. Please linearize that merge; there is also a one-word counterfactual tense fix in the new ScalaDoc. |
…idden output Address review comments: - Make UnresolvedStarBase.mergeHiddenAndVisibleOutput linear. It searched the remaining visible output for every hidden attribute and removed each match from the middle of a buffer, so a qualified star over a wide USING or NATURAL join, or after pipe SET, was quadratic in the schema width. It now indexes the visible occurrences by ExprId in one pass, consumes them in constant expected time and emits the unconsumed ones in one final pass, with the same output. - Look up the metadata output of the children by ExprId in AddMetadataColumns. It scanned that output for every attribute of a node, and pipe SET retains the whole source row as hidden output, so analyzing any wide pipe SET query was quadratic, with or without a qualified star. - Use a consistent counterfactual in the comment on retainExceptedColumnsAsHiddenOutput. - Add AnalysisSuite cases for the merge on disjoint, overlapping and repeated inputs, checking the exact instances it emits, and for a qualified star after repeated pipe SETs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the conflict in ColumnResolutionHelper by taking upstream's rule from SPARK-60462, which supersedes this branch's earlier change to the same predicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ame column resolution SPARK-60462 tags a plan id candidate as hidden at a node that introduces its attribute in its metadataOutput, even when the attribute is also in the node's output. Pipe SET introduces its whole source row as hidden output, to keep the column order of a qualified star, so the source columns it does not assign became hidden at the SET Project. A Spark Connect DataFrame used directly and also under a USING join that hides one of its columns then failed with AMBIGUOUS_COLUMN_REFERENCE instead of resolving to the direct column. Record the source columns that the SET Project keeps visible in the new Project.visibleHiddenOutputTag, and leave them out of the hidden attributes the node introduces. The columns it assigns stay hidden, also once AddMetadataColumns adds them to its output. Replace the AnalysisSuite case that built the SET Project by hand with one that analyzes pipe SET and reuses the DataFrame that way. It fails with AMBIGUOUS_COLUMN_REFERENCE without this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What changes were proposed in this pull request?
The SQL pipe
SEToperator is built inAstBuilder.visitOperatorPipeSetas a starexpansion that excludes the assigned column and appends a replacement of the same name:
The excluded source attribute is dropped from the project list entirely, so nothing is
left for a later qualified reference such as
t.ato resolve to.This PR retains the excluded attribute as the hidden output of the
ProjectthatSETbuilds, using the existing
Project.hiddenOutputTagmechanism that USING joins alreadyuse to hide their duplicated join keys. Concretely:
UnresolvedStarExceptOrReplacegainsretainExceptedColumnsAsHidden(defaultfalse),so that ordinary
SELECT * EXCEPT/* REPLACEis unaffected. At the point of starexpansion the two are otherwise indistinguishable.
visitOperatorPipeSetsets it totrue.ResolveReferences, in the same place it expands the star and builds the newProject,records the excluded attributes on
Project.hiddenOutputTag, mirroringcommonNaturalJoinProcessingandResolveAsOfJoin.The retained attribute is spliced back into the project list by the existing
AddMetadataColumnsrule, and only when it is actually referenced:The docs already describe this behavior, so no wording change was needed. The
SETsection did contain the same example twice by accident; the duplicate is replaced with one
that demonstrates the alias retention. Happy to drop that hunk if it is considered out of
scope.
Why are the changes needed?
docs/sql-pipe-syntax.mddocuments that after aSETassignment "top-level column namesare updated but table aliases still refer to the original row values". That holds for
columns
SETdid not touch, but not for the assigned column itself:The suggestion list shows the asymmetry:
aresolves andt.bresolves, butt.adoes not. SPARK-50772 (#49420) introduced the alias retention for
SET/EXTEND/DROP,but its regression scenario reads only untouched qualified source columns; it does not
read back a column that
SETassigned.This blocks the documented before/after use of pipe
SET-- auditing, computing a deltaagainst the source value, and join disambiguation.
Does this PR introduce any user-facing change?
Yes. A qualified reference to a column affected by pipe
SETnow resolves to the originalinput value instead of failing analysis. This is a change relative to master and to
released versions; it makes the behavior match the documentation.
Before:
After:
The output schema of
SETitself is unchanged; the retained column is hidden and is onlymaterialized when referenced by name.
How was this patch tested?
Golden file tests added to
pipe-operators.sql, covering the reported repro, reading thenew and original values together, that
SELECT *still yields the original schema,qualified access through an alias introduced by
|> AS, and a sequence of twoSETs onthe same column.
The full
SQLQueryTestSuitepasses, which covers the existingSELECT * EXCEPT/* REPLACEgolden files. The regenerated golden diff contains only additions; no existingexpected output changed.
Was this patch authored or co-authored using generative AI tooling?
Yes.