Skip to content

[SPARK-59146][SQL] Retain qualified access to source columns affected by pipe SET - #58760

Open
AMC-hawk wants to merge 7 commits into
apache:masterfrom
AMC-hawk:SPARK-59146
Open

AMC-hawk wants to merge 7 commits into
apache:masterfrom
AMC-hawk:SPARK-59146

Conversation

@AMC-hawk

Copy link
Copy Markdown

What changes were proposed in this pull request?

The SQL pipe SET operator is built in AstBuilder.visitOperatorPipeSet as a star
expansion that excludes the assigned column and appends a replacement of the same name:

UnresolvedStarExceptOrReplace(
  target = None, excepts = Seq(Seq(ident)), replacements = Some(Seq(replacement)))

The excluded source attribute is dropped from the project list entirely, so nothing is
left for a later qualified reference such as t.a to resolve to.

This PR retains the excluded attribute as the hidden output of the Project that SET
builds, using the existing Project.hiddenOutputTag mechanism that USING joins already
use to hide their duplicated join keys. Concretely:

  • UnresolvedStarExceptOrReplace gains retainExceptedColumnsAsHidden (default false),
    so that ordinary SELECT * EXCEPT / * REPLACE is unaffected. At the point of star
    expansion the two are otherwise indistinguishable.
  • visitOperatorPipeSet sets it to true.
  • ResolveReferences, in the same place it expands the star and builds the new Project,
    records the excluded attributes on Project.hiddenOutputTag, mirroring
    commonNaturalJoinProcessing and ResolveAsOfJoin.

The retained attribute is spliced back into the project list by the existing
AddMetadataColumns rule, and only when it is actually referenced:

-- |> SELECT t.a
Project [a#x]
+- Project [(a#x + 1) AS a#x, b#x, a#x]      <- original retained on demand
   +- SubqueryAlias t
      +- LocalRelation [a#x, b#x]

-- |> SELECT *
Project [a#x, b#x]
+- Project [(a#x + 1) AS a#x, b#x]           <- unchanged, no extra column
   +- SubqueryAlias t
      +- LocalRelation [a#x, b#x]

The docs already describe this behavior, so no wording change was needed. The SET
section did contain the same example twice by accident; the duplicate is replaced with one
that demonstrates the alias retention. Happy to drop that hunk if it is considered out of
scope.

Why are the changes needed?

docs/sql-pipe-syntax.md documents that after a SET assignment "top-level column names
are updated but table aliases still refer to the original row values". That holds for
columns SET did not touch, but not for the assigned column itself:

VALUES (1, 10) AS t(a, b)
|> SET a = a + 1
|> SELECT t.a;
[UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or function parameter with name
`t`.`a` cannot be resolved. Did you mean one of the following?
[`a`, `t`.`b`]. SQLSTATE: 42703

The suggestion list shows the asymmetry: a resolves and t.b resolves, but t.a
does not. SPARK-50772 (#49420) introduced the alias retention for SET/EXTEND/DROP,
but its regression scenario reads only untouched qualified source columns; it does not
read back a column that SET assigned.

This blocks the documented before/after use of pipe SET -- auditing, computing a delta
against the source value, and join disambiguation.

Does this PR introduce any user-facing change?

Yes. A qualified reference to a column affected by pipe SET now resolves to the original
input value instead of failing analysis. This is a change relative to master and to
released versions; it makes the behavior match the documentation.

Before:

VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT t.a;
-- [UNRESOLVED_COLUMN.WITH_SUGGESTION] ... `t`.`a` cannot be resolved

After:

VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT t.a;
-- 1

VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT a, t.a, t.b;
-- 2  1  10

The output schema of SET itself is unchanged; the retained column is hidden and is only
materialized when referenced by name.

How was this patch tested?

Golden file tests added to pipe-operators.sql, covering the reported repro, reading the
new and original values together, that SELECT * still yields the original schema,
qualified access through an alias introduced by |> AS, and a sequence of two SETs on
the same column.

The full SQLQueryTestSuite passes, which covers the existing SELECT * EXCEPT /
* REPLACE golden files. The regenerated golden diff contains only additions; no existing
expected output changed.

Was this patch authored or co-authored using generative AI tooling?

Yes.

… by pipe SET

The pipe SET operator is built as a star expansion that excludes the assigned
column and appends a replacement of the same name, so the original attribute is
dropped from the project list and a later qualified reference such as `t.a` has
nothing to resolve to. This contradicts the documented behavior that table
aliases keep referring to the original row values after an assignment.

Retain the excluded attribute as hidden output on the Project that SET builds,
via the existing Project.hiddenOutputTag mechanism that USING joins use for
duplicated join keys. AddMetadataColumns splices it back only when referenced,
so the output schema of SET and the plan for `|> SELECT *` are unchanged.

The retention is flagged from the parser because pipe SET and regular
`SELECT * EXCEPT` / `* REPLACE` produce an identical node at star-expansion
time, and the latter must not be affected.
@AMC-hawk

Copy link
Copy Markdown
Author

I’m not sure who would be the right reviewers for this PR.

Adding @uros-b and @cloud-fan, could you please help out.

@AMC-hawk

Copy link
Copy Markdown
Author

@dongjoon-hyun Can you please take a look and guide

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

This needs changes before it is ready. The new SET hidden-output path can silently reorder the original row returned by a qualified star, and installing its hidden-output tag before normal tag forwarding drops Spark Connect plan identity from a rewritten Project. The scalar qualified-reference coverage does not exercise either qualified-star ordering or a pre-tagged Connect relation.

Findings

2 total: 0 P0, 1 P1, 1 P2, 0 P3.

Blocking (P1)

  • Preserve source column order in qualified-star expansion — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2107 — see inline.

Non-blocking (P2)

  • Preserve the original Project tags when adding hidden output — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2105 — see inline.

if (retain) {
val excepted = expanded.child.output.filterNot(expanded.outputSet.contains)
if (excepted.nonEmpty) {
expanded.setTagValue(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (P2): transformUpWithPruning normally forwards the original Project tags after this rewrite, but copyTagsFrom only does so when the destination tag map is empty. Setting hiddenOutputTag here first makes the map nonempty, so a Spark Connect relation rooted at pipe SET loses PLAN_ID_TAG; DataFrame columns bound to that relation can then fail with CANNOT_RESOLVE_DATAFRAME_COLUMN. Please preserve or merge the original project's tags when adding this tag.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot @cloud-fan for catching both the issues. I've addressed them in d092103.

Moved copyTagsFrom to run before setting hiddenOutputTag. Previously, copying tags afterwards was overwriting PLAN_ID_TAG.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on the current commit: the original Project tags are copied before hiddenOutputTag is installed, and the focused analyzer test preserves both PLAN_ID_TAG and hiddenOutputTag. This issue is resolved.

if (excepted.nonEmpty) {
expanded.setTagValue(
Project.hiddenOutputTag,
excepted.map(_.markAsQualifiedAccessOnly()) ++ expanded.child.metadataOutput)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (P1): UnresolvedStar expands qualified-only metadataOutput before visible output, so prepending the retained attributes here does not preserve the aliased row's column order. For example, VALUES (1, 2, 3) AS t(a, b, c) |> SET b = 20 |> SELECT t.* expands as (b, a, c) and returns (2, 1, 3) instead of (1, 2, 3). Please construct the retained output so t.* follows the original source order, including after repeated SETs.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

The SET Project now retains the full qualified source row, in its original order, as hidden output.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on the current commit: t.* now follows the original source-row order, including across repeated SET operations. This issue is resolved.

@cloud-fan

Copy link
Copy Markdown
Contributor

Thanks for the ping. I've reviewed the current SQL pipe SET patch and will leave the technical feedback on this PR.

…T hidden output

Address review comments:

- Keep the original column order for a qualified star after pipe SET. The SET
  Project now retains the whole qualified source row, in order, as hidden
  output, and the qualified star expansion emits an attribute that is both
  hidden and visible once, at its hidden position. Previously
  `VALUES (1, 2, 3) AS t(a, b, c) |> SET b = 20 |> SELECT t.*` returned
  (2, 1, 3). DataFrame column resolution treats an attribute present in both
  output and metadataOutput as a regular match.

- Copy the original Project tags before setting the hidden output tag, since
  tag forwarding only applies to a node without tags. This preserves the
  Spark Connect plan id of a relation rooted at pipe SET.

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

The SET source-row carrier needs to preserve the exact ordered row and remain durable until referenced attributes are materialized; fixing only star ordering or selected tag copies leaves that shared analyzer boundary unsound. The current revision resolves the earlier single-column ordering and immediate PLAN_ID_TAG overwrite issues, but it still collapses repeated row positions, depends on an auxiliary tag across analyzer iterations, and does not test the changed plan-ID visible/hidden classification. The repeated-position issue is blocking because it silently changes a valid query's schema and row shape.

Findings

3 total: 0 P0, 1 P1, 2 P2, 0 P3.

Blocking (P1)

  • Preserve repeated source-row positions in qualified stars — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2115 — see inline.

Non-blocking (P2)

  • Make retained SET state durable until materialization — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2120 — see inline.
  • Exercise plan-ID resolution for visible/hidden overlap — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:694 — see inline.

Shared repair plans

Shared repair plan 1

Covered findings:

  • Preserve repeated source-row positions in qualified stars — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2115
  • Make retained SET state durable until materialization — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2120

Recommended change: Represent the retained SET source row as the exact ordered positional sequence and materialize any referenced source attributes within the same bounded analysis operation that resolves their parent references, so later rules no longer depend on an auxiliary tag for correctness. Add SQL analyzer coverage for repeated ExprIds and preservation through the full SET resolution path.

Why this works: Remove ExprId-based sequence deduplication and set-wide visible suppression in favor of occurrence-preserving matching. After parent expressions resolve through the retained row, perform narrowly scoped SET metadata materialization before ResolveReferences returns, leaving the hidden tag as discovery metadata rather than the sole cross-iteration correctness carrier.

Scope: Rework the Catalyst SET retention carrier and qualified-star merge, then extend pipe-operator SQL regression coverage for multiplicity and analyzer-lifetime behavior.

Compatibility: Pipe SET continues to expose assigned values through top-level names and original values only through pre-existing qualifiers, without widening its visible schema.

Risks: Eagerly materializing retained attributes too broadly could leak hidden columns into SET output or unqualified stars. Occurrence matching must preserve qualifiers and metadata while distinguishing repeated positions that share an ExprId.

Constraints: Ordinary SELECT * EXCEPT and * REPLACE must remain unchanged. USING/NATURAL and AS OF join hidden-output behavior must remain unchanged. The visible output schema of pipe SET and unqualified SELECT * must remain unchanged.

Success: Qualified t.* after SET returns every original aliased row position in order, including repeated positions with the same ExprId. Qualified scalar references to assigned source columns are backed by produced child attributes before subsequent analyzer rules run. Unqualified SET results and ordinary star-except behavior retain their existing schemas and values.

Re-review status

Prior AI findings: 2 addressed, 0 still present; additional unresolved findings in this review: 3.

New attribution: 2 newly introduced, 1 late catch, 0 previously raised, 0 unattributed.

Remaining prior AI findings

No prior AI findings remain.

// then its output. Only qualified attributes are reachable through a qualified star.
val sourceRow = (child.metadataOutput.filter(_.qualifiedAccessOnly) ++ child.output)
.filter(_.qualifier.nonEmpty)
.distinctBy(_.exprId)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (P1): Project output is positional, so repeated projections can legitimately share an ExprId. Here distinctBy(_.exprId) drops one such position, and the later ExprId set removes every matching visible occurrence. For example, VALUES (1, 2) AS s(a, b) |> SELECT a, a, b |> AS t |> SET b = 3 |> SELECT t.* expands to [a, b] instead of [a, a, b], silently changing the schema and row shape. Please preserve source-row occurrences rather than using ExprId as positional identity.

See Shared repair plan 1 in the review body.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan Done, Fixed in 262b18bbe7d

distinctBy(_.exprId) and the set-wide visible suppression are gone. t.* now returns (a, a, b) / 1 1 2

// The rule forwards the original tags, such as the Spark Connect plan id, only onto a
// node without tags, so copy them here before adding the hidden output tag.
expanded.copyTagsFrom(original)
expanded.setTagValue(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (P2): This tag becomes correctness-critical across analyzer iterations: ResolveReferences can resolve t.a from metadata only after AddMetadataColumns has already run, so the attribute is materialized on a later iteration. Generic copyTagsFrom calls do not guarantee that hiddenOutputTag survives every intervening Project replacement. Please keep the retained source-row state at a durable owner, or materialize referenced source attributes within a bounded operation, so a documented qualified SET reference cannot lose its producing child.

See Shared repair plan 1 in the review body.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preference would be to retain hiddenOutputTag as the carrier, consistent with the existing analyzer lifecycle, added golden coverage across the relevant projection-rebuilding rules in 262b18bbe7d, and happy to add a case if any rule does not forward the tag correctly.

// counts as a regular match.
val filtered = candidates.flatMap { c =>
val hidden = c.hidden || c.expr.references.subsetOf(AttributeSet(p.metadataOutput))
val hidden = c.hidden || (!c.expr.references.subsetOf(p.outputSet) &&

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (P2): The new test stops at tag coexistence and never constructs a plan-ID-bound column reference that reaches this visible/hidden overlap. Reverting this predicate would therefore leave the added assertions green while allowing the Spark Connect/DataFrame-column failure to return. Please add a focused analyzer case that invokes plan-ID resolution for an attribute present in both outputs and verifies that the visible candidate wins.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, the earlier test was redundant. Replaced in 262b18bbe7d.

…T hidden output

Address review comments:

- Keep every position of the source row that pipe SET retains. A projection
  may repeat a column, and the repeated positions share an ExprId, so using
  the ExprId as positional identity dropped one of them:
  `VALUES (1, 2) AS s(a, b) |> SELECT a, a, b |> AS t |> SET b = 3
  |> SELECT t.*` returned (a, b) instead of (a, a, b). Retention and
  qualified star expansion now match occurrences through the new
  UnresolvedStarBase.mergeHiddenAndVisibleOutput, where a hidden attribute
  consumes at most one visible occurrence and is emitted in its place.

- Replace the AnalysisSuite case for DataFrame column resolution, which only
  asserted that the two Project tags coexist. It now resolves a plan id bound
  column across a visible and hidden overlap, which fails if the visible
  candidate is classified as hidden.

- Add golden coverage for a qualified SET reference under the rules that
  rebuild the projection above it: filter, sort, aggregate, join, union, an
  enclosing subquery and a CTE.
@AMC-hawk

AMC-hawk commented Sep 23, 2026 •

Copy link
Copy Markdown
Author

@cloud-fan Can you please take a look

@AMC-hawk

AMC-hawk commented Oct 2, 2026 •

Copy link
Copy Markdown
Author

@cloud-fan Addressed in 262b18b:

  • P1 repeated positions: fixed, t.* now returns (a, a, b).
  • P2 plan-ID test: added an analyzer case where the visible attribute wins.
  • P2 durable state: kept hiddenOutputTag and added golden coverage. Happy to move it to a durable owner if you prefer.

CI green, no conflicts. cc @dtenedor

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @AMC-hawk and @cloud-fan for working on this!

@uros-b
uros-b requested a review from cloud-fan October 5, 2026 13:11
@uros-b

uros-b commented Oct 5, 2026

Copy link
Copy Markdown
Member

Let's ping @cloud-fan for re-review, I think this looks good now

@AMC-hawk

Copy link
Copy Markdown
Author

@cloud-fan Requesting re-review

@cloud-fan cloud-fan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

The current revision addresses the prior repeated-position and plan-ID coverage issues. Two newly introduced issues remain: the shared hidden/visible merge can make qualified-star analysis quadratic for wide schemas, including pre-existing USING/NATURAL-join queries, and a new ScalaDoc sentence mixes counterfactual and past tense. The previously raised hidden-tag lifetime concern has no reachable current tag-loss path and is not retained.

Findings

2 total: 0 P0, 0 P1, 1 P2, 1 P3.

Non-blocking (P2)

  • Avoid quadratic hidden/visible output merging — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala:653 — see inline.

Nit (P3)

  • Use consistent counterfactual tense in the ordering comment — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2100 — see inline.

Re-review status

Prior AI findings: 3 addressed, 0 still present; additional unresolved findings in this review: 2.

New attribution: 2 newly introduced, 0 late catch, 0 previously raised, 0 unattributed.

Remaining prior AI findings

No prior AI findings remain.

* The hidden output holds the whole qualified source row, with every position in its original
* order, not just the excluded attributes, because a qualified star expands hidden output
* ahead of the visible output. Retaining only the excluded attributes would move them to the
* front, so that `VALUES (1, 2, 3) AS t(a, b, c) |> SET b = 20 |> SELECT t.*` returned

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (P3): This sentence starts a counterfactual with would move but completes it with returned, which makes the hypothetical (b, a, c) result read like observed current behavior. Please use would return (or make both clauses past-counterfactual) so the ordering rationale is unambiguous.

Verification:

  • Inspection: Read the complete ordering comment and confirm its condition and consequence use consistent counterfactual grammar.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5545d0d: it now reads "would return (b, a, c)".

visibleOutput: Seq[Attribute]): Seq[Attribute] = {
val unconsumed = visibleOutput.to(mutable.ArrayBuffer)
val merged = hiddenOutput.map { hidden =>
val visible = unconsumed.indexWhere(_.exprId == hidden.exprId)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (P2): Each hidden attribute scans the remaining visible buffer, and every overlap is removed from the middle. With wide USING/NATURAL joins this performs H * V searches; with pipe SET's full-width overlap, the removals repeatedly shift the remaining suffix. Qualified-star analysis therefore becomes quadratic before planning for both existing join queries and the new SET path.

Verification:

  • Behavior: Compare merged output for disjoint, overlapping, repeated-ExprId, and repeated-SET inputs and assert exact order, multiplicity, qualifiers, and expression identities.
  • Regression: Measure analysis across increasing schema widths for pipe SET plus a qualified star and for a wide USING/NATURAL join plus a qualified star, demonstrating linear rather than quadratic growth.
  • Inspection: Confirm the merge performs one visible indexing pass, constant-time expected occurrence consumption, and one final visible-order emission pass.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 5545d0d.

  1. Inspection: the merge now indexes visible positions by ExprId in one pass, consumes each in O(1) expected time, and emits the leftovers in one final pass.

  2. Behavior: new AnalysisSuite cases cover disjoint, overlapping, repeated-ExprId and repeated-SET inputs, asserting exact instances. The new merge also matched the old one on 20k random inputs.

  3. Regression: this also turned up AddMetadataColumns scanning the children's metadataOutput once per attribute, which made any wide SET quadratic. It now looks attributes up by ExprId. Two SETs + t.* at 16k columns went from 11.3 s to 0.33 s, now growing linearly. A qualified star over a wide USING/NATURAL join also grows linearly now.

@cloud-fan

Copy link
Copy Markdown
Contributor

I re-reviewed the current commit. The repeated-position, retained-state, and plan-ID changes are supported by the current analyzer paths and coverage. One issue remains: the new hidden/visible merge is quadratic for wide schemas, including existing qualified-star queries above wide USING/NATURAL joins. Please linearize that merge; there is also a one-word counterfactual tense fix in the new ScalaDoc.

AMC-hawk and others added 4 commits October 20, 2026 23:58
…idden output

Address review comments:

- Make UnresolvedStarBase.mergeHiddenAndVisibleOutput linear. It searched the
  remaining visible output for every hidden attribute and removed each match
  from the middle of a buffer, so a qualified star over a wide USING or
  NATURAL join, or after pipe SET, was quadratic in the schema width. It now
  indexes the visible occurrences by ExprId in one pass, consumes them in
  constant expected time and emits the unconsumed ones in one final pass,
  with the same output.

- Look up the metadata output of the children by ExprId in
  AddMetadataColumns. It scanned that output for every attribute of a node,
  and pipe SET retains the whole source row as hidden output, so analyzing
  any wide pipe SET query was quadratic, with or without a qualified star.

- Use a consistent counterfactual in the comment on
  retainExceptedColumnsAsHiddenOutput.

- Add AnalysisSuite cases for the merge on disjoint, overlapping and repeated
  inputs, checking the exact instances it emits, and for a qualified star
  after repeated pipe SETs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the conflict in ColumnResolutionHelper by taking upstream's rule from
SPARK-60462, which supersedes this branch's earlier change to the same
predicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ame column resolution

SPARK-60462 tags a plan id candidate as hidden at a node that introduces its
attribute in its metadataOutput, even when the attribute is also in the
node's output. Pipe SET introduces its whole source row as hidden output, to
keep the column order of a qualified star, so the source columns it does not
assign became hidden at the SET Project. A Spark Connect DataFrame used
directly and also under a USING join that hides one of its columns then
failed with AMBIGUOUS_COLUMN_REFERENCE instead of resolving to the direct
column.

Record the source columns that the SET Project keeps visible in the new
Project.visibleHiddenOutputTag, and leave them out of the hidden attributes
the node introduces. The columns it assigns stay hidden, also once
AddMetadataColumns adds them to its output.

Replace the AnalysisSuite case that built the SET Project by hand with one
that analyzes pipe SET and reuses the DataFrame that way. It fails with
AMBIGUOUS_COLUMN_REFERENCE without this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants