Skip to content

[KYUUBI #6314][SPARK] Preserve lineage for JDBC temporary view inserts - #7791

Open
wayne-x-wang wants to merge 1 commit into
apache:masterfrom
wayne-x-wang:kyuubi-6314-jdbc-temp-view
Open

wayne-x-wang wants to merge 1 commit into
apache:masterfrom
wayne-x-wang:kyuubi-6314-jdbc-temp-view

Conversation

@wayne-x-wang

Copy link
Copy Markdown
Contributor

Why are the changes needed?

Inserts into JDBC temporary views resolve to InsertIntoDataSourceCommand with no target catalogTable. The guarded map throws a MatchError, so lineage extraction fails even though the write succeeds.

Preserve query column names and source lineage when target catalog metadata is absent, matching the existing SaveIntoDataSourceCommand behavior. outputTables remains empty; this change does not infer a JDBC destination identity. Catalog-backed inserts keep their existing behavior.

Fixes #6314.

How was this patch tested?

Added an embedded Derby regression that creates a JDBC temporary view through SQL, executes an insert, reads the stored row through JDBC, and checks the command type, absent catalogTable, source tables, column mappings, and empty output tables. It checks both parse and transformToLineage. Without the fix, two separate JVM runs failed with the same MatchError. Existing tests cover inserts with target catalog metadata.

Spark 3.5.8 / Scala 2.12: all 77 lineage tests passed.

build/mvn -B -ntp test \
  -pl extensions/spark/kyuubi-spark-lineage -am \
  -Pspark-3.5,spark-provided,flink-provided,hive-provided \
  -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.lineage

Spark 4.2.0 / Scala 2.13: the JDBC temporary-view regression passed.

build/mvn -B -ntp test \
  -pl extensions/spark/kyuubi-spark-lineage -am \
  -Pspark-4.2,scala-2.13,spark-provided,flink-provided,hive-provided \
  -Dtest=none \
  '-Dtests=@columns lineage extract - insert into a JDBC temporary view without catalog metadata' \
  -Dsuffixes=TableCatalogLineageParserSuite

dev/reformat, Spotless, Scalastyle, and git diff --check passed. The historical Kyuubi 1.8.0 / Spark 3.2.3 combination and MySQL were not tested.

Was this patch assisted by generative AI tooling?

Assisted-by: Codex (GPT-6)

@github-actions

Copy link
Copy Markdown

Hello @wayne-x-wang,
Thanks for finding the time to submit the Pull request, the Kyuubi community thanks you!
We really appreciate the community's efforts to improve Apache Kyuubi.
If you are new to Apache Kyuubi for code changes, please take some time to read our contributing guide.

@wayne-x-wang

Copy link
Copy Markdown
Contributor Author

Could a maintainer please re-run the failed jobs? All 77 lineage tests passed. The two failures are in EngineRefWithZookeeperSuite and the ZooKeeper 3.7 integration test, outside this patch’s changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Insert into a jdbc table will cause exception when parse lineage in spark

1 participant