[KYUUBI #7206][LINEAGE] Collect input tables from filter subqueries

[wayne-x-wang]

Why are the changes needed?

Tables used only by filter subqueries are missing from input lineage. For example:

INSERT OVERWRITE v2_catalog.db.tb3
SELECT t1.* FROM v2_catalog.db.tb1 t1
WHERE EXISTS (
  SELECT 1 FROM v2_catalog.db.tb2 t2 WHERE t2.col1 = t1.col1
);

Before this change, the analyzed plan contains tb2 under Filter.condition, but lineage lists only tb1. Ordinary children traversal does not visit the subquery plan.

Visit SubqueryExpression.plan in filters and collect its input tables without merging its column mappings. Output columns keep their existing sources. The existing spark.kyuubi.plugin.lineage.legacy.collectInputTablesByColumn=true mode retains its previous input-table semantics. No new configuration or dependency is introduced.

This follows the traversal approach discussed in #7207 while avoiding the additional configuration questioned in review. Five existing input-table expectations are corrected; their column mappings are unchanged.

Fixes #7206.

How was this patch tested?

The correlated EXISTS regression failed in two independent JVM runs on the unmodified master implementation (Spark 3.5.8):

Set("v2_catalog.db.tb1") did not equal Set("v2_catalog.db.tb1", "v2_catalog.db.tb2")

Its output-table and three column-source assertions passed before the input-table assertion failed. With the regression added, the pre-fix lineage suite had 75 passing tests and the same failure in both catalog suites.

Added 36 compatibility cases: nine scenarios × two legacy settings × two catalog suites. These cover EXISTS, NOT EXISTS, IN, NOT IN, scalar subqueries, HAVING, nested subqueries, composite conditions and a filter without subqueries. They check input/output tables and column mappings, and restore the previous configuration in finally. Tests use Spark's parser/analyzer and the existing in-memory V2 catalogs, matching the listener's analyzed-plan input.

After the fix, all 113 lineage tests passed on each matrix entry:

Spark	Scala	Passed	Failed
3.5.8	2.12.19	113	0
4.2.0	2.13.17	113	0

Both runs also passed Spotless and Scalastyle. Commands below omit only the local Maven mirror settings path:

build/mvn -B -ntp test spotless:check scalastyle:check \
  -pl extensions/spark/kyuubi-spark-lineage -am \
  -Pspark-3.5,spark-provided,flink-provided,hive-provided \
  -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.lineage

build/mvn -B -ntp test spotless:check scalastyle:check \
  -pl extensions/spark/kyuubi-spark-lineage -am \
  -Pspark-4.2,scala-2.13,spark-provided,flink-provided,hive-provided \
  -Dtest=none -DwildcardSuites=org.apache.kyuubi.plugin.lineage

dev/reformat and git diff --check passed. The final source hashes match the files used in both successful test runs.

Was this patch assisted by generative AI tooling?

Assisted-by: Codex (GPT-6)

[github-actions]

Hello @wayne-x-wang,
Thanks for finding the time to submit the Pull request, the Kyuubi community thanks you！
We really appreciate the community's efforts to improve Apache Kyuubi.
If you are new to Apache Kyuubi for code changes, please take some time to read our contributing guide.