[722] Implement iceberg versions for TableFormat and HoodieTableMetadata - #723
[722] Implement iceberg versions for TableFormat and HoodieTableMetadata#723bvaradar wants to merge 4 commits into
Conversation
| <spark.version>3.5.2</spark.version> | ||
| <spark.version.prefix>3.5</spark.version.prefix> | ||
| <iceberg.version>1.5.2</iceberg.version> | ||
| <delta.version>3.0.0</delta.version> |
There was a problem hiding this comment.
Can we pull the spark, iceberg, and delta version upgrades into their own PR? It would help shrink the size of this PR
There was a problem hiding this comment.
+1, we can split it into version upgrades and iceberg pluggable tf PR's
|
Hi @bvaradar, are you still working on this, and are there any in-progress changes on your side? I'm interested in taking this forward. |
|
@yihua I will get this PR rebased on top of latest main with hudi 1.2.0 version pinned as I have context on the changes made as part of the 1.x upgrade. After that you can take over. |
The pluggable table format landed in apache/hudi#13216 and ships in Hudi 1.2.0, which main already depends on. The merged SPI differs from the pre-merge API this module was written against: - org.apache.hudi.common.TableFormat is now HoodieTableFormat, so the META-INF/services resource is renamed to match. - TimelineFactory adds an abstract createArchivedTimeline(metaClient, boolean) and drops createCompletionTimeQueryView(metaClient, String). - HoodieTableFormat.archive() takes a Supplier<List<HoodieInstant>>. - HoodieAvroUtils.addMetadataFields moved to HoodieSchemaUtils and now operates on HoodieSchema. Also align the module with current main: the parent POM version, the scala-suffixed artifactId, and the PathBasedPartitionSpecExtractor and PathBasedPartitionValuesExtractor renames. TableSyncMetadata keeps a four-argument of() overload so existing callers do not change. Add delta-core to the module test scope, because the shared HudiTestUtil.getSparkConf registers the Delta catalog and extension.
…ormat The module's integration tests never activated the plugin, because nothing set hoodie.table.format. PR apache#723 worked around this by hardcoding the Iceberg format and disabling the metadata table inside the shared TestAbstractHudiTable, which would have changed every Hudi test table in xtable-core. Add an opt-in hook instead. TestJavaHudiTable.forStandardSchema now takes an optional Properties bag that a single test applies to one table. The bag reaches both hoodie.properties and the write config, so a test can also relax defaults its table format does not support. Three defaults now read from it rather than being hardcoded: the table version, the metadata table, and the column stats index. Existing callers pass an empty bag and are unaffected; xtable-core still passes 494 tests. Three defects surfaced once the hook let the plugin run: - HoodieCommitMetadata.getFullPathToInfo keys its map by the absolute path, but HudiDataFileExtractor.getDiffForCommit looked up the file name and passed the resulting null into HoodieBaseFile. Look up the absolute path, and fail with a clear message rather than a NullPointerException. - ConversionTargetFactory iterated the ServiceLoader directly, so a registered target whose engine is absent from the classpath aborted the lookup with a ServiceConfigurationError. Skip such providers, matching the fix already reviewed on PR apache#843. - IcebergTimelineFactory builds on the v2 timeline, so a table using this format needs table version 8, not the version 6 that xtable-core pins its other Hudi test tables to. Add two tests. TestIcebergTableFormatDiscovery checks that Hudi resolves IcebergTableFormat through the ServiceLoader and defaults to the native format otherwise. ITIcebergPluggableFormatSync proves the end to end contract: a plain Hudi write on a table configured with the Iceberg format produces a readable Iceberg snapshot with a matching row count, and no XTable sync job runs.
13fc241 to
3aac36c
Compare
|
Thanks @vinishjail97 |
|
Thanks @bvaradar for building this out, and @yihua for picking it up. Closing this in favour of #894, which carries the work forward. #894 contains all four commits from this branch with the original authorship intact, and adds savepoint, restore and rollback support along with more test coverage. Both branches share the same merge base ( @the-other-tim-brown's ask on this thread, to split the Spark, Iceberg and Delta version bumps into their own PR, is already handled. Neither this branch's current head nor #894 touches the root Further review on #894 please. |
4 participants
Important Read
ISS-722 Add Iceberg Pluggable table format managed directly by Hudi
What is the purpose of the pull request
ISS-722 Add Iceberg Pluggable table format managed directly by Hudi
Feature Request: Support for Pluggable Table Format in Apache Hudi
Apache Hudi is introducing a pluggable table format to enable interoperability with other popular table formats like Iceberg, Delta Lake, and Lance. This allows external engines (e.g., Snowflake) to write to Hudi tables using their native formats while maintaining compatibility. The design focuses on decoupling metadata and storage layers, inspired by Snowflake's metadata architecture. This enhancement supports the modular, composable lakehouse model and promotes broader ecosystem adoption.
This is the first PR
Corresponding Hudi PR : apache/hudi#13216
Brief change log
(for example:)
Verify this pull request
(Please pick either of the following options)
This pull request is a trivial rework / code cleanup without any test coverage.
(or)
This pull request is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:
(example:)