[896] Shrink the xtable-utilities bundled jar from 1.1 GB to 423 MB

[slachiewicz]

Closes #896.

What is the purpose of the pull request

xtable-utilities builds a 1.1 GB bundled jar from 795 lines of Java. This cuts it to 423 MB without touching any source.

Brief change log

• Replaced software.amazon.awssdk:bundle, the all-services jar, with the four artifacts hadoop-aws S3A actually uses: s3, sts, netty-nio-client, s3-transfer-manager. The AWS SDK goes from 1957 MB to 50 MB uncompressed.
• Root pom now imports the AWS BOM rather than managing SDK artifacts one at a time.
• Dropped RocksDB natives for ppc64le, s390x and 32-bit Linux, 84 MB. x86_64 and aarch64 keep their musl builds; macOS and Windows stay.
• Dropped io.grpc:grpc-xds (40 MB) and org.codehaus.groovy:groovy-all (9.5 MB) from the shade whitelist.

	jar	entries
before	1098 MB	582,980
after	423 MB	203,041

One detail worth flagging for reviewers: the RocksDB excludes had to go in the root pom's shade execution, not the module's plugin configuration. An execution-level <filters> replaces a plugin-level one instead of merging, so a filter added in the module is silently ignored — the build passes and nothing is filtered.

Verify this pull request

The AWS reduction was derived rather than guessed: I scanned the constant pools of all 418 org.apache.hadoop.fs.s3a classes in hadoop-aws 3.4.1. They reference exactly one AWS service package, services/s3, plus STSClient and the transfer, http, core, awscore and auth packages. No other AWS service appears.

This has not been run against live S3, Azure or GCS. Static analysis cannot see a credentials provider or signer named by class in configuration, so a deployment setting fs.s3a.aws.credentials.provider to an AWS SDK class directly could still hit NoClassDefFoundError — the same failure mode as #736. I would like someone to run RunSync against a real bucket before this merges.

Builds green on Temurin 11; spotless:check clean.

This change was created with AI assistance.

[slachiewicz]

jdeps on the built jar found a gap the earlier analysis missed, fixed in 16778bb.

s3 and sts pull arns, aws-query-protocol, aws-xml-protocol and crt-core. All four were in the old bundle, but the shade <artifactSet><includes> is an explicit whitelist, so naming s3 as a dependency is not enough — every transitive artifact has to be listed or shade drops it. The S3 and STS clients would have failed at runtime.

Also: hadoop-aws brings software.amazon.awssdk:bundle:2.24.6 transitively. The direct 2.29.40 declaration had been masking it, so removing that let an older fat bundle back into the tree. Now excluded explicitly.

After the fix the package counts match the original bundle exactly — arns 9, protocols/xml 67, protocols/query 50, services/s3/transform 99, services/sts 357 — and nothing in the runtime tree is left unwhitelisted. Jar stays at 423 MB.

One thing jdeps flags that is not a regression: software.amazon.awssdk.crt.*. The AWS Common Runtime is a separate software.amazon.awssdk.crt:aws-crt artifact, absent from bundle as well, so XTable has never shipped it. Unchanged by this PR.

The live S3 run is still worth doing before merge.

This comment was created with AI assistance.

[slachiewicz]

Tested this against live S3, live ADLS Gen2 and live GCS. No regression found.

Differential: baseline jar at ef88e02 (the commit under this PR) versus this PR's jar, identical
configs and identical cloud paths. EC2 m6i.xlarge in eu-north-1, Corretto 11, AWS credentials
from an instance profile so no static key was involved.

Sizes confirm the table in the description exactly: baseline 1,097,347,589 bytes, this PR
423,222,450.

ServiceLoader diff — the #736 failure shape. 131 of 137 service files are byte-identical.
Exactly six differ, and all six are intended drops:

io.grpc.xds.XdsCredentialsProvider
javax.script.ScriptEngineFactory
org.codehaus.groovy.plugins.Runners
org.codehaus.groovy.runtime.ExtensionModule
org.codehaus.groovy.source.Extensions
org.codehaus.groovy.transform.ASTTransformation

No AWS SDK registration was lost. javax.script.ScriptEngineFactory is the Groovy JSR-223
engine leaving with groovy-all.

AWS, real bucket:

run	this PR	baseline
RunSync DELTA→ICEBERG over s3a://, shipped default creds	pass	pass
RunSync over s3a://, assume-role config	pass	not run
write+read, DefaultCredentialsProvider	pass	—
write+read, AWS SDK provider named by class in config	pass	—
write+read, AssumedRoleCredentialProvider	pass	pass
9 MB upload at 5 MB multipart threshold	pass	—

-verbose:class shows the swapped artifacts are genuinely reached, not merely present: sts
195 classes including DefaultStsClient, AssumeRoleRequest and s3a.auth.STSClientFactory;
s3-transfer-manager 73; netty-nio-client 194. software.amazon.awssdk.crt loads 0 classes,
matching your note that CRT was never shipped.

Azure, real ADLS Gen2 with hierarchical namespace, through the shipped
OAuth/ClientCredsTokenProvider default with a scoped service principal: ABFS
list/mkdir/write/read pass, and RunSync DELTA→ICEBERG over abfss:// passes on both jars.

GCS, real bucket, service-account JSON: RunSync DELTA→ICEBERG over gs:// passes on both
jars. Worth noting for anyone reasoning about blast radius — a GCS operation loads 0
software.amazon.awssdk classes, so the AWS artifact swap cannot reach that path.

Re-tested rebased onto #906

The first pass needed a workaround: the bundled jar cannot read Delta sources at all, because
shade had no ServicesResourceTransformer and Delta's DataSourceRegister entry was being
overwritten. That is pre-existing and unrelated to this PR — the service file is byte-identical
in both jars — and is now #905, fixed by #906.

I rebased this branch onto #906 and re-ran everything with no workaround. Clean rebase, no
conflicts; the two PRs touch different sections of xtable-utilities/pom.xml. Combined jar is
423,224,546 bytes, so the size win survives at a cost of about 2 KB of merged service files.
RunSync over s3a:// and with assume-role both pass, the credential matrix passes against a
config that no longer needs fs.file.impl, and 0 of 428 ServiceLoader registrations dangle.

One interaction worth flagging for reviewers. With #906 merging service files, this PR's
grpc-xds removal becomes visible where it previously was not:
io.grpc.LoadBalancerProvider and io.grpc.NameResolverProvider lose their 11 io.grpc.xds.*
entries. Before #906 those entries were being clobbered anyway, so nothing changed observably.
Classes and registrations go together here — 4,805 io/grpc/xds classes to 0, 1,398 groovy to 0
— so nothing is left dangling, and the DNS, grpclb, pick-first, round-robin, RLS and
GoogleCloudToProd resolvers all remain. Only xds:// targets are affected, which is the stated
intent.

What this does not settle

The gap in the description is narrowed, not closed. Three credential providers named by class
resolved, including an AWS SDK class named directly in configuration. What no amount of testing
here can enumerate is an arbitrary provider or signer some deployment names that falls outside
the whitelist.

Also untested: Hudi in either direction, RunCatalogSync, and whether the RocksDB natives this
PR trims are reachable from RunSync at all — nothing I ran loaded RocksDB.

This comment was created with AI assistance.