[865][701] Make the bundled-jar license metadata correct, and actually check it

[the-other-tim-brown]

What

Three commits, each fixing one thing, in the order they were found.

[865]	META-INF/licenses/ now matches the bundled non-Apache-2.0 dependencies, both ways, and a check enforces it
[772]	the xtable-hive-metastore shade whitelist regenerated for Hive 3.1.3
[701]	one canonical LICENSE/NOTICE per jar, one licenses/ directory, and a check enforcing that too

Closes #865. Closes #701.

The License Check has never run

validate_shaded_license_coverage.sh discovered modules with rg, which is not installed on
ubuntu-latest. Every run since #822 landed in June logged

release/scripts/validate_shaded_license_coverage.sh: line 92: rg: command not found
No modules with maven-shade-plugin were found.

and then exit 0. See the run for ef88e02.
Discovery now uses git ls-files, and an empty module list fails rather than passing.

This is the reason for the other two commits. Turning the check on is what surfaced everything below,
and it means any release-time conclusion drawn from a green License Check since June is unsupported.
validate_bundled_license_texts.py from #857 was running throughout, which is why the texts stayed
broadly right even while the dependency set did not.

[865] Orphaned and missing license texts

@zabetak's follow-up on #865
was against the released 0.4.0 jar. #857 had already removed the specific orphans he named from main,
but left gaps in the other direction, and nothing prevented either from recurring.

validate_shaded_license_coverage.sh gains a check that maps each shade whitelist entry to its family in
LICENSE-bundled and requires LICENSE-<artifactId> for everything outside Apache 2.0 — failing both on a
dependency with no text and on a text with no dependency behind it.

Nine texts added to satisfy it, each taken from the artifact itself. javax.transaction:jta and
transaction-api were filed under Apache 2.0 while #857 shipped correct CDDL texts for them; the
GROUP_OVERRIDES entry was what was wrong, so that is corrected rather than the texts deleted. Their texts
also carried cp1252 mojibake from the Sun originals, now normalised.

[772] The hive-metastore bundle was three years of Hive behind

Its <artifactSet><includes> still described the Hive 2.3.9 tree from before #772 raised hive.version to
3.1.3. 97 runtime dependencies were never shaded, and 25 listed artifacts no longer resolve — shade
skips those silently, which is why the jar shipped license texts with no classes behind them. Same symptom
as #865, different cause.

<includes> regenerated from the runtime tree, 182 → 254 entries, ${scala.binary.version} preserved, then
LICENSE-bundled and NOTICE-bundled regenerated from it. Two families had to be taught to the generator:
org.apache.hadoop, because hadoop-project's pom contains <Xlint:-unchecked/> and is not
namespace-well-formed, and EPL 1.0, because junit resolves to 4.12 under EPL, not the 4.11 under CPL the
stale metadata claimed.

16 texts added, 5 removed, and the three asm texts rebased from asm:asm:3.1 onto org.ow2.asm:9.6.

[701] Duplicate and scattered license files

Every jar carried its license twice: LICENSE-bundled and NOTICE-bundled are inputs to the shade
transformers, read from target/classes, but were packaged as resources as well. Since the transformers
never read the jar, maven-jar-plugin can simply exclude them, and both artifacts lose the duplicate.

The hive-metastore bundle also had three directories of license text: ours, the META-INF/license netty
ships, and a top-level license/ from xml-apis that neither #701 nor #885 spotted. Netty's ten texts are
checked in under META-INF/licenses and xml-apis' W3C and SAX notices are carried in
LICENSE.xml-apis-dom-sax.txt, so the originals are filtered out with nothing lost and one directory left.

validate_bundled_jar_license_layout.py enforces this against the built jars. It is deliberately separate
from validate_bundled_license_texts.py: that one checks which texts exist, this one checks where they
sit.

Running both found three wrong license families

Neither check could have caught these alone — the coverage check trusts LICENSE-bundled, which is
generated from these same overrides, so a wrong override is self-consistent and invisible to it.

artifact	was	is, per its own pom
javax.servlet.jsp-api	Apache 2.0	CDDL + GPLv2 with classpath exception — was shipping with no text at all
stax-api	CDDL + GPLv2	Apache 2.0
xml-apis	Apache SL 1.1	Apache 2.0 (pom silent, jar ships the Apache 2.0 text)

The last one answers the question #885 asked for a second opinion on and never got. No Apache Software
License 1.1 dependency remains in any bundle.

Credit

The license texts and validate_bundled_license_texts.py are @rangareddy's and @vinishjail97's work from
#857 and #885. Every point of #885's analysis that I initially doubted turned out to be correct — the
22-of-29 bundled measurement, junit being 4.12/EPL, and the xml-apis heading. The stale Hive metadata is
what made all three look wrong.

Testing

apache-rat:check                              Unapproved: 0 across all modules
validate_shaded_license_coverage.sh           exit 0
validate_bundled_license_texts.py             exit 0 — 5 / 42 / 3 non-AL2 deps, all with texts
validate_bundled_jar_license_layout.py        exit 0 — all 9 jars

Each new check was negative-tested: removing a required text, adding an unbacked one, injecting a duplicate
LICENSE-bundled and a second license directory each produce the expected failure and exit 1.

Note that -Dmaven.build.cache.enabled=false is needed to see packaging changes locally — the build cache
will serve a stale jar across a root-pom edit. CI's shaded-bundle step already passes it.

Not fixed here

• Multiple LICENSE files present in some jars #701 also asks that thin jars stop carrying META-INF/licenses/. They cannot: shade builds the
  bundled jar from the thin jar, so excluding the directory there removes it from the bundle too. Fixing it
  properly needs the texts moved out of src/main/resources and re-added via ~50 per-file shade
  transformers, which is worse than the problem.
• junit and hamcrest-core are bundled at all, arriving transitively at compile scope. Shipping test
  libraries in a production bundle is arguably the real defect; excluding them beats licensing them, but
  that is a behavioural change. [700] Ship a license text for every bundled non-Apache-2.0 dependency #885 raised this and it was never answered.
• branch-0.4 has all of the above, plus xtable-spark-runtime, which bundles checker-qual under MIT
  with no META-INF/licenses/ at all, and whose LICENSE-bundled is hand-written rather than generator
  output. It is an allowlist rather than a mirror of the runtime tree, so the now-live includes-vs-tree
  check will likely fail there on cherry-pick.
• generate_shaded_license_metadata.py crashes on xtable-utilities, which has no
  dependency-tree-runtime.txt and no LICENSE-bundled, though its shade transformers reference one.
  Pre-existing; the validator skips the module so it is never surfaced.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MisuWSVMDBniJz95BEAHAe