[701] Generate the bundled LICENSE and NOTICE instead of maintaining them - #898
[701] Generate the bundled LICENSE and NOTICE instead of maintaining them#898slachiewicz wants to merge 4 commits into
Conversation
…them ASF policy needs no LICENSE entry for a bundled ALv2 dependency, and NOTICE must carry nothing that is not legally required. 715 of this reactor's 860 third-party artifacts are ALv2. NOTICE now comes from ApacheNoticeResourceTransformer. LICENSE is rendered per module by maven-remote-resources-plugin from release/license/bundled/META-INF/LICENSE.vm, listing only the non-ALv2 components with a pointer to their texts. The generator, the coverage script and the six hand-maintained LICENSE-bundled and NOTICE-bundled files are gone. xtable-utilities is added to the License Check, which built three modules, not four. It fails there: 55 bundled non-ALv2 dependencies with no licence text, and it ships no META-INF/LICENSE today. Left failing rather than excluded. Verified on Temurin 11: xtable-aws renders 5 of 5 with no UNMAPPED lines, and the jar has one META-INF/LICENSE, one META-INF/NOTICE and its five texts.
20 licence texts extracted from the artifacts' own jars into xtable-utilities. 33 still missing. junit:junit and hamcrest-core were in the hive-metastore shade includes. hbase-protocol:2.0.0-alpha4 declares JUnit at compile scope, so it reaches the bundle as a normal dependency. Dropping both from the shade removes the module's only EPL 1.0 obligation. The template now collapses 23 declared licence spellings to 8 families, and checker-compat-qual is pinned to MIT - it is dual licensed, and the GPL-2.0 entry it declares first is not the option taken.
It is not deployed and not in the source release, and nothing depends on it, so a licence gap there should not break the build. The 20 texts added for it stay, ready for whenever the bundle is reconsidered.
Its <pom> delimiter lists the XML root elements it knows; supplemental-models.xml uses <supplementalDataModels> and spotless errors out rather than skipping it.
| skipped. | ||
| --> | ||
| <execution> | ||
| <id>process-bundled-resources</id> |
There was a problem hiding this comment.
Hi @slachiewicz, any idea which ASF projects generate the bundled LICENSE this way? Naming them would help weigh this against #895.
HBase looks like the closest precedent: hbase-resource-bundle renders META-INF/LICENSE.vm and NOTICE.vm through maven-remote-resources-plugin with supplemental-models.xml, and hbase-shaded pairs that with ApacheNoticeResourceTransformer the same way this does.
I have looked at Apache Spark, Apache Hudi, Apache Iceberg and Apache Kafka all of them seem to have hand-maintained and validation scripts like XTable. Please correct me if I'm wrong.
https://github.com/apache/iceberg/blob/main/dev/check-license
There was a problem hiding this comment.
HBase is the precedent I modeled this on, and you found the exact machinery. Two more projects use the same pattern for shaded jars: Apache Ratis (ratis-resource-bundle is a direct copy of the HBase setup) and hbase-connectors. Beyond shaded jars, the supplemental-models.xml + velocity mechanism is standard ASF tooling — Karaf, TomEE, CXF, ServiceMix, James, Qpid Broker-J, AsterixDB and Maven's own distribution all render LICENSE from it.
One correction on the hand-maintained side: dev/check-license in Spark and Iceberg is an Apache RAT run — it checks source headers, not bundled LICENSE content. Spark's LICENSE-binary and Kafka's licenses/ are hand-maintained with no automated validation at all. The real precedent for #895's model is Flink, whose flink-ci-tools LicenseChecker diffs shade output against hand-maintained NOTICE files.
So the field splits roughly: HBase/Ratis generate, Flink hand-maintains with strict validation, Spark/Kafka/Hudi/Iceberg hand-maintain with none — and XTable's history shows what the last option leads to. Either #898 or #895 would put us in the top two groups. Note validate_bundled_license_texts.py stays as the validation layer in both.
This comment was created with AI assistance.
|
Overall, I am in favor of generating the documentation automatically instead of manually curating it which can be error-prone and go stale. Unfortunately, I don't have bandwidth right now to check the proposal in the PR. |
|
If direction is right, I would propose to first try to upgrade to Spark 3.5 that would make few unknown/problematic artifacts to go. |
3 participants
Addresses #701, and #865 and #717 by construction. Part of #536.
What is the purpose of the pull request
ASF policy (licensing-howto) asks for less than XTable currently maintains:
So this generates the bundled LICENSE instead of maintaining it, and lets the Apache shade transformers own NOTICE.
Brief change log
generate_shaded_license_metadata.py,validate_shaded_license_coverage.sh, and the six hand-maintainedLICENSE-bundled/NOTICE-bundledfiles (248 to 1175 lines each).maven-remote-resources-pluginfromrelease/license/bundled/META-INF/LICENSE.vm, listing only the non-Apache-2.0 components with a pointer to their text.supplemental-models.xmlcarries the components whose POM declares nothing.ApacheNoticeResourceTransformer. Forxtable-awsit is 110 lines against the 248 maintained by hand.junit:junitandhamcrest-coreintoxtable-hive-metastore.hbase-protocol:2.0.0-alpha4declares JUnit at compile scope, so it reached the bundle as a normal dependency; it was the module's only EPL 1.0 obligation.Net 424 insertions, 776 deletions.
validate_bundled_license_texts.pyfrom #857 is kept and unchanged. It decides what is bundled from class membership in the built jar rather than from the shade include list, which makes it immune to that list being stale.Relationship to the other open PRs
#895 addresses the same issue by repairing the hand-maintained model — keeping the generator,
LICENSE-bundledandMETA-INF/licenses/, and fixing the coverage script. This PR deletes that model instead. The two are alternatives, not increments, and both touchgenerate_shaded_license_metadata.pyandvalidate_shaded_license_coverage.shin incompatible ways. I am not proposing that either supersedes the other; that is for the PMC to weigh.Two things from #895 hold regardless of which direction wins, and are not duplicated here:
[772]regeneration of thextable-hive-metastoreshade includes for Hive 3.1.3. This PR assumes that lands and rebases onto it.validate_shaded_license_coverage.shhas been a no-op since June, because it discovered modules withrg, which is absent onubuntu-latest. Deleting the script removes the bug but not the fact that release-time conclusions drawn from a green License Check since then are unsupported.#876 and #875 target the same issue on
mainandbranch-0.4; #743 is an earlier attempt at the same generation problem.Verify this pull request
validate_bundled_license_texts.pyexits 0 on all three released bundles:xtable-aws5 of 5,xtable-hive-metastore20 of 20,xtable-hudi-support-extensions3 of 3.apache-rat:checkpasses. Built on Temurin 11 from a clean target.Checked by hand on the
xtable-awsbundle: oneMETA-INF/LICENSE, oneMETA-INF/NOTICE, its five licence texts underMETA-INF/licenses/, no dependency LICENSE or NOTICE left behind, andMETA-INF/DISCLAIMERstill present.Licence texts are still taken from the artifacts themselves rather than generated.
license-maven-plugin'sdownload-licenseswas tried for that and rejected: of 24 files it fetched, 18 were HTML web pages —checker-qualresolved to the OSI WordPress page — one dependency fetched nothing at all, and the build still passed. Filed upstream as mojohaus/license-maven-plugin#723 and #724.Two open questions for reviewers
javolutionis recorded as MIT per the text bundled with the artifact. Its POM says BSD, and the bundled text is MIT carrying a 2017 copyright for a 2010 artifact. Left as a question rather than settled here.xtable-utilitiesis not covered. It is not deployed and not in the source release, and its bundle needs 33 further licence texts; Shrink the xtable-utilities bundled jar #896 tracks shrinking it first.This change was created with AI assistance.