Repository navigation
HADOOP-19970. Resolve one Jetty release and one servlet API on every module classpath - #8699
Open
joseluisll wants to merge 6 commits into
Open
joseluisll wants to merge 6 commits into
joseluisll wants to merge 6 commits into
Conversation
joseluisll
force-pushed
the
jetty-phase-a
branch
from
August 21, 2026 18:10
9aa1997 to
ce75ddb
Compare
|
💔 -1 overall
This message was automatically generated. |
joseluisll
added a commit
to joseluisll/hadoop
that referenced
this pull request
Aug 23, 2026
ShuffleChannelHandler is a Netty handler. It imported org.eclipse.jetty.http. HttpHeader only to name the Connection and Keep-Alive headers, and the same file already imports Netty's HttpHeaderNames.CONTENT_TYPE. Use HttpHeaderNames and HttpHeaderValues throughout. That was the module's only Jetty reference in main sources, so the module no longer uses Jetty at all outside tests. Netty's constants are lowercase AsciiStrings, so the response now carries "connection: keep-alive" where it carried "Connection: Keep-Alive". Field names are case-insensitive per RFC 7230 section 3.2, connection options are case-insensitive per section 6.1, and the response already carried Netty's lowercase "content-length" from HttpUtil.setContentLength, so this makes the response more internally consistent rather than less. No shuffle client reads these headers by name; the fetcher uses HttpURLConnection, which is case-insensitive. Note for HADOOP-19970 (apache#8699): that PR adds a jetty-http dependency to this module's pom for exactly this import. With the import gone the declaration is no longer needed and should be dropped from that PR. Contains content generated by Claude. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
joseluisll
added a commit
to joseluisll/hadoop
that referenced
this pull request
Aug 23, 2026
ShuffleChannelHandler is a Netty handler. It imported org.eclipse.jetty.http. HttpHeader only to name the Connection and Keep-Alive headers, and the same file already imports Netty's HttpHeaderNames.CONTENT_TYPE. Use HttpHeaderNames and HttpHeaderValues throughout. That was the module's only Jetty reference in main sources, so the module no longer uses Jetty at all outside tests. Netty's constants are lowercase AsciiStrings, so the response now carries "connection: keep-alive" where it carried "Connection: Keep-Alive". Field names are case-insensitive per RFC 7230 section 3.2, connection options are case-insensitive per section 6.1, and the response already carried Netty's lowercase "content-length" from HttpUtil.setContentLength, so this makes the response more internally consistent rather than less. No shuffle client reads these headers by name; the fetcher uses HttpURLConnection, which is case-insensitive. Note for HADOOP-19970 (apache#8699): that PR adds a jetty-http dependency to this module's pom for exactly this import. With the import gone the declaration is no longer needed and should be dropped from that PR. Contains content generated by Claude. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
slfan1989
self-requested a review
August 24, 2026 01:27
pan3793
reviewed
Aug 24, 2026
Member
This is the beginning of chaos. And in practice, during the migration, it's likely that we need both |
joseluisll
added a commit
to joseluisll/hadoop
that referenced
this pull request
Aug 24, 2026
ShuffleChannelHandler is a Netty handler. It imported org.eclipse.jetty.http. HttpHeader only to name the Connection and Keep-Alive headers. That was the module's only Jetty reference in main sources, so with it gone the module does not use Jetty at all, tests included. The names are spelled out as constants rather than taken from Netty's HttpHeaderNames and HttpHeaderValues, whose constants are lowercase AsciiStrings: using those would have sent "connection: keep-alive" where this handler has always sent "Connection: Keep-Alive". Field names and connection options are both case-insensitive per RFC 7230, and Hadoop's own fetcher uses HttpURLConnection, but the shuffle response is read by other projects too and there is no reason to change a byte of it here. Note for HADOOP-19970 (apache#8699): that PR adds a jetty-http dependency to this module's pom for exactly this import. With the import gone the declaration is no longer needed and should be dropped from that PR. Contains content generated by Claude. Co-Authored-By: Claude <noreply@anthropic.com>
joseluisll
added a commit
to joseluisll/hadoop
that referenced
this pull request
Aug 24, 2026
ShuffleChannelHandler is a Netty handler. It imported org.eclipse.jetty.http. HttpHeader only to name the Connection and Keep-Alive headers. That was the module's only Jetty reference in main sources, so with it gone the module does not use Jetty at all, tests included. The names are spelled out as constants rather than taken from Netty's HttpHeaderNames and HttpHeaderValues, whose constants are lowercase AsciiStrings: using those would have sent "connection: keep-alive" where this handler has always sent "Connection: Keep-Alive". Field names and connection options are both case-insensitive per RFC 7230, and Hadoop's own fetcher uses HttpURLConnection, but the shuffle response is read by other projects too and there is no reason to change a byte of it here. Note for HADOOP-19970 (apache#8699): that PR adds a jetty-http dependency to this module's pom for exactly this import. With the import gone the declaration is no longer needed and should be dropped from that PR. Contains content generated by Claude. Co-Authored-By: Claude <noreply@anthropic.com>
joseluisll
force-pushed
the
jetty-phase-a
branch
from
August 24, 2026 20:16
cff271a to
0427e2a
Compare
joseluisll
changed the title
joseluisll
changed the title
Member
The jakarta.servlet groupId does not imply the jakarta.servlet package. Version 4.0.x was just the artifact rename under the Eclipse Foundation — the classes are still javax.servlet.http.HttpServlet. The actual package rename ("big bang") landed in 5.0.0. So if you see jakarta.servlet-api:4.0.4 on a classpath, your imports are still javax.*. |
This was referenced Aug 24, 2026
Contributor
Author
|
With the three substasks HADOOP-19970, HADOOP-19971 and HADOOP-19972, we have migrated to jetty12-ee8 as an intermediate step, not breaking downstream consumers. I'm still testing the pr's results, it would be convenient to create a specific subtask for compatibility testing before releasing the intermediate step. Any idea on how to approach the compatibility testing is welcome. The breaking changes should be delivered in a future and additional release, giving time anf instructions for consumers to adapt to the namespace change, that is, using the jakarta instead of the javax classes. This changes also introduce jdk-17 as a requirement, they are no longer jdk-8 compatible due to jetty12. |
Member
@pjfanning that's true. so in the process of namespace migration, we might mix in
but this might not be a real issue for hadoop project, since it stays near the root of the big data ecosystem dependency tree |
pan3793
reviewed
Aug 26, 2026
pan3793
reviewed
Aug 26, 2026
|
💔 -1 overall
This message was automatically generated. |
joseluisll
force-pushed
the
jetty-phase-a
branch
from
August 31, 2026 11:11
077558a to
236e4bd
Compare
joseluisll
force-pushed
the
jetty-phase-a
branch
2 times, most recently
from
September 4, 2026 04:38
45defb4 to
cfb6dbb
Compare
joseluisll
force-pushed
the
jetty-phase-a
branch
2 times, most recently
from
September 24, 2026 19:53
9f9bd35 to
e1e6700
Compare
|
💔 -1 overall
This message was automatically generated. |
joseluisll
marked this pull request as draft
September 24, 2026 23:11
Contributor
Author
|
The "Test mr" job fails on TestRuntimeEstimators. It is not caused by this PR's |
joseluisll
force-pushed
the
jetty-phase-a
branch
2 times, most recently
from
September 25, 2026 12:19
9907045 to
0fc27ce
Compare
|
💔 -1 overall
This message was automatically generated. |
3 of 6 tasks
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
This was referenced Sep 28, 2026
joseluisll
marked this pull request as ready for review
October 1, 2026 21:32
joseluisll
force-pushed
the
jetty-phase-a
branch
from
October 3, 2026 16:07
7806a24 to
8344728
Compare
This was referenced Oct 4, 2026
|
💔 -1 overall
This message was automatically generated. |
… module classpath Several modules resolved more than one Jetty release, and more than one servlet API, on a single classpath. Both combinations compile and then fail at run time, on whichever code path reaches the wrong jar. * hadoop-project excluded javax.servlet-api from jetty-server under the group org.eclipse.jetty, which jetty-server has never used, so the exclusion matched nothing and javax.servlet:javax.servlet-api stayed on the classpath beside the managed jakarta.servlet:jakarta.servlet-api. Both publish the javax.servlet packages, so which one a module compiled and ran against was decided by the order of the jars, and 73 modules carried both. An exclusion naming the real coordinates is added beside the existing one, and the same on javax-websocket-server-impl, which reaches the API again through websocket-servlet. hadoop-common now declares jakarta.servlet-api itself: HttpServer2 and the filters around it are written against javax.servlet, and the API used to arrive there only as a transitive of jetty-server. * Three modules used org.eclipse.jetty from their main sources while no pom in their inheritance chain declared Jetty: hadoop-mapreduce-client-shuffle, hadoop-yarn-server-router and hadoop-yarn-server-applicationhistoryservice. Each now declares the artifact its main sources use, without a version, and jetty-http gains the dependencyManagement entry it was missing. The Router declares jetty-webapp at compile scope, since a test scope would override the compile-scoped copy it inherits and take it off the Router's own runtime classpath. * solr-core carried nine Jetty artifacts, and a servlet API, into hadoop-yarn-applications-catalog-webapp's tests at a much older release. Those tests reach Solr only through EmbeddedSolrServer, which runs no servlet container, so none of it is needed. The exclusions are applied to solr-test-framework as well, whose own path to solr-core would otherwise reintroduce the same jars. javax.servlet:javax.servlet-api no longer ships, so its LICENSE-binary and NOTICE-binary entries go with it. One module keeps two servlet APIs, hadoop-yarn-server-timelineservice-hbase-tests, where the second arrives with HBase's own test stack. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
jersey-test-framework-provider-jetty runs the Jersey tests on a Jetty of its own choosing, which is a second Jetty release on the test classpath of thirteen modules and the reason hadoop-client-runtime had to stop shipping jetty-util. The tests do not care which server answers them, so they move to jersey-test-framework-provider-jdk-http, which runs on com.sun.net.httpserver in the JDK and adds no third-party server at all. With the Jetty test container gone, hadoop-client-runtime ships jetty-util again. hadoop-client-minicluster had excluded it from hadoop-minicluster, hadoop-yarn-server-nodemanager and hadoop-yarn-server-web-proxy on the understanding that the runtime jar owns it; YARN-11793 had to break that when the test container brought jetty-util into the minicluster along a path those exclusions miss. Leaving the exclusion in place now would ship org.eclipse.jetty.util in neither jar. jetty-continuation goes the same way: the test container was the last thing pulling it in. hadoop-client-minicluster stops relocating com/sun/net/httpserver, which the JDK owns, and LICENSE-binary picks up the two artifacts that replace the Jetty provider. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
javax-websocket-server-impl brings jetty-annotations, and with it javax.annotation:javax.annotation-api 1.3.2. That is a second copy of the javax.annotation classes jakarta.annotation:jakarta.annotation-api 1.3.5 already provides through Jersey, so every classpath with both the NodeManager and Jersey carried both, and which one a module ran against was decided by the order of the jars. It is the same duplicate this change removes for javax.servlet-api, brought in the same way. Exclude it on the same javax-websocket-server-impl entry. The NodeManager, the only module that depends on it, keeps the classes through jakarta.annotation-api, and no module resolves javax.annotation-api any more apart from hadoop-maven-plugins, which gets it from Maven's own plugin API at build time. It no longer ships, so its LICENSE-binary and NOTICE-binary entries go. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Jetty 12's ee8 environment depends on org.eclipse.jetty.toolchain:
jetty-servlet-api 4.0.9 (ee8.jetty.servlet.api.version in 12.1.x). Settle
on that coordinate here rather than on jakarta.servlet:jakarta.servlet-api
4.0.4, so the Jetty 12 upgrade (HADOOP-19972) does not have to change it a
second time. Both publish the same javax.servlet 4.0 classes; the Jetty
artifact also carries the web.xml schemas.
- hadoop-project manages jetty-servlet-api at ${jetty-servlet-api.version}
in place of jakarta.servlet-api 4.0.4.
- Every module that declared jakarta.servlet-api declares jetty-servlet-api.
- jersey-test-framework-core declares jakarta.servlet-api at compile scope;
it is excluded there and on the JDK HTTP provider that brings it.
- hadoop-client-minicluster excludes jetty-servlet-api where it excluded
jakarta.servlet-api, as hadoop-client-runtime ships it.
- LICENSE-binary lists jetty-servlet-api 4.0.9 under Apache 2.0 and drops
jakarta.servlet-api 4.0.4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joseluisll
force-pushed
the
jetty-phase-a
branch
from
October 5, 2026 21:10
8344728 to
29c391f
Compare
|
💔 -1 overall
This message was automatically generated. |
…elease ClasspathTestUtils in hadoop-common's test jar asserts that one jar provides javax.servlet (jetty-servlet-api) and that every Jetty jar on the classpath comes from one release. TestJettyClasspath runs them in hadoop-common, hadoop-yarn-common, the NodeManager, Router, application history service and shuffle modules. The NodeManager test also checks that javax.annotation comes only from jakarta.annotation-api. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on (has to be merged first):
mutual: each PR clears a CI failure of the other, so neither goes green alone - merge them back to back
clears a CI failure: it edits TestRouterWebServicesREST.java:1346-1370, where TestFederationWebApp fails at line 1365 in this PR's precommit, still red in the latest run
Required by (waits for this one to be merged):
mutual: each PR clears a CI failure of the other, so neither goes green alone - merge them back to back
Description of PR
After this change, every module classpath resolves one Jetty release (the managed 9.4.58) and one servlet API,
org.eclipse.jetty.toolchain:jetty-servlet-api4.0.9. That is the artifact Jetty 12's ee8 environment depends on, so HADOOP-19972 keeps it as is. Before, modules could carryjavax.servlet-api3.1.0 besidejakarta.servlet-api4.0.4, and some carried Jetty 9.4.44 or 9.4.55 beside 9.4.58. Which jar a module used then depended on classpath order. The Jetty version itself does not change.One module keeps a second servlet API:
hadoop-yarn-server-timelineservice-hbase-tests, wherejavax.servlet-api3.1.0 arrives at test scope withhbase-server.jetty-servlet-api4.0.9 in place ofjakarta.servlet-api4.0.4, and every module that declaredjakarta.servlet-apinow declaresjetty-servlet-api. It is the same Servlet 4.0 API in the samejavax.servletpackages, and it also carries the web.xml schemas. jetty-server'sjavax.servlet-apiexclusion now names the real coordinates (the old one matched nothing). The same exclusion is added tojavax-websocket-server-impl. hadoop-common declares the API directly.javax-websocket-server-implalso bringsjavax.annotation-api1.3.2 through jetty-annotations. It is a second copy of the classesjakarta.annotation-apiprovides through Jersey, and is excluded on the same entry.jersey-test-framework-provider-jdk-httpinstead of the Jetty provider. The Jetty provider brought Jetty 9.4.55 and jetty-continuation, and has no Jetty 12 counterpart for javax.servlet.jersey-test-framework-coredeclaresjakarta.servlet-apiat compile scope, so it is excluded there and on the JDK HTTP provider. hadoop-client-minicluster leavescom.sun.net.httpserverunrelocated.EmbeddedSolrServer.jetty-httpis added to dependencyManagement.jetty-servlet-api.jetty-servlet-api4.0.9 under Apache 2.0 (it is dual Apache-2.0/EPL-2.0), dropsjakarta.servlet-api4.0.4,javax.servlet-api3.1.0 andjavax.annotation-api1.3.2, and lists the JDK HTTP Jersey artifacts. NOTICE-binary dropsjavax.servlet-apiandjavax.annotation-api.Unchanged on purpose: hadoop-common's
jakarta.servlet.jsp-api, which downstream projects inherit, and the Jetty version.How was this patch tested?
New tests guard the result:
ClasspathTestUtils(hadoop-common test jar) asserts that exactly one jar providesjavax.servlet, and that it isjetty-servlet-api, and that every Jetty jar on the classpath, websocket included, comes from one release.TestJettyClasspathruns those checks in hadoop-common, hadoop-yarn-common, the NodeManager, the Router, the application history service and mapreduce shuffle. The NodeManager one also asserts thatjavax.annotationcomes only fromjakarta.annotation-api.Locally, Maven 3.9.16 on JDK 17. With
-Dmaven.test.failure.ignore=false, so a failing test fails the build.On the head of this PR, with
jetty-servlet-api:dependency:treefiltered tojavax.servlet,jakarta.servletandorg.eclipse.jetty.toolchain): all 87 modules that resolve a servlet API resolvejetty-servlet-api4.0.9, and none resolvesjakarta.servlet-api. The only second API is the HBase one inhadoop-yarn-server-timelineservice-hbase-testsnoted above.hadoop-client-api,-runtime,-miniclusterand both invariant modules):BanDuplicateClasses,BannedDependenciesand the jar-content checks pass. hadoop-client-runtime carries the servlet API relocated, 143 classes plus its schemas underorg/apache/hadoop/shaded/javax/servlet/, with nothing left underjavax/servlet.jetty-utilships only in hadoop-client-runtime; the minicluster carries just the relocatedjetty-util-ajax.ITUseMiniClusteragainst the shaded jars: 2 tests, 0 failures, 0 errors.TestNMContainerWebSocket(2),TestNMWebTerminal(1) andTestNMWebServer(3): 6 tests, 0 failures, 0 errors, withjetty-servlet-api-4.0.9.jaras the only servlet API on the test classpath. They are the websocket path the annotation exclusion touches.Before the switch to
jetty-servlet-api, on the earlier commits of this PR:javax.annotation-apiremains only in hadoop-maven-plugins, which is build-time only.ITUseHadoopCodecspasses.TestRuntimeEstimatorspasses with CI's classpath layout.For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
Contains content generated by Claude.
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html