YARN-12004. Fix Router-vs-RM read races in TestRouterWebServicesREST. by joseluisll · Pull Request #8787 · apache/hadoop

joseluisll · 2026-10-03T13:38:58Z

Required by (waits for this one to be merged):

YARN-11984. Federation mock sub-cluster and router do not stop cleanly on SIGTERM #8681 (YARN-11984) - YARN-11984 Federation mock sub-cluster and router do not stop cleanly on SIGTERM
HADOOP-19970. Resolve one Jetty release and one servlet API on every module classpath #8699 (HADOOP-19970) - HADOOP-19970. Resolve one Jetty release and one servlet API on every module classpath
HADOOP-19972. Upgrade to Jetty 12 without changing the servlet namespace #8704 (HADOOP-19972) - HADOOP-19972. Upgrade to Jetty 12 without changing the servlet namespace
HADOOP-19982. Re-enable TestDFSClientRetries and TestBalancerWithHANameNodes in GHA #8725 (HADOOP-19982) - HADOOP-19982. Re-enable TestDFSClientRetries and TestBalancerWithHANameNodes in GHA
YARN-11993. PublicLocalizer thread exits permanently when pending.remove() returns null #8755 (YARN-11993) - YARN-11993. PublicLocalizer thread exits permanently when pending.remove() returns null
YARN-12002. Stop the Router delegation token secret manager on service stop #8784 (YARN-12002) - YARN-12002. Stop the Router delegation token secret manager on service stop
YARN-12003. Shut down the Router scheduled executor on service stop #8785 (YARN-12003) - YARN-12003. Shut down the Router scheduled executor on service stop

Description of PR

JIRA: https://issues.apache.org/jira/browse/YARN-12004

Supersedes #8786.

TestRouterWebServicesREST, and TestFederationWebApp which extends it, fail intermittently on CI because testAppXML, testAppStateXML, testAppAttemptXML and testGetContainersXML each compared one Router read with one RM read of a value the RM was still changing (AM host, state, attempts, containers), e.g. expected: <ACCEPTED> but was: <SUBMITTED>. The change is test-only, in TestRouterWebServicesREST.java.

A new assertRouterMatchesRM(path, type, key) reads the RM, then the Router, then the RM again, compares a key extracted from each answer, and retries (up to 10s) until the RM was stable around the Router read and the Router matches it.
testGetContainersXML compares container IDs rather than counts, so one AM container replacing another is seen as a change.
A single-address performGetCall replaces the duplicated Router/RM request code, and every client it creates is closed.
The four tests that now wait get a 30s timeout instead of 2s.

How was this patch tested?

Locally on Linux with OpenJDK 21 and Maven 3.9.11, in hadoop-yarn-server-router, on this PR's head:

mvn test -Dtest=TestRouterWebServicesREST,TestFederationWebApp: 94 tests (42 + 52), 0 failures, 0 errors.
The four changed tests looped 300 times against one cluster in one JVM (1200 runs), no Router-vs-RM comparison failed.

For code changes:

Does the title of this PR start with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: Have the integration tests been executed and the endpoint
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required. N/A, no object storage changes.
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0? N/A, no new dependencies.
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files? N/A

AI Tooling

If an AI tool was used:

The PR includes the phrase "Contains content generated by "
where is the name of the AI tool used.
My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html

Contains content generated by Claude Code.

hadoop-yetus · 2026-10-03T14:56:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 46s		trunk passed
+1 💚	compile	0m 38s		trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	compile	0m 40s		trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	checkstyle	0m 33s		trunk passed
+1 💚	mvnsite	0m 41s		trunk passed
+1 💚	javadoc	0m 33s		trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javadoc	0m 37s		trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	spotbugs	0m 56s		trunk passed
+1 💚	shadedclient	16m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javac	0m 17s		the patch passed
+1 💚	compile	0m 17s		the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	javac	0m 17s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 15s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
+1 💚	mvnsite	0m 19s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	spotbugs	0m 41s		the patch passed
+1 💚	shadedclient	15m 40s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	7m 16s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 25s		The patch does not generate ASF License warnings.
		76m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/1/artifact/out/Dockerfile
GITHUB PR	#8787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 4eb8958cde79 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `3d703f1`
Default Java	Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/1/testReport/
Max. process+thread count	1321 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/1/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-10-03T20:03:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 2s		trunk passed
+1 💚	compile	0m 38s		trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	compile	0m 38s		trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	checkstyle	0m 37s		trunk passed
+1 💚	mvnsite	0m 40s		trunk passed
+1 💚	javadoc	0m 40s		trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javadoc	0m 39s		trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	spotbugs	0m 54s		trunk passed
+1 💚	shadedclient	16m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 19s		the patch passed
+1 💚	compile	0m 18s		the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javac	0m 18s		the patch passed
+1 💚	compile	0m 19s		the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	javac	0m 19s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 14s		the patch passed
+1 💚	mvnsite	0m 20s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javadoc	0m 17s		the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	spotbugs	0m 41s		the patch passed
+1 💚	shadedclient	15m 28s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	8m 3s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 26s		The patch does not generate ASF License warnings.
		76m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/2/artifact/out/Dockerfile
GITHUB PR	#8787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 89c4f10575a5 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7 15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `cce62ea`
Default Java	Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/2/testReport/
Max. process+thread count	1302 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/2/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

joseluisll · 2026-10-05T04:56:17Z

@slfan1989 Hi, as you reviewed also YARN-11984, would you mind having a look at this router PR as well?

testAppXML, testAppStateXML, testAppAttemptXML and testGetContainersXML compared one Router read with one RM read of a value the RM was still changing. assertRouterMatchesRM reads the RM, then Router, then the RM again, and retries until the RM is stable around the Router read and Router matches it. testGetContainersXML compares container IDs rather than counts. A single-address performGetCall replaces the duplicated request code and closes every client it creates. Contains content generated by Claude Code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

joseluisll · 2026-10-05T14:27:04Z

I cut the PR in two parts, so it is easier to review independently. The other fix is in YARN-12006.

hadoop-yetus · 2026-10-05T16:46:02Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	49m 8s		trunk passed
+1 💚	compile	0m 58s		trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	compile	0m 53s		trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	checkstyle	0m 49s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javadoc	0m 51s		trunk passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	spotbugs	1m 23s		trunk passed
+1 💚	shadedclient	34m 12s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 28s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javac	0m 26s		the patch passed
+1 💚	compile	0m 26s		the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	javac	0m 26s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 22s		the patch passed
+1 💚	mvnsite	0m 30s		the patch passed
+1 💚	javadoc	0m 25s		the patch passed with JDK Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu
+1 💚	javadoc	0m 23s		the patch passed with JDK Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
+1 💚	spotbugs	1m 8s		the patch passed
+1 💚	shadedclient	32m 0s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	10m 20s		hadoop-yarn-server-router in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		140m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.56 ServerAPI=1.56 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/3/artifact/out/Dockerfile
GITHUB PR	#8787
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 35aafbd5e459 5.15.0-186-generic #196-Ubuntu SMP Sat Jun 20 16:09:34 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `762577c`
Default Java	Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12.1+1-1-24.04.4-Ubuntu /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20.1+1-1-24.04-Ubuntu
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/3/testReport/
Max. process+thread count	1317 (vs. ulimit of 10000)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8787/3/console
versions	git=2.43.0 maven=3.9.15 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2026-10-06T02:42:45Z

@slfan1989 Hi, as you reviewed also YARN-11984, would you mind having a look at this router PR as well?

@joseluisll Thanks for reaching out! I’m happy to review this Router PR and will take a look.

slfan1989

Thanks for the fix. Reading the RM before and after the Router response appropriately handles the lifecycle races, and comparing container IDs improves the validation.

I verified that the helper retries when the RM changes and still fails when the Router consistently disagrees with a stable RM. CI is green.

LGTM, +1.

slfan1989 · 2026-10-06T09:47:40Z

@joseluisll Thanks for the contribution! Merged into trunk.

github-actions Bot added YARN trunk labels Oct 3, 2026

joseluisll force-pushed the YARN-12004 branch from cce62ea to 762577c Compare October 5, 2026 14:24

joseluisll mentioned this pull request Oct 5, 2026

YARN-12006. Wait for the first attempt in TestRouterWebServicesREST.getAppAttempt. #8791

Merged

3 of 6 tasks

slfan1989 approved these changes Oct 6, 2026

View reviewed changes

slfan1989 merged commit e5defed into apache:trunk Oct 6, 2026
6 checks passed

slfan1989 mentioned this pull request Oct 7, 2026

YARN-12005. Fix flaky TestRouterWebServicesREST#testUpdateNodeResource #8788

Merged

3 of 6 tasks

joseluisll mentioned this pull request Oct 7, 2026

Credit fixes in a parent test class to the failing subclass in qbt_jira.py joseluisll/hadoop-prtracker#21

Merged

Navigation Menu

Repository navigation

YARN-12004. Fix Router-vs-RM read races in TestRouterWebServicesREST. - #8787

YARN-12004. Fix Router-vs-RM read races in TestRouterWebServicesREST.#8787
slfan1989 merged 1 commit into
apache:trunkfrom
joseluisll:YARN-12004

joseluisll commented Oct 3, 2026 •

edited

Loading

Uh oh!

hadoop-yetus commented Oct 3, 2026

Uh oh!

hadoop-yetus commented Oct 3, 2026

Uh oh!

joseluisll commented Oct 5, 2026 •

edited

Loading

Uh oh!

joseluisll commented Oct 5, 2026 •

edited

Loading

Uh oh!

hadoop-yetus commented Oct 5, 2026

Uh oh!

slfan1989 commented Oct 6, 2026

Uh oh!

slfan1989 left a comment

Uh oh!

Uh oh!

slfan1989 commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

joseluisll commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

hadoop-yetus commented Oct 3, 2026

Uh oh!

hadoop-yetus commented Oct 3, 2026

Uh oh!

joseluisll commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joseluisll commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hadoop-yetus commented Oct 5, 2026

Uh oh!

slfan1989 commented Oct 6, 2026

Uh oh!

slfan1989 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

slfan1989 commented Oct 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

joseluisll commented Oct 3, 2026 •

edited

Loading

joseluisll commented Oct 5, 2026 •

edited

Loading

joseluisll commented Oct 5, 2026 •

edited

Loading