Repository navigation
YARN-12004. Fix Router-vs-RM read races in TestRouterWebServicesREST and TestFederationWebApp - #8786
Closed
joseluisll wants to merge 3 commits into
Closed
joseluisll wants to merge 3 commits into
joseluisll wants to merge 3 commits into
Conversation
TestRouterWebServicesREST, and TestFederationWebApp which extends it, fail intermittently on CI because several tests do not synchronise with the RM's asynchronous application lifecycle: - getAppAttempt read getAttempts().get(0) straight after the submission returned, before the RM had created the first attempt, and threw IndexOutOfBoundsException. It now waits for the attempt to appear. testGetAppAttemptXML and testGetContainersXML (which testGetAppsMultiThread calls directly) share the helper. - testAppXML, testAppStateXML, testAppAttemptXML and testGetContainersXML each compare one read from the Router with one read from the RM, taken one after the other, of a value the RM is still changing on a newly submitted application (the AM host, the application state, the number of attempts, the attempt's containers). Looping the first three 3000 times against unchanged code failed 8, 14 and 38 times respectively, e.g. "expected: <ACCEPTED> but was: <SUBMITTED>"; testGetContainersXML failed on CI with "expected: <0> but was: <1>". The comparison moves into an assertRouterMatchesRM helper used by all four tests. Retrying until the two reads agree would pass a broken Router as soon as the RM's value happened to reach whatever the Router returns, so each attempt reads the RM, then the Router, then the RM again. If the RM changed in between, retry; if not, the Router is expected to match, and a Router that disagrees with a stable RM twice fails at once. Twice rather than once because a value can change and change back within one Router round trip. For the same reason the checks compare values that cannot repeat. testGetContainersXML compares container IDs instead of the container count: the RM reports the current attempt's containers, so the count goes 1, 0, 1 as one attempt fails and the next starts. testAppXML also compares the AM container log URL, which names the attempt's AM container where the AM host alone does not. With a Router altered to always report no attempts, testAppAttemptXML fails 100 times out of 100, in about 0.5s each. A Router altered to always report no containers still gets past testGetContainersXML almost every time: these applications' AM containers exit almost at once, so the RM itself reports none most of the time. The tests that now wait get a 30s timeout instead of 2s, so the wait, bounded at 10s, cannot be cut short. performGetCalls now closes the two clients it creates. Tests comparing values that do not change once the application is submitted (priority, queue, timeouts, application counts) are left as they were. Looping each of the four methods 3200 times, in three fresh JVMs, fails none. Split out of YARN-11984. Contains content generated by Claude Code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- performGetCall reads one address; performGetCalls and getAppAttempt use it, so each client is closed and the duplicated Router/RM request code goes away. - assertRouterMatchesRM reads RM, Router, RM with one request each; the "RM before" read no longer goes through performGetCalls, which also made a Router request that was discarded. - Tests pass a key extractor instead of an assertion lambda, so the helper compares keys directly rather than catching AssertionError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hu5iS6ESgWfF9BkTnUiGPG
…sXML. The containers endpoint reports the live containers of the current attempt, which here is only each attempt's AM container, alive for 20-100 ms (median 31 ms over 862 applications) before it exits. The RM's answer therefore goes none, first AM container, none, next AM container, none, and each AM container can come and go entirely within one Router read. With two attempts per application that is two "stable" mismatches, which reached STABLE_MISMATCHES_TO_FAIL and failed the test once in 150 loops: Router answered [container_..._0052_02_000001] where the RM, stable around the read, answered [] (2 times) testGetContainersXML now retries such mismatches until the timeout. The other tests keep failing fast: application state and attempt count only move forward, and the AM host and log URL can blip at most once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Hu5iS6ESgWfF9BkTnUiGPG
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: https://issues.apache.org/jira/browse/YARN-12004
TestRouterWebServicesREST, andTestFederationWebAppwhich extends it, fail intermittently on CI because several tests don't wait for the RM's asynchronous application lifecycle. The change is test-only, inTestRouterWebServicesREST.java.getAppAttemptreadgetAttempts().get(0)right after the submission returned, before the RM had created the first attempt, and threwIndexOutOfBoundsException. It now waits for the attempt, up to 10s.testGetAppAttemptXMLandtestGetContainersXMLuse it, andtestGetAppsMultiThreadcallstestGetContainersXML.testAppXML,testAppStateXML,testAppAttemptXML,testGetContainersXMLeach compared one Router read with one RM read of a value the RM was still changing (AM host, state, attempts, containers). Failures look likeexpected: <ACCEPTED> but was: <SUBMITTED>andexpected: <0> but was: <1>.assertRouterMatchesRMhelper reads the RM, then the Router, then the RM again:testGetContainersXMLcompares container IDs instead of counts, which go 1 → 0 → 1 as one attempt fails and the next starts.testAppXMLalso compares the AM container log URL, which identifies the attempt.@Timeoutinstead of 2s, so the 10s wait can't be cut short.performGetCallsnow closes the two clients it creates.Tests that compare values which don't change after submission (priority, queue, timeouts, app counts) are unchanged.
How was this patch tested?
testAppXML,testAppStateXMLandtestAppAttemptXML3000 times each failed 8, 14 and 38 times. Upstream CI on YARN-11984. Federation mock sub-cluster and router do not stop cleanly on SIGTERM #8681 failed inTestFederationWebAppandTestRouterWebServicesREST, and the same two classes are the failing router tests in the CI reports on HADOOP-19970, HADOOP-19972 and HADOOP-19982.testAppAttemptXMLfails 100 out of 100 runs, in about 0.5s each.TestRouterWebServicesREST42/42 andTestFederationWebApp52/52 pass.For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.