Repository navigation
MAPREDUCE-7545. Make TaskId.hashCode independent of the JVM. - #8754
Merged
Merged
Conversation
TaskId.hashCode mixed in getTaskType().hashCode(). TaskType is an enum, and
an enum's hashCode is its identity hash, which HotSpot draws from a sequence
that shifts with whatever else asked for identity hashes first: which jars
are on the classpath, which classes get initialized. So every HashMap keyed
by TaskId iterated in an order that depended on the environment rather than
on the tasks.
TestRuntimeEstimators depends on that order. The five slow maps it builds
are speculated in the order the speculator meets them, and it expects three
speculations to win. In some orders only one does, and the test then fails
deterministically on that environment:
We got the wrong number of successful speculations.
==> expected: <3> but was: <1>
It does, for instance, on GitHub Actions with the classpath HADOOP-19970
produces, while passing locally on the same commit. Consuming four extra
identity hashes before the test starts reproduces the failure locally.
Hash the task type's ordinal instead. The hash no longer changes between
JVMs, so neither does the order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 of 6 tasks
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
@slfan1989 @pan3793 Would you mind reviewing this one? It's a one-line change plus a unit test. |
5 of 6 tasks
slfan1989
approved these changes
Oct 4, 2026
slfan1989
left a comment
slfan1989
left a comment
Contributor
There was a problem hiding this comment.
+1.
Using the task type ordinal removes the identity hash dependency while preserving the equals/hashCode contract.
I verified that the new regression test fails before the fix and that all four TestIds tests pass afterward.
Cross-JVM hash checks and protobuf round-trip map lookups also passed.
Thanks for the fix!
Contributor
|
@joseluisll Thanks for the contribution! Merged into trunk. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
JIRA: MAPREDUCE-7545. Make TaskId.hashCode independent of the JVM.
TaskId.hashCodemixed ingetTaskType().hashCode().TaskTypeis an enum, so that is anidentity hash: it changes from one JVM to the next, and so does the iteration order of every
HashMapkeyed byTaskId.Hash the task type's ordinal instead.
Expected result: a
TaskId's hash is now a function of its fields alone, identical acrossJVMs and runs. Task iteration order stops depending on the classpath, and
TestRuntimeEstimators— which fails withexpected: <3> but was: <1>speculations undersome orders — stops flaking. No behaviour changes beyond the hash;
equalscompares tasktypes by reference, which the ordinal agrees with.
How was this patch tested?
New unit test
TestIds#testTaskIdHashCodeUsesTheTaskTypeOrdinalasserts the computed hashfor every
TaskType, so the identity hash cannot come back unnoticed.hadoop-mapreduce-client-commonandhadoop-mapreduce-client-app(which holdsTestRuntimeEstimators) pass.For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.