Repository navigation
Retry peer-list reset during AddRegionPeer rollback - #19118
Open
d-wang-commit wants to merge 1 commit into
Open
d-wang-commit wants to merge 1 commit into
d-wang-commit wants to merge 1 commit into
Conversation
Wait five seconds between up to three additional reset RPC batches. Cover retry success, exhaustion, early completion and interruption.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Retry rollback cleanup while DataNodes restart
After a whole-cluster restart during region migration, AddRegionPeer rollback can reach the target DataNode before its internal RPC service is ready. All six immediate RPC attempts can fail, leaving the target consensus group directory behind. This occurred in IoTDBRegionMigrateClusterCrashIoTV2StreamIT#clusterCrash2.
Retry incomplete or unsuccessful peer-list reset batches up to three additional times, waiting five seconds between batches. Stop when all requested nodes return success, or stop waiting and preserve the interrupt flag when interrupted. Keep the existing rollback flow and completion behavior.
Tests
AddRegionPeerProcedureTestfor delayed success, retry exhaustion, immediate success and interruption. All five tests, including the existing serialization test, passed.IoTDBRegionMigrateClusterCrashIoTV2StreamIT#clusterCrash2passed with a full 3-ConfigNode/3-DataNode restart. Reset succeeded in the first batch in this run; delayed retries are covered by the unit tests.Side effects and risks
Failed rollback cleanup can occupy a procedure worker for up to 15 additional seconds of waiting, plus RPC time. Retries repeat the idempotent reset on all related nodes. Cleanup remains best effort if retries are exhausted; procedure states and serialization are unchanged.
This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
AddRegionPeerProcedure: add bounded delayed retries around rollback peer-list reset RPCs.AddRegionPeerProcedureTest: cover retry timing, attempt limits, early success and interruption.