Skip to content

[ISSUE #11151] Coordinate graceful shutdown of remoting sub-servers - #11152

Open
qianye1001 wants to merge 3 commits into
apache:developfrom
qianye1001:codex/fix-subserver-graceful-shutdown
Open

[ISSUE #11151] Coordinate graceful shutdown of remoting sub-servers#11152
qianye1001 wants to merge 3 commits into
apache:developfrom
qianye1001:codex/fix-subserver-graceful-shutdown

Conversation

@qianye1001

@qianye1001 qianye1001 commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Which Issue(s) This PR Fixes

Fixes #11151.

Brief Description

A sub-server now honors the existing graceful-shutdown configuration: it enters the draining state, keeps its listener open for its own grace period, and then closes that listener. Requests continue to use the existing GO_AWAY handling and client-version condition. Other ports and shared executors remain available when a child shuts down independently.

Parent shutdown starts draining all children before waiting. Each server records its own deadline on its first transition to draining; an already draining child retains its original deadline. The parent waits for the latest deadline before closing child listeners and releasing shared resources, so the grace periods overlap instead of accumulating sequentially. An atomic compare-and-set records each child's first deadline, including when main and child shutdown run concurrently. Repeated shutdown calls return immediately without repeating cleanup or resetting the deadline. Coordination uses only atomic state and adds no executor or timer thread.

How Did You Test This Change?

  • JDK 11: 14 focused remoting server/config/lifecycle tests passed, with Checkstyle and SpotBugs enabled.
  • JDK 21: all 6 new plaintext-TCP shutdown tests passed (jacoco.skip=true, spotbugs.skip=true; static analysis was run under JDK 11).
  • The new real-TCP tests cover existing and fresh connections during draining, independent child shutdown and unaffected sibling/parent ports, parent-only shutdown of multiple children, both parent/child shutdown orders, simultaneous and repeated calls, a pre-existing child deadline longer than the parent's wait, disabled graceful shutdown, older client versions, and interruption.
  • Listener closure is verified through the actual channel close future, avoiding platform-specific connection-refusal versus connection-timeout behavior.
  • On the unmodified base, the parent-shutdown regression test fails because children never enter the draining state.

CI test stabilization

  • TlsTest.disabledServerRejectsSSLClient accepts either connection or send failure: the server rejects TLS by closing the connection, which may happen before the client checks channel activity or during its write. TLS tests use the loopback IP to avoid hostname resolution.
  • TimerMessageStoreTest.testTimerFlowControl waits for each accepted message to reach the timer wheel before testing the next admission, replacing a fixed 5 ms sleep. This preserves the flow-control assertions without assuming the asynchronous reput/enqueue work has already completed.
  • JDK 11: all 13 TLS tests and the timer flow-control test passed with Checkstyle and SpotBugs enabled.

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR coordinates graceful shutdown of remoting sub-servers so that parent and child shutdown grace periods overlap instead of accumulating sequentially. The implementation uses CompletableFuture for concurrent shutdown tracking and a ReentrantLock for state transitions only — waiting and channel closure happen outside the lock. Well-structured with comprehensive test coverage (270 lines of tests covering edge cases like pre-existing deadlines, disabled graceful shutdown, older client versions, and interruption).

LGTM — clean concurrency design, good test coverage, and the PR description clearly explains the rationale.


Automated review by github-manager-bot

@codecov-commenter

codecov-commenter commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.30435% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.34%. Comparing base (d38f81e) to head (685fdb8).
⚠️ Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
...e/rocketmq/remoting/netty/NettyRemotingServer.java 91.30% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             develop   #11152      +/-   ##
=============================================
- Coverage      49.39%   49.34%   -0.05%     
+ Complexity     14237    14227      -10     
=============================================
  Files           1390     1390              
  Lines         103122   103157      +35     
  Branches       13484    13494      +10     
=============================================
- Hits           50932    50902      -30     
- Misses         46029    46079      +50     
- Partials        6161     6176      +15     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR coordinates graceful shutdown across the parent NettyRemotingServer and its SubRemotingServer instances. The refactor replaces the sequential drain-and-close pattern with a concurrent approach: all sub-servers begin draining in parallel, the parent waits for the latest deadline, then closes listeners and releases shared resources.

The design is solid — AtomicBoolean guards ensure idempotent shutdown, AtomicReference<Long> for the deadline allows either caller (parent or child) to set it first, and interruption handling correctly preserves the interrupt flag. The 6 test methods comprehensively cover concurrent shutdown, disabled graceful mode, interrupted threads, and child-before-parent ordering.

LGTM.


Automated review by "github-manager-bot"

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Well-designed coordination of graceful shutdown across parent and sub-servers. The atomic state management correctly handles concurrent shutdown scenarios, and the deadline overlap logic avoids sequential accumulation. Comprehensive test coverage including edge cases (pre-existing child deadline, disabled graceful shutdown, interruption handling).


Automated review by github-manager-bot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Remoting sub-servers do not honor graceful shutdown

3 participants