feat: integrate URMA with the common transport handshake by zchuango · Pull Request #3645 · apache/brpc

zchuango · 2026-10-23T08:07:49Z

What problem does this PR solve?

Issue Number: N/A

Related Discussion: #3432

Related PR: #3535

Problem Summary:

PR #3535 introduces the common transport-level handshake framework and migrates RDMA and UBSHM to it. This PR extends the same framework to URMA.

URMA previously managed TCP handshake I/O, framing, negotiation, state transitions and fallback inside its endpoint. Moving this orchestration into AdapterTransport gives the high-speed transports a consistent connection setup flow, while URMA retains ownership of its resources and data-plane operations.

What is changed and the side effects?

Changed:

Handshake integration:

Add the URMA adapter to the common handshake framework.
Support the existing binary v2 (URMA) and protobuf v3 (URM3) wire formats.
Use the common session for negotiation, activation and TCP fallback.
Preserve disabled-Hello and disabled-ACK behavior for legacy peers and TCP-only servers.
Start CQ callbacks after the final ACK has been consumed and the TCP handshake parser has returned.

URMA data-plane updates:

Configure standalone bonding and import the peer segment before its Jetty.
Apply placement and completion ordering to payload and credit SENDs.
Limit each CTP SEND to 4096 bytes and split larger RPCs across multiple WRs.
Retain the default 8192-byte buffer blocks.
Pin the main Socket before CQ callbacks access the endpoint.
Synchronize CQ dispatch with resource teardown and reject callbacks from an earlier reset generation.
Add bounded stream tracing and update the Chinese and English documentation.

Side effects:

URMA connection setup uses the common AdapterTransport handshake.
Existing URMA v2/v3 wire formats remain compatible.
Resource unavailability or declined negotiation selects TCP.
Malformed handshake frames and established URMA transport errors close the connection.

Validation

Automated tests:

URMA: 59 tests; TCP fallback: 8 tests.
Common handshake: 28 tests; RDMA: 92 tests; UBSHM: 43 tests.
Legacy/common interoperability, final-ACK handoff and CQ lifecycle tests passed 100 shuffled repetitions with the mock provider.
CQ lifecycle tests also passed 100 shuffled repetitions under AddressSanitizer.
Large-attachment tests verify response contents byte-for-byte across event/polling and copy/zero-copy modes.
TCP-only fallback tests, Release library build/install and performance-example builds passed.

Hardware tests:

Two-node v3 common-handshake tests using bonding_dev_0 in standalone mode.
Covered event/polling × copy/zero-copy.
Each run used 4 workers, queue depth 16, 128-byte echoed attachments and a 180-second duration.
Restarted the server during client traffic.
All four combinations recovered to approximately 1000 QPS, with cumulative error counts stabilizing after recovery.
Completion, import, protocol parsing, FATAL and TCP fallback log searches returned no matches.

Development PR CI: all 24 checks passed at 91bf2d1.

Check List

Build and run related regression tests.
Add handshake compatibility, fallback and concurrent teardown tests.
Update documentation and validate on URMA hardware.

…lity Resolve UBSHM handshake conflicts using the tested rebased tree. Retain backend-specific format negotiation, remote shared-memory lengths, and upstream IPC V2 tests.

Resolve current master conflicts and preserve UBSHM IPC V2 negotiation

Resolve master conflict and preserve shared memory name safety

…refresh Resolve modular CMake conflict and preserve handshake protobuf generation

* Unify transport handshakes on current upstream master * Fix certificate name extraction with const OpenSSL accessors * Integrate URMA with the common transport handshake Port URMA v2 and v3 negotiation into AdapterTransport while preserving the existing RDMA and UBSHM implementations and regression tests. Include standalone bonding import, ordered SEND work requests, the 4096-byte CTP payload limit, bounded diagnostics, mock regression coverage and concise documentation. * Run Linux CI for transport handshake pull requests * Fix URMA handshake fallback compatibility * Use nullptr and refresh URMA handshake factory tests * Test legacy URMA interoperability and final ACK handoff * Fix URMA CQ callback lifetime during teardown Pin the main Socket through a carrier-owned context before endpoint access. Serialize CQ dispatch with teardown and reject callbacks from an earlier reset generation. Add barrier-controlled event/polling and copy/zero-copy regressions for concurrent close, last-reference recycling, and stale callbacks after reset. Keep this lifecycle fix separate from handshake wire compatibility. * Simplify URMA documentation * Restore upstream CI and URMA documentation * Clarify URMA handshake comments and align local style --------- Co-authored-by: Chuang Zhang <zchuang185@gmail.com>

Preserve the common handshake source directory together with the upstream FlatBuffers conditional sources in Makefile. Keep upstream workflows and existing URMA documentation unchanged.

Spicy-cream · 2026-10-23T08:23:56Z

Could you give some advice? @chenBright @wwbmmm

Copilot

🟡 Changes recommended

Unresolved handshake initialization, failure handling, and fallback resource cleanup issues can cause undefined behavior or retain connection resources.

3 open issues

What changed in this PR

This PR extends the common transport handshake framework from #3535 to URMA, centralizing connection setup and TCP fallback while keeping transport-specific resources and data transfer in their endpoints.

Changes:

Integrates URMA v2/v3 negotiation and legacy fallback behavior with the common handshake.
Updates transport activation, receive-event handoff, URMA ordering, and completion lifetime management.
Adds fallback tests, stream tracing, documentation, and build integration.

File	Description
test/CMakeLists.txt	Orders test protobuf compilation after core generation.
test/brpc_urma_fallback_unittest.cpp	Tests URMA framing and TCP fallback without hardware.
src/brpc/urma/urma_stream_trace.h	Adds bounded stream diagnostics.
src/brpc/urma/urma_helper.h	Declares bonding import-layout selection.
src/brpc/urma/urma_helper.cpp	Handles standalone bonding import layout.
src/brpc/urma/urma_endpoint.h	Adds deferred completion callbacks and lifetime tracking.
src/brpc/urma/mock_urma.cpp	Checks SEND contracts and delays completion until copying finishes.
src/brpc/urma_transport.h	Exposes URMA upgrade resource operations.
src/brpc/urma_transport.cpp	Separates URMA data transfer from handshake and fallback orchestration.
src/brpc/ubshm/ub_endpoint.h	Removes handshake state and defers receive startup.
src/brpc/ubshm_transport.h	Exposes common-handshake resource operations.
src/brpc/ubshm_transport.cpp	Delegates connection orchestration to AdapterTransport.
src/brpc/transport_handshake.h	Defines shared handshake contracts and session state.
src/brpc/transport_handshake.cpp	Implements negotiation, activation, and fallback sequencing.
src/brpc/transport_factory.h	Updates the factory description comment.
src/brpc/transport_factory.cpp	Creates AdapterTransport for supported socket modes.
src/brpc/socket.h	Grants common handshake components socket access.
src/brpc/rdma/rdma_handshake.h	Removes legacy RDMA handshake interfaces.
src/brpc/rdma/rdma_handshake.cpp	Removes the legacy RDMA handshake implementation.
src/brpc/rdma/rdma_handshake_server.h	Removes the old server handshake interface.
src/brpc/rdma/rdma_handshake_server.cpp	Removes the old server fallback implementation.
src/brpc/rdma/rdma_endpoint.h	Keeps resource operations separate from handshake orchestration.
src/brpc/rdma_transport.h	Exposes RDMA upgrade operations and adapter factories.
src/brpc/rdma_transport.cpp	Integrates RDMA resources with common orchestration.
src/brpc/rdma_handshake.proto	Provides the relocated RDMA v3 schema.
src/brpc/policy/transport_handshake_protocol.h	Declares the shared handshake parser.
src/brpc/policy/transport_handshake_protocol.cpp	Routes parsing through AdapterTransport.
src/brpc/policy/rdma_handshake_protocol.h	Retains the legacy parser compatibility interface.
src/brpc/policy/rdma_handshake_protocol.cpp	Forwards legacy entry points to the shared parser.
src/brpc/input_messenger.h	Declares transport-aware TCP read stopping.
src/brpc/input_messenger.cpp	Stops TCP draining when the transport changes.
src/brpc/handshake/urma_handshake.h	Declares common URMA adapters and legacy helpers.
src/brpc/handshake/urma_handshake.cpp	Implements URMA v2/v3 codecs and server integration.
src/brpc/handshake/urma_fallback_handshake.cpp	Supports URMA rejection in TCP-only builds.
src/brpc/handshake/ubshm_handshake.h	Defines UBSHM handshake fields and adapter interfaces.
src/brpc/handshake/ubshm_handshake.cpp	Implements UBSHM negotiation and server fallback.
src/brpc/handshake/rdma_handshake.h	Defines common RDMA protocol adapters.
src/brpc/handshake/rdma_handshake.cpp	Implements RDMA codecs and server integration.
src/brpc/handshake/rdma_handshake_constants.h	Defines shared RDMA frame specifications.
src/brpc/handshake/handshake_io.h	Defines blocking and buffered handshake I/O.
src/brpc/handshake/handshake_io.cpp	Implements socket-backed handshake reads and writes.
src/brpc/handshake/handshake_frame.h	Defines frame specifications and codec operations.
src/brpc/handshake/handshake_frame.cpp	Implements framing and length validation.
src/brpc/handshake/handshake_adapter.h	Defines the server parser adapter interface.
src/brpc/handshake/handshake_adapter.cpp	Preserves parser context across handshake steps.
src/brpc/global.cpp	Registers the shared parser under the existing protocol identifier.
src/brpc/CMakeLists.txt	Updates the RDMA protobuf path.
src/brpc/adapter_transport.h	Defines top-level transport selection and orchestration.
Makefile	Includes handshake sources and fallback protobuf dependencies.
docs/cn/handshake_common_design.md	Documents architecture, fallback, and data-path handoff.
BUILD.bazel	Updates protobuf discovery and URMA source selection.

_{Review effort: Balanced}

+SocketHandshakeIO::SocketHandshakeIO(Socket* socket)
+    : _socket(socket)
+    , _read_butex(bthread::butex_create_checked<butil::atomic<int> >()) {
+}


+        [this](const timespec* duetime) {
+            return _socket->WaitEpollOut(
+                _socket->fd(), true, duetime);
+        });


wwbmmm · 2026-10-25T09:46:02Z

What's your plan to deal with this PR and #3535? It seems that they are conflicting.
I suggest to work on #3535 first, resolve the comments and merge that PR. Then we deal with this PR.

zchuango · 2026-10-26T03:03:43Z

What's your plan to deal with this PR and #3535? It seems that they are conflicting. I suggest to work on #3535 first, resolve the comments and merge that PR. Then we deal with this PR.

#3645 builds on #3535, so I’ll focus on resolving the remaining review comments in #3535 first. Once it is merged, I’ll rebase #3645 onto master so that it contains only the URMA-specific changes. I’ll hold off on requesting further review here until then.

Copilot

🔵 Needs a closer look

The handshake counter initialization defect remains unresolved, and the cross-transport state-machine and concurrent completion-lifecycle changes warrant final human review.

3 open issues

1 resolved since last review

Reset receive buffers before TCP fallback

_{Review effort: Balanced}

+SocketHandshakeIO::SocketHandshakeIO(Socket* socket)
+    : _socket(socket)
+    , _read_butex(bthread::butex_create_checked<butil::atomic<int> >()) {
+}


Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues

_{Review effort: Lite}

+int SocketHandshakeIO::PushBack(const void* data, size_t len) {
+    CHECK(_socket != nullptr);
+    if (len != 0) {
+        return _socket->fd_input_processor().read_buf().append(data, len) == 0 ? 0 : -1;


Spicy-cream and others added 11 commits October 8, 2026 23:58

Unify transport handshakes on current upstream master

f318a68

Merge current upstream master and preserve IPC V2 handshake compatibi…

d9fc07d

…lity Resolve UBSHM handshake conflicts using the tested rebased tree. Retain backend-specific format negotiation, remote shared-memory lengths, and upstream IPC V2 tests.

Fix certificate name extraction with const OpenSSL accessors

2e51361

Merge pull request #75 from Spicy-cream/review/pr3535-sync-master-oct15

7ebd2f2

Resolve current master conflicts and preserve UBSHM IPC V2 negotiation

Merge current master and retain shared memory name validation

3da2dde

Merge pull request #79 from Spicy-cream/review/pr3535-shm-conflict-oct18

d819d93

Resolve master conflict and preserve shared memory name safety

Merge modular CMake master and preserve handshake proto generation

ebcb46d

Merge pull request #82 from Spicy-cream/review/pr3535-cmake-conflict-…

29f3185

…refresh Resolve modular CMake conflict and preserve handshake protobuf generation

Merge Apache master into URMA handshake

6afd001

Preserve the common handshake source directory together with the upstream FlatBuffers conditional sources in Makefile. Keep upstream workflows and existing URMA documentation unchanged.

Merge Apache master and preserve handshake source discovery

bacfda1

Spicy-cream added 3 commits October 23, 2026 01:58

Retrigger community CI

96babbd

Rename the common transport handshake protocol

9ae366f

Apply handshake review style and test cleanup

0b9a6fd

wwbmmm requested a balanced review from Copilot October 24, 2026 07:25

Copilot started reviewing on behalf of wwbmmm October 24, 2026 07:25

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

Spicy-cream added 7 commits October 25, 2026 20:20

Fix transport handshake cleanup and dispatch

822d5de

Handle handshake startup errors and preserve failure causes

cebd46c

Retrigger transport handshake CI

9ab0f4d

Release URMA fallback buffers and preserve handshake errors

0c36adb

Move legacy URMA handshake peers into interoperability tests

071ee1b

Synchronize handshake read cancellation test and clarify protocol naming

c0ef0ea

Condense the common handshake design document

82cfd76

wwbmmm requested a balanced review from Copilot October 26, 2026 07:39

Copilot started reviewing on behalf of wwbmmm October 26, 2026 07:45

View session

Copilot AI reviewed Oct 26, 2026

View reviewed changes

Comment thread src/brpc/handshake/handshake_io.cpp

SocketHandshakeIO::SocketHandshakeIO(Socket* socket)

: _socket(socket)

, _read_butex(bthread::butex_create_checked<butil::atomic<int> >()) {

}

Spicy-cream added 3 commits October 26, 2026 01:12

Forward application connection stop notifications through the adapter

f3012a3

Initialize handshake wait state and stop writes on socket failure

0296dd4

Stop handshake reads before consuming data from failed sockets

baf2aca

wwbmmm requested a balanced review from Copilot October 26, 2026 09:44

Copilot started reviewing on behalf of wwbmmm October 26, 2026 10:00

View session

Copilot AI reviewed Oct 26, 2026

View reviewed changes

Comment thread src/brpc/handshake/handshake_io.cpp Outdated

int SocketHandshakeIO::PushBack(const void* data, size_t len) {

CHECK(_socket != nullptr);

if (len != 0) {

return _socket->fd_input_processor().read_buf().append(data, len) == 0 ? 0 : -1;

Spicy-cream added 2 commits October 26, 2026 18:44

Merge transport handshake fixes into URMA integration

51d2e18

Restore handshake pushback bytes before buffered input

2c38095

Navigation Menu

Repository navigation

feat: integrate URMA with the common transport handshake - #3645

feat: integrate URMA with the common transport handshake#3645
zchuango wants to merge 26 commits into
apache:masterfrom
LinQuickDev:urma-handshake

zchuango commented Oct 23, 2026

Uh oh!

Spicy-cream commented Oct 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

wwbmmm commented Oct 25, 2026

Uh oh!

zchuango commented Oct 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

zchuango commented Oct 23, 2026

What problem does this PR solve?

What is changed and the side effects?

Validation

Check List

Uh oh!

Spicy-cream commented Oct 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

wwbmmm commented Oct 25, 2026

Uh oh!

zchuango commented Oct 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants