[Connor-Matthew]

Description

Replaces #54 after renaming the source branch from codex/tent4-integration to ub/tent4-integration. The code commits are unchanged.

Migrate only original TENT 4 commit d78e20dd3f9568c8223fbdbb3e98e00f6ed34c67 onto official kvcache-ai/Mooncake main at 1a6d00d0f4dc7f54c573c5fcd4ff5f80bd1f2813, after TENT 3 landed upstream in kvcache-ai#4605.

This adds receiver-credit authority/exchange, runtime dispatch gating, transport session fences, staging propagation, and the native UB validation guide. Conflict resolution preserves current-main cancellation, safe TCP retry, payload limits, registered-memory permissions, copy-error propagation, runtime policy snapshots and submit-stage failover. Required-credit scatter submissions are rejected explicitly because scatter bypasses the credit-owning runtime queue; a regression test covers the rejection before transport handoff. A separate migration compatibility fix validates the receiver session/epoch against the pinned HP TCP metadata before initial or retry handoff, with missing/stale/version/flags rejection and matching-session success coverage. HP TCP was introduced upstream after the original TENT 4 snapshot.

Related design discussions: kvcache-ai#2849 (receiver-advertised credits, closed historical RFC) and kvcache-ai#3093 (native UB/URMA). This is TENT 4 follow-up work, distinct from the existing TENT 3 integration PR #53 and native UB/bonding PR #40. No duplicate TENT 4 PR was open when checked.

Base note: the fork's main has been synced to official kvcache-ai/Mooncake main, and official main was merged into this branch (latest 93bc4ff5). The commits on top of official main are the migrated TENT 4 change, the HP TCP compatibility fixes, and the staging/receiver-credit regression test.

Validation PR: do not merge. Keep draft until human review and the owner's two-node UB traffic/data-integrity validation finish.

Resolved: staged failure released receiver credit early

Static review found that original TENT 4 could release an outer receiver-credit reservation after a staged parent reported failure while credit-exempt sibling chunks were still writing into the receiver. That would let the receiver grant the same capacity to another sender while the old write was still landing.

Official main fixed the underlying lifetime in kvcache-ai#4720: ProxyManager now keeps the staged parent PENDING until in-flight sibling copies and remote operations are terminal, so the queued owner (and its credit) is not finished early. This PR adds an end-to-end regression, RuntimeQueueDispatch.StagedFailureKeepsReceiverCreditUntilChunksDrain, through the runtime queue and receiver-credit gate:

• chunk 2 fails while chunk 1's cross stage is still writing into the receiver;
• the staged owner stays PENDING, and a second credit-gated write is deferred and never reaches the transport;
• once the held write drains, the owner fails, returns the credit, and the second write completes with correct data.

Reverting the kvcache-ai#4720 drain makes all three checks fail. The test passes 400/400 under 8-way concurrent repetition. CI hosts have no device memory, so a test-only setStagingPolicyForTest hook forces host staging for selected requests.

Known limitation: credit held by senders that exit uncleanly is not reclaimed

ReceiverCreditAuthority lowers a sender's outstanding credit only when that sender reports a release. If a sender exits uncleanly (crash, kill, or engine shutdown with committed in-flight work), the credit it held at that moment stays outstanding, and the sender keeps a max_senders slot, until the receiver process restarts and starts a new session.

This is a deliberate, conservative choice: RDMA/UB writes are one-sided, and the receiver cannot prove that a silent sender's old DMA is gone. Reclaiming on silence alone could hand capacity to another sender while a stale write is still landing. The impact is bounded:

• receiver credit is off by default (receiver_credit.enable = false);
• the failure mode is capacity exhaustion (senders wait for credit), never data corruption;
• each unclean exit leaks only the credit in flight at that moment, not the sender's total grants;
• restarting the receiver clears it.

Possible follow-up (not in this PR): when a grant is denied for capacity, probe senders that hold outstanding credit and reclaim only on positive proof that the instance is gone: connection refused (process exited, so its QPs/Jettys are torn down) or a different instance ID (process restarted). Unreachable senders would stay unreclaimed by default.

Suggested Merge Priority

• P0 — Urgent
• P1 — High
• P2 — Normal
• P3 — Low

Module

• Transfer Engine (mooncake-transfer-engine)
• Docs

Type of Change

• New feature
• Documentation update

How Has This Been Tested?

Test commands:

# macOS standalone builds use clang++ -std=c++20, Homebrew gtest/glog,
# tent/include, and the production status.cpp + relevant runtime source.
/tmp/mooncake-tent4-migration/receiver_credit_test --gtest_brief=1
/tmp/mooncake-tent4-migration/admission_queue_test --gtest_brief=1
/tmp/mooncake-tent4-migration/request_merge_test --gtest_brief=1
# Request merge test links the production merge function extracted unchanged
# from transfer_engine_impl.cpp, avoiding Linux-only engine dependencies.
make -C docs html SPHINXBUILD=/tmp/mooncake-tent4-migration/venv/bin/sphinx-build
pre-commit run --files $(git diff --name-only --diff-filter=ACMR 1a6d00d0f...HEAD)

# Linux (Ubuntu 22.04 arm64 container), -DUSE_TENT=ON -DUSE_CUDA=OFF:
./tent_runtime_queue_dispatch_test
./tent_runtime_queue_dispatch_test --gtest_filter='*StagedFailure*' --gtest_repeat=100
./tent_proxy_shutdown_timeout_test

Test results: 26 receiver-credit + 50 admission-queue + 11 request-merge tests pass (87 total). Before migration, the official baseline passed its 14 receiver-credit + 43 admission-queue tests. Documentation HTML builds successfully and the changed pages/navigation were inspected. All applicable pre-commit hooks pass, including clang-format 20.1.8 on changed lines and codespell.

• Targeted portable unit tests pass
• Full Linux/TENT CI passes (all 31 checks; tent-ci ran 66 TENT tests, 0 failed)
• New scatter, HP TCP session-fence, and staged-failure receiver-credit regressions execute in Linux CI (tent_runtime_queue_dispatch_test, tent_hp_tcp_transport_test)
• Native UB two-node integration and traffic/data-integrity tests pass (owner will run on server)

Checklist

• Agent self-review performed
• Code formatted using ./scripts/code_format.sh via the pre-commit hook
• Pre-commit run on changed files; applicable hooks pass
• Documentation updated
• Original feature tests migrated; scatter and HP TCP compatibility regressions added
• Existing RFC discussions linked above for this >500 LOC change
• Human submitter has reviewed every changed line and can defend the change end-to-end

AI Assistance Disclosure

• No AI tools were used
• AI tools were used

Codex migrated the original commit, resolved conflicts against current main, added the scatter compatibility guard/regression, and ran the available local checks. Independent Codex agents reviewed migration risks. An AI coding assistant also wrote the staged-failure receiver-credit regression and the test-only staging hook, verified that reverting kvcache-ai#4720 makes it fail, and analyzed the unclean-exit credit limitation documented above. The human submitter's full review and hardware validation remain pending; this PR must stay unmerged.