Skip to content

Make remote connections recover reliably#867

Merged
arul28 merged 25 commits into
mainfrom
ade/relay-end-to-end-resilience-3fd52b73
Jul 21, 2026
Merged

Make remote connections recover reliably#867
arul28 merged 25 commits into
mainfrom
ade/relay-end-to-end-resilience-3fd52b73

Conversation

@arul28

@arul28 arul28 commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Summary

  • coordinate Clerk refresh-token rotation and keep trusted secure routes reachable across stale presence
  • guarantee fair CRDT delivery and recover acknowledgements without socket churn
  • roll long terminal transcripts with monotonic offsets and repair gaps on web and iOS
  • harden relay readiness, authorization renewal, frame limits, and hibernation
  • add v1.2.33 release notes for desktop, web, and iOS build 36

Validation

  • quality and test skills: pass
  • desktop typecheck, lint, all 8 shards, build: pass
  • ADE CLI typecheck, 2,113 tests, build: pass
  • web typecheck and production build: pass
  • tunnel-relay unit and workerd suites: pass
  • iOS targeted recovery and pairing suites: pass
  • documentation validation: 198 files

Release plan

After merge: deploy hosted web client and tunnel Worker, tag v1.2.33 and verify/publish per-arch desktop artifacts, then archive/upload/distribute iOS 1.1.10 build 36 with widgets and App Clip.

ADE   Open in ADE  ·  ade/relay-end-to-end-resilience-3fd52b73 branch  ·  PR #867

Summary by CodeRabbit

  • New Features

    • Machines with verified relay routes can remain connectable despite stale or offline presence.
    • Added more reliable terminal snapshots, input acknowledgements, retries, and recovery across reconnects.
    • Improved relay connection negotiation, authorization renewal, and connection-attempt handling.
    • Relay health reporting now includes protocol and deployment version details.
  • Bug Fixes

    • Improved token refresh and session validation during account and machine operations.
    • Prevented stale terminal data, duplicate input, and lost synchronization updates.
  • Documentation

    • Clarified machine availability, relay behavior, and local relay testing guidance.

Greptile Summary

This PR improves recovery for remote connections and terminal sessions. The main changes are:

  • Coordinates rotating account credentials across processes.
  • Keeps verified relay routes usable across stale presence.
  • Improves relay readiness, authorization renewal, and socket recovery.
  • Adds fair CRDT delivery and acknowledgement recovery.
  • Rolls terminal transcripts with monotonic offsets and repairs client gaps.
  • Adds release notes for desktop, web, and iOS.

Confidence Score: 5/5

This looks safe to merge.

The updated PTY pipeline canonicalizes split and malformed surrogate sequences before transcript accounting and broadcast. Pending high surrogates are flushed during exit and disposal.

No blocking issue related to the previous terminal offset failure remains.

T-Rex T-Rex Logs

What T-Rex did

  • T-Rex ran the PR acceptance harness against the merge-base revision 6b7f66a, but the setup failed with HTTP 401 instead of WebSocket 101.
  • T-Rex executed a local vitest run from the tunnel-relay app using npm exec -- vitest run --config vitest.workerd.config.ts --reporter=verbose, which completed with exit code 0 and 2 files and 6 tests passing in 1.02s.
  • The successful run explicitly validated that attachments and routing for an established epoch-v2 triple were preserved and that epoch-v2 attachments and routing are restored across deterministic hibernation eviction.

View all artifacts

T-Rex Ran code and verified through T-Rex

Important Files Changed

Filename Overview
apps/desktop/src/main/services/pty/ptyService.ts Adds canonical PTY output handling, rolling transcript retention, monotonic byte offsets, and exact snapshot reads.
apps/desktop/src/main/services/pty/ptyService.test.ts Covers split surrogate pairs, malformed surrogate replacement, and teardown flushing.

Reviews (2): Last reviewed commit: "fix(sync): close remaining recovery gaps" | Re-trigger Greptile

arul28 and others added 24 commits July 21, 2026 13:55
Average users on the iOS connection surfaces should see only concrete
states and machine names, never the transport route. Two primary
machine-row sites still leaked the route kind, and a pairing help line
recommended one route as fastest.

- Account machine rows (Account + Settings > Machines) rendered
  machine.routeLabel (Tailscale / Local network / ADE relay) as their
  subtitle. Use the presence-only machineStatusHint instead, matching
  the saved rows in the same list. Route kind stays in the Connection
  details diagnostics section.
- Pairing help no longer recommends Tailscale as fastest; it keeps the
  plain reachable-from-any-network message.
- Add a regression test locking the account row hint route-neutral even
  when the machine advertises tailnet/relay endpoints.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 21, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
ade Ignored Ignored Preview Jul 21, 2026 11:17pm

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The pull request updates account authentication, Relay transport negotiation and authorization, sync recovery, terminal transcript streaming, desktop and iOS connection flows, tunnel-relay deployment/testing, and machine reachability UI across the CLI, desktop, iOS, and worker applications.

Changes

Relay worker and deployment

Layer / File(s) Summary
Epoch-aware relay and deployment verification
.github/workflows/*, apps/tunnel-relay/*
Relay sockets use epoch-aware signatures, readiness negotiation, legacy migration, bounded forwarding, and expanded health metadata; deployment verifies the published worker tag and protocol version.
Relay test infrastructure
apps/tunnel-relay/test/*, apps/tunnel-relay/vitest.*, apps/tunnel-relay/package.json
Unit, Workerd, smoke URL, hibernation, health, and compatibility coverage is added with updated test scripts and dependencies.

Account and sync services

Layer / File(s) Summary
Account refresh and machine reachability
apps/ade-cli/src/services/account/*, apps/desktop/src/shared/accountDirectory.ts, apps/ios/ADE/Services/Account*.swift
Attestation failures receive typed codes, refresh rotation uses exact-session coordination, directory requests retry with refreshed tokens, publisher retries use backoff, and verified Relay routes keep stale-presence machines connectable.
Relay authorization lifecycle
apps/ade-cli/src/services/sync/relayAuthorization.ts, apps/desktop/src/renderer/webclient/sync/{connection,dpop}.ts, apps/desktop/src/shared/types/sync.ts
Relay reauthorization leases, canonical DPoP proofs, lifecycle validation, retry/expiry behavior, and handoff snapshots are implemented.
Sync host and peer recovery
apps/ade-cli/src/services/sync/{syncHostService,syncPeerService,brainProjectActionsSyncHandler}.ts
Lifecycle generation guards, connection-attempt arbitration, terminal ACK deduplication, snapshot barriers, chat deferral, and adaptive changeset recovery are added.

Terminal and client recovery

Layer / File(s) Summary
Logical transcript rollover
apps/desktop/src/main/services/pty/*
PTY transcripts use logical UTF-8 offsets, atomic rollover journals, retained windows, safe range boundaries, and snapshot APIs with recovery tests.
Webclient terminal recovery
apps/desktop/src/renderer/webclient/sync/*, apps/desktop/src/renderer/webclient/adapter/*, apps/desktop/src/renderer/components/terminals/*
Readiness generations, terminal input queues and ACK handling, snapshot replacement events, stale hydration cancellation, transport fallback, and remembered read results are implemented.
iOS sync and terminal recovery
apps/ios/ADE/Services/Sync*.swift, apps/ios/ADETests/*
Authenticated connection races, Relay negotiation, reauthorization, terminal snapshot recovery, input queues, outbound changeset recovery, and associated tests are added.

Machine connection presentation

Layer / File(s) Summary
Reachability and connection UI
apps/desktop/src/renderer/webclient/shell/*, apps/ios/ADE/Views/*, apps/ade-cli/src/tuiClient/*
Machine actions and status labels now reflect verified route availability and authenticated connection state rather than directory presence alone; connection error and reconnect copy is updated.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

  • arul28/ADE#554 — Related terminal streaming and byte-offset transcript handling.
  • arul28/ADE#705 — Related tunnel-relay health and epoch signature changes.
  • arul28/ADE#825 — Related account-machine discovery and verified Relay route selection.

Suggested labels: ci, desktop, ios, docs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 6.79% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is clearly related to the PR’s main theme of making remote connections more reliable.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ade/relay-end-to-end-resilience-3fd52b73

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread apps/desktop/src/main/services/pty/ptyService.ts Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🧹 Nitpick comments (1)
apps/ios/ADETests/PairingAndDpopTests.swift (1)

297-310: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Verify the relay proof cryptographically, not only by inequality.

These assertions would pass if the nonce and signature merely changed between calls, even when the signature was not generated over the canonical relay reauthorization challenge. Extract the proof fields and verify each signature against buildRelayReauthorizationChallenge(...); also assert that changing the nonce causes verification to fail.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@apps/ios/ADETests/PairingAndDpopTests.swift` around lines 297 - 310,
Strengthen testRelayReauthorizationProofUsesDistinctNoncePerAttempt by
extracting each proof’s nonce and signature, then cryptographically verifying
both signatures against buildRelayReauthorizationChallenge(...) using the
corresponding nonce and relay parameters. Also verify that substituting one
proof’s nonce into the other signature fails, while retaining the distinct-nonce
assertion.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@apps/desktop/src/renderer/webclient/sync/client.ts`:
- Around line 1207-1231: Guard payload.error before accessing retryable, code,
or message in the terminal input acknowledgment handling block. Update the
!payload.ok path in the terminal input queue flow to treat a missing or
malformed error object as a definitive failure via failTerminalInput, ensuring
the queued operation cannot remain stuck.

In `@apps/ios/ADE/Services/SyncTerminalInputQueue.swift`:
- Around line 147-153: Update SyncTerminalInputQueue.prepareForReconnect to
reset each queued item’s retry accounting when restoring delivery to .unsent:
clear attemptCount, firstSentUptime, and lastSentUptime along with delivery.
Preserve all other item data so the resumed connection receives a fresh retry
budget.

In `@apps/ios/ADE/Views/Settings/SettingsSupportTypes.swift`:
- Around line 84-97: Update statusLabel(for:canReconnectToSavedHost:) so the
.disconnected case always returns "Not connected"; reserve "Can't reach this
Mac" for the .unreachable case, preserving the existing neutral disconnected
styling and reconnect behavior elsewhere.

In `@apps/ios/ADE/Views/Work/SwiftTermSessionView.swift`:
- Line 113: Reset inputStatusMessage to nil when input is successfully accepted
or the stream re-hydrates, updating the relevant success and rehydration paths
in SwiftTermSessionView. Preserve the existing failure assignments for rejected
input while ensuring later blocked-input events fall back to the “Reconnecting”
caption.
- Around line 260-262: Update handleConnectionChange(isConnected:) to gate the
PTY resize re-assert on the reconnect-persistent desiredTerminalSessionIds
signal instead of subscribedTerminalSessionIds. Preserve the resize behavior
immediately after resetTerminalTransportStateForReconnect(), before
restoreTerminalSubscriptions() completes asynchronously.

In `@apps/tunnel-relay/src/tunnelDo.ts`:
- Around line 24-26: Update the “Close codes” table in the relay README to
document CLOSE_STALE_PIPE (4508), CLOSE_FORWARD_FAILED (4509), and
CLOSE_NOT_READY (4510), using the existing descriptions from the corresponding
constants in tunnelDo.ts so operators can interpret these codes during
debugging.

---

Nitpick comments:
In `@apps/ios/ADETests/PairingAndDpopTests.swift`:
- Around line 297-310: Strengthen
testRelayReauthorizationProofUsesDistinctNoncePerAttempt by extracting each
proof’s nonce and signature, then cryptographically verifying both signatures
against buildRelayReauthorizationChallenge(...) using the corresponding nonce
and relay parameters. Also verify that substituting one proof’s nonce into the
other signature fails, while retaining the distinct-nonce assertion.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 0555554e-514c-41fb-81b1-32d3a8dc7f4b

📥 Commits

Reviewing files that changed from the base of the PR and between 6b7f66a and 7b74efe.

⛔ Files ignored due to path filters (14)
  • CHANGELOG.md is excluded by !*.md
  • apps/ios/ADE.xcodeproj/project.pbxproj is excluded by !**/*.xcodeproj/project.pbxproj
  • apps/tunnel-relay/package-lock.json is excluded by !**/package-lock.json, !**/package-lock.json
  • changelog/index.mdx is excluded by !changelog/**
  • changelog/v1.2.33.mdx is excluded by !changelog/**
  • docs.json is excluded by !docs.json
  • docs/ARCHITECTURE.md is excluded by !docs/**
  • docs/features/sync-and-multi-device/README.md is excluded by !docs/**
  • docs/features/sync-and-multi-device/crdt-model.md is excluded by !docs/**
  • docs/features/sync-and-multi-device/ios-companion.md is excluded by !docs/**
  • docs/features/terminals-and-sessions/README.md is excluded by !docs/**
  • docs/features/terminals-and-sessions/pty-and-processes.md is excluded by !docs/**
  • docs/features/terminals-and-sessions/ui-surfaces.md is excluded by !docs/**
  • docs/features/web-client/README.md is excluded by !docs/**
📒 Files selected for processing (89)
  • .github/workflows/ci.yml
  • .github/workflows/deploy-web.yml
  • apps/ade-cli/README.md
  • apps/ade-cli/src/cli.ts
  • apps/ade-cli/src/services/account/accountAttestationVerifier.test.ts
  • apps/ade-cli/src/services/account/accountAttestationVerifier.ts
  • apps/ade-cli/src/services/account/accountAuthService.test.ts
  • apps/ade-cli/src/services/account/accountAuthService.ts
  • apps/ade-cli/src/services/account/accountMachineDirectoryService.test.ts
  • apps/ade-cli/src/services/account/accountMachineDirectoryService.ts
  • apps/ade-cli/src/services/account/accountMachinePublisherService.test.ts
  • apps/ade-cli/src/services/account/accountMachinePublisherService.ts
  • apps/ade-cli/src/services/sync/brainProjectActionsSyncHandler.ts
  • apps/ade-cli/src/services/sync/relayAuthorization.test.ts
  • apps/ade-cli/src/services/sync/relayAuthorization.ts
  • apps/ade-cli/src/services/sync/sharedSyncListener.ts
  • apps/ade-cli/src/services/sync/syncCloudRelayStore.ts
  • apps/ade-cli/src/services/sync/syncHostService.test.ts
  • apps/ade-cli/src/services/sync/syncHostService.ts
  • apps/ade-cli/src/services/sync/syncPeerService.test.ts
  • apps/ade-cli/src/services/sync/syncPeerService.ts
  • apps/ade-cli/src/services/sync/syncTunnelClientService.test.ts
  • apps/ade-cli/src/services/sync/syncTunnelClientService.ts
  • apps/ade-cli/src/tuiClient/__tests__/remoteLauncher.test.ts
  • apps/ade-cli/src/tuiClient/remoteLauncher.ts
  • apps/desktop/src/main/services/pty/ptyService.test.ts
  • apps/desktop/src/main/services/pty/ptyService.ts
  • apps/desktop/src/main/services/remoteRuntime/syncPairedMachineStore.test.ts
  • apps/desktop/src/main/services/remoteRuntime/syncPairedMachineStore.ts
  • apps/desktop/src/main/services/sync/syncHostService.test.ts
  • apps/desktop/src/main/services/usage/usageTrackingService.test.ts
  • apps/desktop/src/renderer/components/terminals/TerminalView.test.tsx
  • apps/desktop/src/renderer/components/terminals/TerminalView.tsx
  • apps/desktop/src/renderer/webclient/adapter/__tests__/adapter.test.ts
  • apps/desktop/src/renderer/webclient/adapter/analytics.ts
  • apps/desktop/src/renderer/webclient/adapter/app.ts
  • apps/desktop/src/renderer/webclient/adapter/infra/commandCaller.ts
  • apps/desktop/src/renderer/webclient/adapter/sessionsPty.ts
  • apps/desktop/src/renderer/webclient/shell/MachinePicker.tsx
  • apps/desktop/src/renderer/webclient/shell/WebClientRoot.tsx
  • apps/desktop/src/renderer/webclient/shell/WebShell.tsx
  • apps/desktop/src/renderer/webclient/shell/__tests__/MachinePicker.test.tsx
  • apps/desktop/src/renderer/webclient/shell/__tests__/WebClientRoot.test.tsx
  • apps/desktop/src/renderer/webclient/shell/shellTokens.ts
  • apps/desktop/src/renderer/webclient/sync/__tests__/sync.test.ts
  • apps/desktop/src/renderer/webclient/sync/client.ts
  • apps/desktop/src/renderer/webclient/sync/connection.ts
  • apps/desktop/src/renderer/webclient/sync/dpop.ts
  • apps/desktop/src/shared/accountDirectory.ts
  • apps/desktop/src/shared/types/sessions.ts
  • apps/desktop/src/shared/types/sync.ts
  • apps/ios/ADE/Services/AccountDirectory.swift
  • apps/ios/ADE/Services/AccountService.swift
  • apps/ios/ADE/Services/Database.swift
  • apps/ios/ADE/Services/DpopKeyService.swift
  • apps/ios/ADE/Services/SyncConnectionRace.swift
  • apps/ios/ADE/Services/SyncRecoveryPolicy.swift
  • apps/ios/ADE/Services/SyncService.swift
  • apps/ios/ADE/Services/SyncTerminalInputQueue.swift
  • apps/ios/ADE/Views/Account/AccountConnectionsSection.swift
  • apps/ios/ADE/Views/Account/MobileAccessGateView.swift
  • apps/ios/ADE/Views/Components/MachineRowView.swift
  • apps/ios/ADE/Views/Hub/HubQuickConnect.swift
  • apps/ios/ADE/Views/Settings/ConnectionSettingsView.swift
  • apps/ios/ADE/Views/Settings/SettingsConnectionHeader.swift
  • apps/ios/ADE/Views/Settings/SettingsDiagnosticsSection.swift
  • apps/ios/ADE/Views/Settings/SettingsPairingSection.swift
  • apps/ios/ADE/Views/Settings/SettingsSupportTypes.swift
  • apps/ios/ADE/Views/Work/SwiftTermSessionView.swift
  • apps/ios/ADE/Views/Work/TerminalSessionScreen.swift
  • apps/ios/ADE/Views/Work/WorkArtifactTerminalViews.swift
  • apps/ios/ADETests/ADETests.swift
  • apps/ios/ADETests/PairingAndDpopTests.swift
  • apps/ios/ADETests/SyncRecoveryPolicyTests.swift
  • apps/tunnel-relay/README.md
  • apps/tunnel-relay/package.json
  • apps/tunnel-relay/src/relay.ts
  • apps/tunnel-relay/src/tunnelDo.ts
  • apps/tunnel-relay/test/env.d.ts
  • apps/tunnel-relay/test/health.workerd.test.ts
  • apps/tunnel-relay/test/relay.test.ts
  • apps/tunnel-relay/test/relay.workerd.test.ts
  • apps/tunnel-relay/test/smoke-url.mjs
  • apps/tunnel-relay/test/smoke-url.test.mjs
  • apps/tunnel-relay/test/smoke.mjs
  • apps/tunnel-relay/tsconfig.json
  • apps/tunnel-relay/vitest.unit.config.ts
  • apps/tunnel-relay/vitest.workerd.config.ts
  • apps/tunnel-relay/wrangler.jsonc
💤 Files with no reviewable changes (1)
  • apps/desktop/src/main/services/remoteRuntime/syncPairedMachineStore.ts

Comment thread apps/desktop/src/renderer/webclient/sync/client.ts
Comment thread apps/ios/ADE/Services/SyncTerminalInputQueue.swift
Comment thread apps/ios/ADE/Views/Settings/SettingsSupportTypes.swift
Comment thread apps/ios/ADE/Views/Work/SwiftTermSessionView.swift
Comment thread apps/ios/ADE/Views/Work/SwiftTermSessionView.swift Outdated
Comment thread apps/tunnel-relay/src/tunnelDo.ts
@arul28
arul28 merged commit 0268bc6 into main Jul 21, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant