Skip to content

Drain buffered inbound connections on cancel to avoid writer trap - #151

Open
onlyarnav wants to merge 1 commit into
swift-server:mainfrom
onlyarnav:fix/drain-buffered-connections-on-cancel
Open

onlyarnav wants to merge 1 commit into
swift-server:mainfrom
onlyarnav:fix/drain-buffered-connections-on-cancel

Conversation

@onlyarnav

Copy link
Copy Markdown

Motivation

Resolves #137.

When cancelling the task running NIOHTTPServer.serve, child channels that have already been accepted by the listening socket but not yet yielded by the inbound connection stream's for try await iteration are dropped when NIOThrowingAsyncSequenceProducer transitions directly to .cancelled. Because child channels wrap their underlying NIO channel synchronously with closeOnDeinit: false, deallocating them without finishing their outbound NIOAsyncWriter triggers a fatal error:

NIOCore/NIOAsyncWriter.swift:177: Fatal error: Deinited NIOAsyncWriter without calling finish()

Modifications

  • Added serveInboundConnections and AcceptLoopCoordinator in NIOHTTPServer.swift:
    • On outer task cancellation, the accept loop continues to drain remaining buffered connections yielded by connectionStream as the listening socket closes cleanly via .sourceFinished.
    • Dispatched connections run inside executeThenClose in withDiscardingTaskGroup, finishing their NIOAsyncWriter cleanly on cancellation.
    • Once the stream finishes draining, group.cancelAll() cancels active child tasks so in-flight requests abort promptly.
  • Refactored serveInsecureHTTP1_1 and serveSecureUpgrade to delegate connection serving to serveInboundConnections.
  • Added reproduction test testCancellationWithBufferedAcceptedConnections in ConnectionLifecycleTests.swift covering bursts of 32 connections across 5 iterations on both .plaintextHTTP1_1 and .http1_1.

Result

Cancelling NIOHTTPServer.serve while connections are buffered drains and terminates them cleanly without trapping.

### Motivation

Resolves swift-server#137.

When cancelling the task running `NIOHTTPServer.serve`, child channels that
have already been accepted by the listening socket but not yet yielded by
the inbound connection stream's `for try await` iteration are dropped when
`NIOThrowingAsyncSequenceProducer` transitions directly to `.cancelled`.
Because child channels wrap their underlying NIO channel synchronously with
`closeOnDeinit: false`, deallocating them without finishing their outbound
`NIOAsyncWriter` triggers a fatal error:

    NIOCore/NIOAsyncWriter.swift:177: Fatal error: Deinited NIOAsyncWriter without calling finish()

### Modifications

- Added `serveInboundConnections` and `AcceptLoopCoordinator` in `NIOHTTPServer.swift`.
  On outer task cancellation, the accept loop continues to drain remaining buffered
  connections yielded by `connectionStream` as the listening socket closes via `.sourceFinished`.
- Each drained connection is dispatched to `withDiscardingTaskGroup` where its `NIOAsyncWriter`
  is safely finished via `executeThenClose`.
- Once the stream finishes draining, `group.cancelAll()` cancels active child tasks so
  in-flight requests terminate promptly.
- Refactored `serveInsecureHTTP1_1` and `serveSecureUpgrade` to delegate connection serving
  to `serveInboundConnections`.
- Added reproduction test case `testCancellationWithBufferedAcceptedConnections` in
  `ConnectionLifecycleTests.swift` covering both plaintext HTTP/1.1 and TLS HTTP/1.1.

*Note: Antigravity was used to assist with development, root cause analysis, and test verification.*

### Result

Cancelling `NIOHTTPServer.serve` while connections are buffered drains and terminates
them cleanly without trapping.
@onlyarnav

Copy link
Copy Markdown
Author

Test Status:

  • Tests run & verified:
    • Added reproduction test testCancellationWithBufferedAcceptedConnections passed 100% on Linux (swift:6.4-resolute container) across all iterations for both .plaintextHTTP1_1 and .http1_1.
    • Full ConnectionLifecycleTests suite passed cleanly (all 9 tests passed in 1.112s).
  • Tests deferred to CI:
    • The entire test suite (swift test across the full repo) was initiated in Docker, but compiling and executing across all 2,000+ dependency targets was taking too long locally, so full matrix verification is deferred to GitHub Actions CI.

(Note: Google Antigravity pair programming assistant was used to assist with root cause analysis, test verification, and development.)

@onlyarnav

Copy link
Copy Markdown
Author

I am relatively new to Swift and SwiftNIO, so please bear with me if repetitive feedback or extra iterations are needed during code review. Happy to make any adjustments required!

@FranzBusch FranzBusch left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might need to fix this at the NIO level instead of here so it is fixed for all users. @aryan-25 @gjcairo this is the existing problem of NIOAsyncChannel where if we stop the iteration of it while there are still elements in there we crash. Lately, I was thinking that the right fix might be to close the underlying channel pipeline on cancel but still return all the elements to the user. To ensure that no element is dropped. This would be a localized fix in the NIOAsyncSequenceProducer + the channel handler.

@onlyarnav

Copy link
Copy Markdown
Author

Thank you for clarification! fixing this directly in swift-nio (NIOThrowingAsyncSequenceProducer / NIOAsyncChannelHandler) would make complete sense and solve the root problem rather than working around it at the server level

can i try to implement the upstream fix in apple/swift-nio if thats the preferred direction?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cancelling serve traps with "Deinited NIOAsyncWriter without calling finish()" when accepted connections are still buffered

2 participants