Repository navigation
Heap corruption (0xc0000374) on disconnect: realtime transport stopped after per-context cleanup frees shared state #98
Description
Summary
Intermittent heap corruption (Windows fatal exception: code 0xc0000374, STATUS_HEAP_CORRUPTION) when a multi-instrument TradingSuite (with features=["orderbook"]) disconnects — i.e. on reconnect, restart, or the daily CME session-boundary feed rollover. The process dies hard; a scheduler/supervisor sees a non-zero exit and respawns.
Version: project-x-py 3.5.9 (Windows, Python 3.14). Config: TradingSuite.create(instruments=[...4 index futures...], timeframes=["1min","5min","15min","30min","1hr"], features=["orderbook"]).
Faulthandler stack at the crash
bounded_statistics.py: get_statistics
bounded_statistics.py: _cleanup_counters
bounded_statistics.py: cleanup_bounded_statistics
realtime_data_manager/core.py: cleanup # gated by `if self.use_bounded_statistics:`
trading_suite.py: _disconnect_single_context
Concurrently, other threads in the same dump are inside realtime_data_manager/data_processing.py _update_timeframe_data / _process_tick_data (i.e. live SignalR tick dispatch).
Root cause (teardown ordering race)
At disconnect, RealtimeDataManager.cleanup() runs while the shared SignalR reader thread is still alive, because the realtime transport is stopped last:
TradingSuite.disconnect()→_disconnect_instrument_contexts()runs each context'sdata.cleanup(), which (a) callscleanup_bounded_statistics()→_cleanup_counters()sweeping the per-counter stats under an asyncio lock, and (b) clears the Polars frames / tick deques.realtime.disconnect()(which actually stops the SignalR user+market hubs and joins the receive thread) is only awaited after those per-context cleanups.
An asyncio lock does not coordinate the SignalR worker threads, so a trade/quote can be dispatched into a manager whose stats/frames are being torn down → free-during-read → native heap corruption. It surfaces intermittently (only when a tick lands in the teardown window) and always at a disconnect.
Suggested fix
Stop the realtime transport first, so the reader threads are joined before any per-context state is freed:
- In
TradingSuite.disconnect(),await self.realtime.disconnect()(or the equivalent hub-stop) before_disconnect_instrument_contexts()/ the per-contextdata.cleanup().realtime.disconnect()preserves callbacks/subscriptions, so moving it earlier is state-safe.
A secondary hardening: the disconnect-time cleanup_bounded_statistics() sweep does no useful work at teardown (the counters are about to be GC'd) — skipping it there removes one more concurrent reader of shared state. The periodic in-session cleanup (via the CleanupScheduler) is a separate path and can stay.
Workaround (for anyone else hitting this)
Downstream, we invert the order in our own broker wrapper — await suite.realtime.disconnect() before suite.disconnect() — and separately monkeypatch BoundedStatisticsMixin.cleanup_bounded_statistics to drop the disconnect-time counter sweep. Both eliminate the crash in our testing across repeated reconnects/rollovers.
Happy to open a PR with the reorder if that's useful.
Activity
fix: stop realtime feed before freeing data frames on disconnect (Tex…
TexasCoding commentedon Aug 30, 2026
v4.0.0 (#102) now disconnects realtime before freeing Polars frames, but the Windows 0xc0000374 heap crash itself is still unverified. Leaving this open.
fix(realtime): skip bounded-stats sweep on data manager teardown (#147)
TexasCoding commentedon Sep 27, 2026
Daily maintenance note (2026-09-27)
Both suggested hardenings are already on
main/ PyPI 4.3.2:TradingSuite.disconnect()awaitsrealtime.disconnect()(hub stop + join) before per-context cleanup — since v4.0.0 (v4.0.0: Revive SDK for the current TopstepX Gateway #102).RealtimeDataManager.cleanup()skips the bounded-statistics counter sweep at teardown — fix(realtime): skip bounded-stats sweep on data manager teardown (#98) #147 / v4.3.2.Leaving this issue open until someone confirms the Windows
0xc0000374crash is gone on a multi-instrument + orderbook reconnect/rollover (still not reproducible in CI). Please comment here if you still see the crash on ≥4.3.2, or if it is resolved.TexasCoding commentedon Sep 28, 2026
Daily maintenance (2026-09-28): no new code change. Suggested hardenings remain on
main/ PyPI 4.3.2. Still waiting on Windows multi-instrument + orderbook reconnect/rollover confirmation that0xc0000374is gone. Please comment if still repro on ≥4.3.2 or if resolved.blurxy commentedon Sep 30, 2026
Field report from a Windows 11 / Python 3.14 user running multi-instrument realtime + orderbook
(NQ/ES/MES/MNQ, reconnects and daily rollover), in case it helps close this out.
Caveat first: we are pinned to a fork of 3.5.9 (signalrcore transport), so this is not the
What we found (3.5.9 / signalrcore): the crash we hit was a steady-state race, not a teardown
one. signalrcore's
WebSocketClientreads frames on one thread (_recv_frame->_recv_exactly->
sock.recv) whileConnectionStateCheckersends the keep-alive ping from a second thread(
ping_function->WebSocketClient.send->sock.sendall, everykeep_alive_interval=15s).Both hit the same
ssl.SSLSocket. OpenSSL SSL objects must not be used from two threads at once,so an overlapping
SSL_read+SSL_writecorrupts the heap ->0xc0000374(process exit3221226356). It is intermittent because the ping has to overlap a read inside OpenSSL; CPU
contention widens the window.
Our fix: a per-connection
threading.Lockaround the socket I/O.sendtakes the lock;_recv_exactlyreads under the lock with a bounded socket timeout (1s) and releases it betweenattempts, so the ping is never starved and the two SSL calls never overlap. The keep-alive stays on.
Result: at least 8
0xc0000374crashes between 2026-04-16 and 2026-07-09 (7 process exits inour scheduler log plus the 07-09 one faulthandler caught, which is the stack we diagnosed from).
0 in the 82 days since the lock went in (2026-07-09 -> 2026-09-29), on three channels: the
scheduler's exit log (continuous), the faulthandler crash dump (never written again after 07-09),
and the Windows Application event log (which only retains back to 2026-08-05 here). Honest caveat:
we also disabled the bounded-statistics / dynamic-limits sweep the day before, so we can't
attribute the fix to one change alone.
Why this may already be moot on 4.x: 4.x moved to
pysignalr, which is asyncio +websockets(single event loop, no separate reader and ping threads), so this particular race should not exist
structurally there. That would fit "not reproducible in CI". If someone is still on 3.x with
signalrcore, the lock above is the fix we'd suggest.
Happy to share the monkeypatch if it's useful. Thanks for maintaining this.
TexasCoding commentedon Sep 30, 2026
Daily maintenance (2026-09-30) — reply to @blurxy's field report
Thank you — this is genuinely useful.
On your 3.5.9 / signalrcore finding: the SSL read/write race (
WebSocketClient._recv_*thread vsConnectionStateCheckerping thread sharing onessl.SSLSocket) is a credible root cause for intermittent0xc0000374, and the per-connection I/O lock you landed matches that diagnosis. Agree that anyone still on 3.x + signalrcore should treat that lock as the fix path.On 4.x:
main/ PyPI ≥4.3.2 (current 4.3.3) usespysignalr+websocketson a single asyncio event loop (src/project_x_py/realtime/async_hub.py), so that particular dual-thread OpenSSL race should not exist structurally. The earlier teardown hardenings (TradingSuite.disconnect()awaits hub stop before per-context cleanup;RealtimeDataManager.cleanup()skips the bounded-statistics sweep) also remain onmain.Still open: we still need a confirmation from someone on ≥4.3.2 (pysignalr stack) with multi-instrument + orderbook reconnect/rollover that
0xc0000374is gone — your report is explicitly not that confirmation (fork of 3.5.9). If you or anyone else can try stock 4.3.3 on Windows 11 with that workload, please comment yes/no + repro notes.Leaving this issue open until we get that ≥4.3.2 signal. Happy to look at a monkeypatch gist for the 3.x docs/FAQ if you want to share it.
blurxy commentedon Oct 22, 2026
Follow-up field data. Windows 11, Python 3.14, NQ/ES/MES/MNQ + orderbook, daily rollovers and reconnects. Still our 3.5.9 fork on signalrcore, so this is still not the >=4.3.2 confirmation you asked for.
Scheduler exit codes before and after 2026-07-09, when we landed the SSL I/O lock together with the disconnect reorder, the teardown bounded-stats skip and a per-tick bar-write coalescer:
Four changes landed the same day, so this does not isolate the lock. It is consistent with your read that 4.x (pysignalr on one event loop) removes the dual-thread OpenSSL race. We have not run stock 4.3.x, so we cannot close this for you.
On the gist: yes, we can share our 3.x per-connection I/O lock monkeypatch for the docs/FAQ.
Separately: we carry a small bracket fix. When an entry is in an undefined state, it polls the fill status up to 6 x 1 s before abandoning, where main rechecks once (order_manager/bracket_orders.py:651). It comes with a test. We are happy to open a PR if that is useful.
TexasCoding commentedon Oct 25, 2026
Thanks @blurxy, this is really helpful. Yes please on both: a gist of the 3.x I/O-lock monkeypatch would be great for the FAQ, and a PR for the bracket fill-status recheck (with the test) is welcome; please target main. Leaving #98 open until we get a stock ≥4.3.2 confirmation.
blurxy commentedon Oct 28, 2026
Both are up.
It wraps
WebSocketClient.sendand_recv_exactlywith one lock per connection. The read uses a timeout (default 1 s,PROJECTX_WS_RECV_TIMEOUT_MS) and releases the lock between attempts, so the keepalive ping can still send. Callapply()once before the SDK connects.PROJECTX_WS_SSL_LOCK=0turns it off. This is the file we run in production, unchanged.main: fix(bracket): poll fill status before abandoning an undefined-state entry #179. It has two tests, and both fail onmainwithout the change.We still have not run stock 4.3.x, so this is still not the >=4.3.2 confirmation for this issue.
chore(release): v4.4.2