fix: bound the Redis read so a dead socket cannot park a worker forever - #3
Merged
Merged
Conversation
The PHP client carries the same defect that stalled the Python fleet on 2026-08-15, reached by a different route. `blPop(['ml_tasks'], 1)` already passed a timeout, so this looked safe. That timeout is the *server's* — it tells Redis how long to hold the pop before answering "nothing". It says nothing about how long this process will wait for that answer to arrive. phpredis defaults its read timeout to 0, meaning wait forever, and `connect()` was called without one. So when a connection is silently dropped the reply can never arrive, the read never returns, and the worker parks permanently on a socket the server has already discarded. - `connect()` now passes CONNECT_TIMEOUT and READ_TIMEOUT. - OPT_TCP_KEEPALIVE is set so the kernel probes an idle peer, guarded by defined() since it needs phpredis 5+. - The blocking pop uses BLPOP_TIMEOUT, which is held below READ_TIMEOUT. That ordering is the point: invert it and every idle poll aborts the read before Redis has replied, turning a correct empty result into an exception. Callers that inject their own Redis are unaffected; modelslab-frontend-v2 already sets a 5s connect and read timeout on the client it passes in. Tests: 3 unit tests, no Redis required. Mutation-tested — replacing the constant in blPop and restoring READ_TIMEOUT to 0 each turn them red. Suite goes 130 -> 133 with the same 13 pre-existing Integration failures and no new phpstan errors (36 before, 36 after).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to modelq#16. Same defect, different route.
Why this looked safe and wasn't
The worker already passed a timeout:
That
1is the server's timeout. It tells Redis how long to hold the pop before answering "nothing". It says nothing about how long this process waits for that answer to arrive.phpredis defaults its read timeout to
0— wait forever — andconnect()was called without one:So when the connection is silently dropped, the reply can never arrive, the read never returns, and the worker parks permanently on a socket the server has already discarded. Identical outcome to the Python side, where the same thing happened via an unbounded
BLPOP.On 2026-08-15 this took out seven
server_types within fifteen minutes;sfx_serversat dead for two days before anyone noticed.Changes
connect()passesCONNECT_TIMEOUTandREAD_TIMEOUT.OPT_TCP_KEEPALIVEis set so the kernel probes an idle peer, guarded bydefined()since it needs phpredis 5+.BLPOP_TIMEOUT, deliberately held belowREAD_TIMEOUT. That ordering is the whole point — invert it and every idle poll aborts the read before Redis has replied, turning a correct empty result into an exception on every loop.Blast radius
Only affects the path where ModelQ builds its own client. Callers injecting their own Redis are untouched —
modelslab-frontend-v2already sets a 5s connect and read timeout on the client it passes in, which is why the frontend never exhibited this.Testing
3 unit tests, no Redis required. Mutation-tested:
self::BLPOP_TIMEOUTwith a literal inblPopREAD_TIMEOUT = 0(phpredis default)Suite goes 130 → 133. The same 13 pre-existing Integration failures remain (they need a live Redis);
commagainst the recorded baseline confirms none added. phpstan level 6: 36 errors before, 36 after, zero introduced.Not a duplicate of #2
#2 removes the redundant
prune_old_task_resultsSCAN storm — a throughput fix. This is a liveness fix. They touch different code and are independently mergeable in either order.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is enabled.