Skip to content

fix: bound the Redis read so a dead socket cannot park a worker forever - #3

Merged
adhikjoshi merged 1 commit into
mainfrom
fix/redis-read-timeout-and-keepalive
Aug 15, 2026
Merged

fix: bound the Redis read so a dead socket cannot park a worker forever#3
adhikjoshi merged 1 commit into
mainfrom
fix/redis-read-timeout-and-keepalive

Conversation

@adhikjoshi

@adhikjoshi adhikjoshi commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Companion to modelq#16. Same defect, different route.

Why this looked safe and wasn't

The worker already passed a timeout:

$taskData = $this->redis->blPop(['ml_tasks'], 1);

That 1 is the server's timeout. It tells Redis how long to hold the pop before answering "nothing". It says nothing about how long this process waits for that answer to arrive.

phpredis defaults its read timeout to 0 — wait forever — and connect() was called without one:

$this->redis->connect($host, $port);   // no connect timeout, no read timeout

So when the connection is silently dropped, the reply can never arrive, the read never returns, and the worker parks permanently on a socket the server has already discarded. Identical outcome to the Python side, where the same thing happened via an unbounded BLPOP.

On 2026-08-15 this took out seven server_types within fifteen minutes; sfx_server sat dead for two days before anyone noticed.

Changes

  • connect() passes CONNECT_TIMEOUT and READ_TIMEOUT.
  • OPT_TCP_KEEPALIVE is set so the kernel probes an idle peer, guarded by defined() since it needs phpredis 5+.
  • The blocking pop uses BLPOP_TIMEOUT, deliberately held below READ_TIMEOUT. That ordering is the whole point — invert it and every idle poll aborts the read before Redis has replied, turning a correct empty result into an exception on every loop.

Blast radius

Only affects the path where ModelQ builds its own client. Callers injecting their own Redis are untouched — modelslab-frontend-v2 already sets a 5s connect and read timeout on the client it passes in, which is why the frontend never exhibited this.

Testing

3 unit tests, no Redis required. Mutation-tested:

mutation result
replace self::BLPOP_TIMEOUT with a literal in blPop 1 test red
restore READ_TIMEOUT = 0 (phpredis default) 2 tests red

Suite goes 130 → 133. The same 13 pre-existing Integration failures remain (they need a live Redis); comm against the recorded baseline confirms none added. phpstan level 6: 36 errors before, 36 after, zero introduced.

Not a duplicate of #2

#2 removes the redundant prune_old_task_results SCAN storm — a throughput fix. This is a liveness fix. They touch different code and are independently mergeable in either order.


View with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.

The PHP client carries the same defect that stalled the Python fleet on
2026-08-15, reached by a different route.

`blPop(['ml_tasks'], 1)` already passed a timeout, so this looked safe. That
timeout is the *server's* — it tells Redis how long to hold the pop before
answering "nothing". It says nothing about how long this process will wait for
that answer to arrive. phpredis defaults its read timeout to 0, meaning wait
forever, and `connect()` was called without one. So when a connection is
silently dropped the reply can never arrive, the read never returns, and the
worker parks permanently on a socket the server has already discarded.

- `connect()` now passes CONNECT_TIMEOUT and READ_TIMEOUT.
- OPT_TCP_KEEPALIVE is set so the kernel probes an idle peer, guarded by
  defined() since it needs phpredis 5+.
- The blocking pop uses BLPOP_TIMEOUT, which is held below READ_TIMEOUT. That
  ordering is the point: invert it and every idle poll aborts the read before
  Redis has replied, turning a correct empty result into an exception.

Callers that inject their own Redis are unaffected; modelslab-frontend-v2
already sets a 5s connect and read timeout on the client it passes in.

Tests: 3 unit tests, no Redis required. Mutation-tested — replacing the
constant in blPop and restoring READ_TIMEOUT to 0 each turn them red. Suite
goes 130 -> 133 with the same 13 pre-existing Integration failures and no new
phpstan errors (36 before, 36 after).
@adhikjoshi
adhikjoshi merged commit be06cc5 into main Aug 15, 2026
1 check passed
@adhikjoshi
adhikjoshi deleted the fix/redis-read-timeout-and-keepalive branch August 15, 2026 07:32
adhikjoshi added a commit that referenced this pull request Aug 15, 2026
Swept in by mistake — the .gitignore rule for it landed in #2, after the
branch for #3 had already been cut off main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant