fix(redis-worker): pop ready worker queue messages without opening a new connection

Batch items are processed faster when the batch queue has a backlog.

Worker queue consumers used to open a new Redis connection for every message they popped, even when messages were already waiting. They now take a ready message on the existing connection and only open a dedicated blocking connection when the queue is empty, which removes several round trips per batch item and most of the Redis CPU spent on connection setup.

Mono-RevId: 7a492d821fc4bdcc212cf08bc1ddd67fa6a8517a

Author: 0ski

packages/redis-worker/src/fair-queue/workerQueue.ts

@@ -97,6 +97,17 @@ export class WorkerQueueManager {
   ): Promise<string | null> {
     const workerQueueKey = this.keys.workerQueueKey(workerQueueId);
 
+    if (signal?.aborted) {
+      return null;
+    }
+
+    // Fast path: under load the queue is rarely empty, so pop on the shared connection
+    // and only pay for a dedicated blocking connection when there is nothing to take
+    const readyMessageKey = await this.redis.lpop(workerQueueKey);
+    if (readyMessageKey) {
+      return readyMessageKey;
+    }
+
     // Create a separate client for blocking operation
     // This is required because BLPOP blocks the connection
     const blockingClient = this.redis.duplicate();