test(reliability): add reconnection-storm (thundering herd) test - #85
test(reliability): add reconnection-storm (thundering herd) test#85Harxhit wants to merge 1 commit into
Conversation
socket.io-client test that connects many auto-reconnecting clients, then triggers a simultaneous mass drop (forcedrop) or a manual pod kill (killpod) and measures recovery-time distribution + success rate — quantifies recovery, which the cold spike test never did. - load-test/reconnection-storm.js — forcedrop|killpod modes; reconnect+rejoin timing - load-test/results/RECONNECTION-STORM.md — modes, interpretation, backoff/jitter notes Authored by Claude (Anthropic) via Claude Code. Not executed against a live backend — results are placeholders, not fabricated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
3 issues found across 2 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="load-test/reconnection-storm.js">
<violation number="1" location="load-test/reconnection-storm.js:217">
P2: In `killpod` mode, unrelated disconnects after the prompt are scored as victims, so recovery percentiles and success rate do not represent the killed pod. Track the affected pod/client set or otherwise distinguish the kill-induced disconnects before tallying.</violation>
<violation number="2" location="load-test/reconnection-storm.js:237">
P2: In `forcedrop` mode, connected clients whose baseline `joinRoom` failed or is still pending are counted as recovered/stalled storm members. Require `baselineJoined` when selecting clients to drop.</violation>
<violation number="3" location="load-test/reconnection-storm.js:307">
P2: In `killpod` mode, operator delay is included in every recovery time and can consume the recovery window before the pod is killed. Wait for an explicit post-kill confirmation or set `triggerAt` when the actual kill is initiated.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
|
||
| // ---- trigger the storm ---- | ||
| stormTriggered = true; | ||
| triggerAt = performance.now(); |
There was a problem hiding this comment.
P2: In killpod mode, operator delay is included in every recovery time and can consume the recovery window before the pod is killed. Wait for an explicit post-kill confirmation or set triggerAt when the actual kill is initiated.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At load-test/reconnection-storm.js, line 307:
<comment>In `killpod` mode, operator delay is included in every recovery time and can consume the recovery window before the pod is killed. Wait for an explicit post-kill confirmation or set `triggerAt` when the actual kill is initiated.</comment>
<file context>
@@ -0,0 +1,395 @@
+
+ // ---- trigger the storm ----
+ stormTriggered = true;
+ triggerAt = performance.now();
+
+ if (MODE === "forcedrop") {
</file context>
| }); | ||
|
|
||
| socket.on("disconnect", () => { | ||
| if (stormTriggered) client.droppedAfterTrigger = true; |
There was a problem hiding this comment.
P2: In killpod mode, unrelated disconnects after the prompt are scored as victims, so recovery percentiles and success rate do not represent the killed pod. Track the affected pod/client set or otherwise distinguish the kill-induced disconnects before tallying.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At load-test/reconnection-storm.js, line 217:
<comment>In `killpod` mode, unrelated disconnects after the prompt are scored as victims, so recovery percentiles and success rate do not represent the killed pod. Track the affected pod/client set or otherwise distinguish the kill-induced disconnects before tallying.</comment>
<file context>
@@ -0,0 +1,395 @@
+ });
+
+ socket.on("disconnect", () => {
+ if (stormTriggered) client.droppedAfterTrigger = true;
+ });
+
</file context>
| for (const client of clients) { | ||
| // Only drop clients that were actually in steady state; a client that never | ||
| // connected was not part of the herd and should not count against recovery. | ||
| if (!client.socket.connected) continue; |
There was a problem hiding this comment.
P2: In forcedrop mode, connected clients whose baseline joinRoom failed or is still pending are counted as recovered/stalled storm members. Require baselineJoined when selecting clients to drop.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At load-test/reconnection-storm.js, line 237:
<comment>In `forcedrop` mode, connected clients whose baseline `joinRoom` failed or is still pending are counted as recovered/stalled storm members. Require `baselineJoined` when selecting clients to drop.</comment>
<file context>
@@ -0,0 +1,395 @@
+ for (const client of clients) {
+ // Only drop clients that were actually in steady state; a client that never
+ // connected was not part of the herd and should not count against recovery.
+ if (!client.socket.connected) continue;
+
+ client.droppedAfterTrigger = true;
</file context>
| if (!client.socket.connected) continue; | |
| if (!client.socket.connected || !client.baselineJoined) continue; |
1 participant
socket.io-client test that connects many auto-reconnecting clients, then triggers a simultaneous mass drop (forcedrop) or a manual pod kill (killpod) and measures recovery-time distribution + success rate — quantifies recovery, which the cold spike test never did.
Authored by Claude (Anthropic) via Claude Code. Not executed against a live backend — results are placeholders, not fabricated.
Summary by cubic
Adds a reconnection-storm load test that measures herd recovery after a simultaneous client drop, which the existing cold-connect spike test never covered.
socket.io-clientclients with auto-reconnection enabled, drops them all at once (forcedrop) or when a signaling pod is killed (killpod), and reports reconnect+re-join percentiles and recovery success within a configurable window.Written for commit d3b16c9. Summary will update on new commits.