Repository navigation
Raise the bootstrappers' inbound connection limit to 64000 - #7
Merged
Merged
Conversation
All four bootstrap instances had used their 16000 inbound connection slots (System.ConnsInbound), so kubo's resource manager refused new connections while the ports still accepted TCP. Fresh handshakes from outside succeeded about 1 time in 5, which the ProbeLab probes saw as flapping. Kubo computes System.ConnsInbound as 2 x ConnMgr.HighWater, so raise HighWater to 32000 and LowWater to 16000, giving 64000. Raise MaxFileDescriptors to 131072 for the roughly 60000 descriptors that many connections need. Swarm.ResourceMgr.Limits is not an option: kubo removed it in 0.19 and refuses to start with it set. Rolled out one box at a time on 2026-10-16. All four instances show a 64000 limit and complete 5 of 5 handshakes from outside. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
All four bootstrap instances (sg1, am6, ny5, sv15) had used their 16000 inbound connection slots (kubo
System.ConnsInbound). The resource manager then refused new connections while the ports still accepted TCP, so fresh handshakes from outside succeeded about 1 time in 5. ProbeLab saw this as flapping, with each peer alternating OK and FAIL.How it showed up: TCP connect succeeded, then the security handshake ended in
EOF, and QUIC gotCONNECTION_REFUSED. Pods, listeners, ufw, CPU, memory and file descriptors were all fine, and the pods had 0 restarts.ipfs swarm resourcesshowedsystem ConnsInbound 16000 / 16000 (100%)on all four instances. It took about 15 days to fill. The holders were spread across thousands of peers, with no single heavy client.Change
roles/bootstrap/files/kubo-config.json:Swarm.ConnMgr.HighWaterSwarm.ConnMgr.LowWaterSwarm.ResourceMgr.MaxFileDescriptorsKubo computes
System.ConnsInboundas 2 xHighWater, so the limit goes from 16000 to 64000. The old config used about 15000 file descriptors at 16000 connections, so roughly 60000 at the new limit, which fits the new 131072.Swarm.ResourceMgr.Limitsis not an option. Kubo removed it in 0.19 and the init container refuses a config that sets it. An earlier rollout of mine tried it and took sing-1 (sg1) down for about 15 minutes before it was restored.Rollout and verification
Already applied by hand with
ansible-playbook bootstrap.yml -l <box> -t bootstrap, one box at a time (sing-1, lith-1, chic-1), eachfailed=0. After the rollout:system ConnsInbound 64000(3-5% used) andFD 131072./dnsaddr/*.bootstrap.libp2p.ioentries.Follow up
ipfs swarm resourcesin about a week to see whether usage plateaus well under 64000.HighWaterof 8000 did not keep connections down to that mark.bootstrap.yml --checkfails on a check-mode-only conditional atroles/bootstrap/tasks/instance.yml:222(string used as a boolean). It is unrelated to this change.🤖 Generated with Claude Code