Repository navigation
Open
Description
User Story
I ran a mini scale test of OpenShell on Kubernetes with a shared supervisor pool, at up to 450 sandboxes. The supervisor's thread, file-descriptor, and socket counts grew in a straight line with the number of sandboxes, and memory grew until the supervisor ran out of memory. I need one shared supervisor to host many sandboxes without its cost growing this fast.
Problem Statement
Each sandbox in a shared supervisor gets its own:
- OS thread, plus a private Tokio runtime with 2 worker threads (
INSTANCE_WORKER_THREADS = 2), with extra blocking threads added on demand - TLS connections to the gateway, about 7 sockets per sandbox
What we measured:
| Sandboxes | Threads | File descriptors | Sockets |
|---|---|---|---|
| 256 (idle) | ~1,040 | 2,574 | 1,798 |
| 256 (under load) | ~1,182 |
No OS limits were hit. Idle exec sessions are fine: 450 held sessions added no threads, so #5217 works.
Why This Is a Problem
- Cost grows with every sandbox. Hosting more sandboxes on one supervisor is the point of the shared model, but each one adds about 4 threads and 7 sockets whether it's busy or idle.
- Memory. The supervisor is a glibc build with the default allocator. glibc gives busy threads their own malloc arenas, up to 8 per CPU core. With about 1,000 threads, memory spreads across many arenas and is slow to return to the OS, which likely added to the memory growth behind the out-of-memory kill.
- Gateway load. About 7 TLS connections per sandbox means a large shared supervisor holds thousands of connections to the gateway.
Impact / Why This Matters
- The number of sandboxes one supervisor can host is limited by threads and memory, not by real work.
- Operators have to give supervisors much more memory, or run more of them, than the workload needs.
Acceptance Criteria
- Threads per idle sandbox in a shared supervisor drop well below 4. For example, sandboxes could share worker threads while keeping their own identity and logs.
- Gateway connections per sandbox drop, for example by sharing a connection across sandboxes.
- Supervisor memory at 256 and 450 sandboxes is measured before and after, and stays flat at a steady sandbox count.
Reproduction Steps
- Deploy OpenShell on Kubernetes with a shared supervisor pool.
- Create 256 sandboxes.
- On the supervisor pod, count threads, file descriptors, and sockets:
ls /proc/<pid>/task | wc -l,ls /proc/<pid>/fd | wc -l, andls -l /proc/<pid>/fd | grep -c socket. - Repeat at different sandbox counts. All three grow roughly linearly.
Environment
- OpenShell: mini scale test build; code paths confirmed on
mainat 624acee - Deployment: Kubernetes (Helm), shared supervisor pool, glibc supervisor image
Metadata
Metadata
Assignees
Labels
No labels
Activity