Skip to content

perf(supervisor): shared supervisor uses about 4 threads and 7 gateway sockets per sandbox #5660

Open

Description

@FrostGod

User Story

I ran a mini scale test of OpenShell on Kubernetes with a shared supervisor pool, at up to 450 sandboxes. The supervisor's thread, file-descriptor, and socket counts grew in a straight line with the number of sandboxes, and memory grew until the supervisor ran out of memory. I need one shared supervisor to host many sandboxes without its cost growing this fast.

Problem Statement

Each sandbox in a shared supervisor gets its own:

  • OS thread, plus a private Tokio runtime with 2 worker threads (INSTANCE_WORKER_THREADS = 2), with extra blocking threads added on demand
  • TLS connections to the gateway, about 7 sockets per sandbox

What we measured:

Sandboxes Threads File descriptors Sockets
256 (idle) ~1,040 2,574 1,798
256 (under load) ~1,182

No OS limits were hit. Idle exec sessions are fine: 450 held sessions added no threads, so #5217 works.

Why This Is a Problem

  • Cost grows with every sandbox. Hosting more sandboxes on one supervisor is the point of the shared model, but each one adds about 4 threads and 7 sockets whether it's busy or idle.
  • Memory. The supervisor is a glibc build with the default allocator. glibc gives busy threads their own malloc arenas, up to 8 per CPU core. With about 1,000 threads, memory spreads across many arenas and is slow to return to the OS, which likely added to the memory growth behind the out-of-memory kill.
  • Gateway load. About 7 TLS connections per sandbox means a large shared supervisor holds thousands of connections to the gateway.

Impact / Why This Matters

  • The number of sandboxes one supervisor can host is limited by threads and memory, not by real work.
  • Operators have to give supervisors much more memory, or run more of them, than the workload needs.

Acceptance Criteria

  • Threads per idle sandbox in a shared supervisor drop well below 4. For example, sandboxes could share worker threads while keeping their own identity and logs.
  • Gateway connections per sandbox drop, for example by sharing a connection across sandboxes.
  • Supervisor memory at 256 and 450 sandboxes is measured before and after, and stays flat at a steady sandbox count.

Reproduction Steps

  1. Deploy OpenShell on Kubernetes with a shared supervisor pool.
  2. Create 256 sandboxes.
  3. On the supervisor pod, count threads, file descriptors, and sockets: ls /proc/<pid>/task | wc -l, ls /proc/<pid>/fd | wc -l, and ls -l /proc/<pid>/fd | grep -c socket.
  4. Repeat at different sandbox counts. All three grow roughly linearly.

Environment

  • OpenShell: mini scale test build; code paths confirmed on main at 624acee
  • Deployment: Kubernetes (Helm), shared supervisor pool, glibc supervisor image

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions