Skip to content

[docs] add RFCs for per-client admission independent of the substrate - #1072

Open
the-gigi-pplx wants to merge 7 commits into
mainfrom
gigi/pla-6997-rfc-substrate-independent-per-client-admission
Open

the-gigi-pplx wants to merge 7 commits into
mainfrom
gigi/pla-6997-rfc-substrate-independent-per-client-admission

Conversation

@the-gigi-pplx

@the-gigi-pplx the-gigi-pplx commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds two RFCs that define per-client activation admission so it doesn't depend on Kubernetes pods. Together they review the open per-client work (PR #1058, PR #1061, PR #1066 and PR #1068) and say what to keep, rework or drop.

docs/space/rfcs/2026-10-28-substrate-independent-client-admission.md covers what ships. It opens with the terms it relies on (activation, admission, charge, key, record, pending set and others), and states each mechanism's rules and invariants, leaving call-site detail to the implementation pull requests:

  • Who's charged: every activation carries an ActivationClient: an ECW client id, inherit (act for the sandbox's recorded client, as an implicit data-plane wake does) or native.
  • Ids are stored, keys are resolved at use: records and pod labels store the client id the caller sent. One resolver turns it into a bounded key against the current key set for the concurrency limiter, metrics and admission, so allowlist and flag changes take effect everywhere at once. PR [api] tag limiter decisions and reconnect metrics with the bounded client id #1058 and PR [api] record the activating client and label sandbox pods with it #1068 each define their own resolver today.
  • Attribution record: a nullable sandboxes.activating_client_id holding the client id whose activation last created the sandbox's runtime, written with the same value as the pod label. On Kubernetes it follows the active execution row's tracked pod, and on Direct it's written in the start and restore receipts.
  • Immutable pod label: space.pplx.net/client-id is set at pod creation, and a pod counts against the client whose activation created it.
  • Live cap: Substrate::live_sandboxes_for_client counts a client's other live sandboxes from the sandbox-pod reflector, through a per-id index updated from watch events. A shared pending set in Redis covers admissions the reflectors haven't seen yet. The RFC explains why that pair is the right arbiter and compares the alternatives. The cap runs before a create branches, and in ensure_pod_exists before every pod it creates, so a running sandbox is never refused. Direct answers Unknown(Unsupported) and admission admits.
  • Existing limiters: PR api: refuse a cold create with a 503 when every eligible node's network slots are held #1061's slot check, enforcement of the existing per-client concurrency limiter, and a concurrency slot for ECW resume.
  • Precision: an error of about one sandbox in either direction is accepted if it's rare, temporary and self-correcting; systematic or growing drift isn't.
  • Phasing: explains with Little's law why the concurrency limiter bounds the activation rate only on the paths it gates and only while requests are slow, and why the per-client rate can wait for evidence.

docs/space/rfcs/2026-10-29-per-client-activation-rate.md specifies the deferred per-client activation rate: a per-client token bucket consumed atomically with the cluster bucket, charged on warm-pool hits too, with token holds that a refused attempt cancels without over-crediting the bucket, its composition with the first RFC, and its open issues. It's built only if, with the first RFC enforced, a client is seen draining the cluster bucket or the warm pool through paths the concurrency limiter doesn't gate.

Verdicts:

Validation

Documentation only. I checked the RFCs' statements about current behavior against main and against the open PRs it reviews. No workspace code checks apply.

Severity and risk

Severity: does not apply; this is a design proposal.

Risk: none at runtime. Merging them records the design that the open per-client pull requests are reworked to follow. Adopting it changes one existing behavior, which the RFC calls out: an allowlisted client id that isn't a valid Kubernetes label value stops getting its own concurrency bucket.

AI assistance: Claude Code drafted both RFCs and this description.

@the-gigi-pplx
the-gigi-pplx force-pushed the gigi/pla-6997-rfc-substrate-independent-per-client-admission branch 19 times, most recently from f31ee03 to 397f531 Compare October 29, 2026 18:18
@the-gigi-pplx
the-gigi-pplx force-pushed the gigi/pla-6997-rfc-substrate-independent-per-client-admission branch 3 times, most recently from a091816 to eaf4e8a Compare October 29, 2026 21:25
@the-gigi-pplx the-gigi-pplx changed the title [docs] add an RFC for per-client admission independent of the substrate [docs] add RFCs for per-client admission independent of the substrate Oct 29, 2026
@the-gigi-pplx
the-gigi-pplx force-pushed the gigi/pla-6997-rfc-substrate-independent-per-client-admission branch 3 times, most recently from afe2e79 to bdd5071 Compare October 30, 2026 04:02
@the-gigi-pplx
the-gigi-pplx force-pushed the gigi/pla-6997-rfc-substrate-independent-per-client-admission branch from bdd5071 to c5f1d8f Compare October 30, 2026 06:49
@the-gigi-pplx
the-gigi-pplx marked this pull request as ready for review October 30, 2026 06:49
@the-gigi-pplx

Copy link
Copy Markdown
Collaborator Author

Updated since the review request, in 8eb115d (docs only, no decision changed beyond these):

  • PR [db] add the activating client id column to sandboxes #1066 is described as it now is: sandboxes.activating_client_id, migration 20261030070237_sandbox_activating_client_id. That's in the intro, "Why the in-flight attribution design changes", "Alternatives considered" and the keep/rework row, which keeps its reasoning.
  • The column gets a NOT VALID format CHECK (NULL, the reserved ids, or a Kubernetes label value of at most 63 characters). NOT VALID means the migration doesn't scan sandboxes. That's in Decision 3's "Shape" and Rollout step 1.
  • Rollout step 2: a NULL record lasts until an activation with a client of its own (Client or Native) sets it. An inheriting activation keeps NULL, per Decision 1.

@the-gigi-pplx

Copy link
Copy Markdown
Collaborator Author

Follow-up in 29d3825: "Decision 6 / Order" now scopes PR #1061's slot check to cold creates, as that PR does. Other activations can reuse a pod or a pinned node, so a free-slot count would misjudge them.

@the-gigi-pplx

Copy link
Copy Markdown
Collaborator Author

Follow-up in 0962e15: Decision 4 now invalidates the slot on any failed adoption label patch instead of returning it, because a returned slot goes to the front of the claim queue and a repeating refusal would hit every later pool claim.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants