Repository navigation
fix(substrate): keep Substrate page tokens inside the watched namespaces - #3262
Open
danielle-mustillo wants to merge 12 commits into
Open
danielle-mustillo wants to merge 12 commits into
danielle-mustillo wants to merge 12 commits into
Conversation
…entory Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
A kagent that watches only its own namespace listed every tenant's actors and ActorTemplates from a shared Substrate, and served a foreign atespace or namespace when a caller named one. Resolve one scope from the watched namespaces for every Substrate read. An empty request means the watched namespaces, and a namespace or atespace outside them is refused with PermissionDenied. A cluster-wide install still sees everything. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
…refusal wording A controller watching tenant-a and tenant-c must never page tenant-b actors to the caller, must ask ate-api only for its own atespaces when it builds the summary, and must hide a worker that reports no namespace. The refusal message must say the namespace is not watched, since the gRPC layer sends only the message. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
…workers The summary for a controller watching several namespaces asked ate-api for every atespace and filtered the rows itself, so ate-api served every tenant's actors and templates to it. It now walks actors and drains templates once per atespace in scope. ListSubstrateActors still reads one upstream page across all atespaces and filters it, because one upstream token cannot span several requests. A worker that reports no namespace now belongs only to a cluster-wide scope, and the refusal message names the namespace that is not watched, since the gRPC layer sends only the message. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
A controller watching tenant-a and tenant-c must page through its own actors in watched-atespace order, scope every ate-api read to one watched atespace, never hand out an empty page with a token, and never return a token that names a tenant-b actor. A token it did not issue is InvalidArgument, without echoing the token and without an ate-api call. The cluster-wide pass-through test now runs cluster-wide, and the assertion that a filtered-out first page still leads on is dropped: that page is the leak. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
With two or more watched namespaces, ListSubstrateActors asked ate-api for every atespace and passed its page token through. ate-api tokens are readable base64 JSON holding the last row's atespace and name, so a token could name another tenant's actor, and a page filtered to nothing still carried a token. A controller that watches some namespaces now reads each watched atespace with its own scoped list, in sorted order, and returns its own token: the atespace to read and ate-api's token inside it. A token it did not issue, or one longer than 1 KiB, is InvalidArgument and never reaches ate-api. NewService sorts and dedupes the watched list, since tokens resume in its order. A cluster-wide controller still passes ate-api's page and token through. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
… workers A controller watching tenant-a and tenant-c must page its own workers once each, in worker name order whatever order ate-api sends, never hand out an empty page with a token, and never send its own token to ate-api. An ate-api worker list error must reach the caller as its gRPC code only, since the text can name another tenant's worker. The tenant isolation fixtures now name their workers, as ate-api does. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
A controller that watches some namespaces passed ate-api's worker page token through. The token holds the last worker's name, so it could name another tenant's worker, and a page filtered to nothing still carried a token. Such a controller now drains ate-api's workers, keeps the ones in scope, and pages them in worker name order. Its token is the last worker's name: a cursor only, never sent to ate-api. ate-api worker list errors reach the caller as the gRPC code only, for the list and the summary, since the text can name another tenant's worker; the full error is logged. A cluster-wide controller still passes ate-api's page, token and error through. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
…spaces The proto comments, the generated Go and TypeScript, the UI page type and the Playwright deferral notes said an empty atespace lists all atespaces and that every page passes Substrate's token through. A controller that watches some namespaces lists only its watched atespaces, issues its own tokens, and never returns an empty page with a token. The atespace filter placeholder now reads "All watched atespaces". Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
Drop comments that restate the code. The proto now says that an actor page can be empty when another page exists, and gives the worker order for each mode. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
github-actions
Bot
added
bug
… tokens The fake ate-api now hands out base64 JSON tokens naming the page's last row, as ate-api does, so assertOwnToken can see a token that would name another tenant's row. New tests: scoped worker page tokens must be refused unless kagent issued them, a scoped summary hides ate-api worker error text, forged tokens never reach or return an unwatched namespace, and the longest names fit the page token. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
A controller that watches some namespaces used any worker page_token raw as the name to resume after, so an ate-api token or junk silently returned a wrong or short page. Worker tokens now share the actor tokens' shape (base64url of a watched namespace, a slash, then the cursor) and the same 1 KiB cap, and anything else is InvalidArgument. The length check now runs before decoding. Cluster-wide installs still pass ate-api's tokens through. Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
github-actions
Bot
added
bug
danielle-mustillo
marked this pull request as ready for review
October 19, 2026 06:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3261. Until #3261 merges, this diff also shows its four commits. Review the last eight commits.
Why
After #3261, a controller that watches some namespaces still passes ate-api's page tokens to the caller. ate-api tokens are readable base64 JSON that hold the last row's key, so a token can name another tenant's actor (
[atespace, name]) or worker. With two or more watched namespaces, actor pages can also be empty and still have a next-page token.What changed
base64url("atespace/inner"). A bad or foreign token getsInvalidArgument.base64url("namespace/name")of the last worker, and it never goes to ate-api. A bad or foreign token getsInvalidArgument. ate-api worker errors return only the gRPC code.Scope
ListSubstrateActors,ListSubstrateWorkersand the worker count inGetSubstrateSummary. A cluster-wide install passes ate-api's pages and tokens through, as before.Tradeoffs
Blast Radius
Only installs that watch some namespaces change. An actor or worker page token from an older controller gets
InvalidArgument. Reload the UI page, or send the request without the token.Verification
ListSubstrateWorkers {page:{limit:1}}: before, page 1 was empty, and its token decoded to a tenant-b worker UID. After, each page holds one tenant-a worker, and the token is a tenant-a worker name. A foreign actor token getsInvalidArgument.go test ./core/internal/service/...passes, andmake lintreports 0 issues.