Skip to content

fix(substrate): scope actors and templates to the watched namespaces - #3261

Open
danielle-mustillo wants to merge 9 commits into
kagent-dev:mainfrom
danielle-mustillo:danielle-mustillo/substrate-atespace-scope-indep
Open

danielle-mustillo wants to merge 9 commits into
kagent-dev:mainfrom
danielle-mustillo:danielle-mustillo/substrate-atespace-scope-indep

Conversation

@danielle-mustillo

@danielle-mustillo danielle-mustillo commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Why

When a kagent install watches only its own namespace and shares a Substrate with other installs, the Substrate API listed every install's actors and actor templates. It also served another tenant's atespace or namespace when a caller named it. That second path leaked workers too. Before #2771, actors and templates were filtered by watched namespace. The paging rewrite in #2771 dropped that filter for actors and templates and kept it only for workers.

What changed

  • One scope, built from the watched namespaces, now applies to every Substrate read: actors, templates, workers, and worker pools.
  • An empty request means the watched namespaces. A namespace or atespace outside them returns PermissionDenied with a message that says so.
  • The summary asks ate-api for each watched atespace in turn, so it never reads other tenants' rows. A worker with no namespace is shown only on a cluster-wide install.

Scope

  • Covers ListSubstrateActors, ListSubstrateWorkers, and GetSubstrateSummary. No API change.
  • atespace proto comments now say an empty value means the watched namespaces. A namespace listed twice in watchNamespaces is counted once.
  • Not included: reading ahead past empty actor pages when two or more namespaces are watched, and updating the "All atespaces" wording in the UI.

Tradeoffs

  • ListSubstrateActors must pass ate-api's page token through unchanged. So with two or more watched namespaces it reads all atespaces and filters each page. A page can be empty and still have a next-page token, as the worker list already allows. A composite token would avoid this, but it changes the token contract.

Blast Radius

Only installs that watch some namespaces change. They see less, and requests outside their scope are refused. A cluster-wide install behaves as before.

Verification

  • Two-tenant kind cluster: one shared Substrate and two kagent installs, each watching its own namespace, with one session per tenant. Only the controller image changed between runs. As tenant-a:
    • ListSubstrateActors {} returned ["tenant-a","tenant-b"] before and ["tenant-a"] after.
    • GetSubstrateSummary {atespace: tenant-b} returned tenant-b's templates before and PermissionDenied after.
    • ListSubstrateWorkers {namespace: tenant-b} returned tenant-b's worker before and PermissionDenied after.
    • A cluster-wide install still sees both tenants.
  • Unit tests: each test commit fails on the code before it. Deleting the filter in ListSubstrateActors makes the multi-namespace test fail.
  • go test ./core/internal/service/... passes, and make lint reports 0 issues.

…entory

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
A kagent that watches only its own namespace listed every tenant's actors
and ActorTemplates from a shared Substrate, and served a foreign atespace
or namespace when a caller named one.

Resolve one scope from the watched namespaces for every Substrate read.
An empty request means the watched namespaces, and a namespace or atespace
outside them is refused with PermissionDenied. A cluster-wide install
still sees everything.

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
@danielle-mustillo
danielle-mustillo requested a review from a team as a code owner October 19, 2026 00:39
@github-actions github-actions Bot added the bug Something isn't working label Oct 19, 2026
…refusal wording

A controller watching tenant-a and tenant-c must never page tenant-b actors
to the caller, must ask ate-api only for its own atespaces when it builds
the summary, and must hide a worker that reports no namespace. The refusal
message must say the namespace is not watched, since the gRPC layer sends
only the message.

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
…workers

The summary for a controller watching several namespaces asked ate-api for
every atespace and filtered the rows itself, so ate-api served every
tenant's actors and templates to it. It now walks actors and drains
templates once per atespace in scope. ListSubstrateActors still reads one
upstream page across all atespaces and filters it, because one upstream
token cannot span several requests.

A worker that reports no namespace now belongs only to a cluster-wide
scope, and the refusal message names the namespace that is not watched,
since the gRPC layer sends only the message.

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 19, 2026
@danielle-mustillo
danielle-mustillo marked this pull request as draft October 19, 2026 02:01
@danielle-mustillo
danielle-mustillo marked this pull request as ready for review October 19, 2026 05:25

@EItanya EItanya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI-generated review. I looked through them to make sure they weren't crazy and they all seem worth looking at.

Thanks for the fix! I found one correctness issue and two follow-ups:

  1. Can we deduplicate observedNamespaces once when constructing the service? watch_namespaces does not require unique values, and resolveSubstrateScope preserves duplicates. walkActors and substrateActorTemplates then query each entry, so ["tenant-a", "tenant-a"] reports every actor and template twice. I reproduced this with one actor and one template: the summary returned ActorCount == 2 and two template entries. A regression test should include non-adjacent duplicates as well.

  2. The public protobuf comments for GetSubstrateSummaryRequest.atespace and ListSubstrateActorsRequest.atespace still say that an empty value lists all atespaces. With this change, it means the watched namespaces unless the controller is cluster-wide, and an explicitly unwatched atespace returns PermissionDenied. Can we update the source comments and regenerate so clients don't interpret a scoped result as cluster-wide inventory?

  3. TestSubstrateTenantIsolation uses an empty Kubernetes client, so it doesn't exercise worker-pool isolation even though the PR applies the scope to every Substrate read. Can we seed in-scope and foreign pools and assert the exact namespace/name pairs for scoped, multi-namespace, and cluster-wide summaries?

The isolation test built its fake cluster with no WorkerPools, so it never
checked that the summary lists only pools in the watched namespaces. It
also never listed a namespace twice in watchNamespaces, which counted
tenant-a's actors, templates and pools twice.

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
watchNamespaces allows duplicates, and every Substrate read asks ate-api
and Kubernetes once per watched namespace, so a duplicate entry doubled
the summary's actors, templates and worker pools. Drop duplicates once
when the service is built, keeping the configured order.

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
GetSubstrateSummaryRequest.atespace and ListSubstrateActorsRequest.atespace
still said an empty value lists all atespaces. It now means the watched
namespaces unless the controller is cluster-wide, and an unwatched
atespace returns PermissionDenied.

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
@danielle-mustillo

Copy link
Copy Markdown
Contributor Author

Thanks @EItanya. All three were real. Fixed in 3 commits:

  1. Duplicate watch namespaces. e70146e adds a [tenant-a, tenant-c, tenant-a] case. It failed on the old code with 3 actors, agent-a twice and tenant-a/pool-a twice. 9b19e56 drops duplicates once in NewService and keeps the configured order. I left the config itself accepting duplicates, so an existing install with a repeated entry still starts.
  2. Proto comments. 4774891 updates both atespace comments and regenerates the Go and TS output.
  3. Worker pools. e70146e seeds pools in tenant-a, tenant-b and tenant-c. It asserts the exact namespace/name pairs for the single-namespace, two-namespace, duplicate and cluster-wide summaries. To check that the test can fail, I removed the namespace filter in listWorkerPools. Three subtests failed.

@github-actions github-actions Bot added bug Something isn't working and removed bug Something isn't working labels Oct 19, 2026
Comment thread go/core/internal/service/system/service.go Outdated
Sorting before Compact removes duplicates that are not adjacent.
ListNamespaces already sorts its output, so the order change is not
visible to callers.

Verified with go test ./core/internal/service/system, including the
case that watches [tenant-a, tenant-c, tenant-a].

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>
@danielle-mustillo danielle-mustillo self-assigned this Oct 19, 2026
…strate-atespace-scope-indep

Signed-off-by: Danielle Mustillo <danielle.mustillo@solo.io>

# Conflicts:
#	go/core/internal/service/system/service.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants