Repository navigation
Merge upstream worker, snapshot, and preview gate updates - #79
Merged
Merged
Conversation
…trate#2680) Today `deploy ate-system`, `deploy apiserver`, `deploy postgres` and `setup csi` already create missing pools. `deploy podcertificate-controller` didn't. `DeployPodCertificateController` now creates any missing pool before it applies the controller. It doesn't touch pools that already exist, so no rottation. `setup csi` and `deploy postgres` made the same call right before calling it, so I removed those two calls. Found while testing upgrade E2E from v0.4.0 to main on kind. Part of agent-substrate#1686 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
We can still consider more flexible overcommit in the future, but I think leaving actors uncapped is unsafe, and it's hard to pick a reasonable universal default, it's too workload dependent. Actor authors should really be aware of how much resources their actor requires, and tuning them can have significant performance impacts. In the future we could do something like determining how much memory was in use at snapshot time and consider that for scheduling resumes. #### Breaking change ActorTemplates *must* declare CPU/memory limits. Previously `microvm` actors had defaulting, and gvisor actors were simply uncapped. This made the scheduler unaware of placement for gvisor. Requiring it also ensures actor owners are aware of how much resources they cost. Setting actor size will be important for performance particularly with microvms where higher memory limits have a real cost for e.g. snapshot/resume.
…olicy e2e (agent-substrate#2670) `TestNetworkPolicyDataPlaneEnforcement` started a `busybox` pod pulled from Docker Hub and ran `wget || nc` in it with `kubectl exec` to check that the worker NetworkPolicy blocks direct ingress. This PR removes that external image. **Changes** - `testserver http` gains a `/dial?address=host:port[&timeout=]` endpoint. It attempts a TCP connection from inside the pod and returns `{address, ok, error}`. - The test runs `testserver http` as the probe, in a fresh namespace, and calls `/dial` over a port-forward. - New control check: the probe must first reach a pod in the worker's namespace that no policy selects. Before, any failure counted as "blocked", including a broken probe. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…trate#2388) Add --ingress-auth-mode to ate-setup, to pick between static-mtls and insecure modes for the ingress router. By default, static-mtls mode is set up to trust certificates issued by podidentity.podcert.ate.dev/identity, which is the most convenient option for setups running within the scope of a single cluster.
…able snapshot (agent-substrate#2677) Part of agent-substrate#2069 Add a new snapshot state tracker to atelet. It keeps a record of what the latest local and durable snapshot for an actor that this node has seen. The state is written to disk in the actor directory. Checkpoint will update the state and return it in its response. Right now the response is used by ateapi, but once we support asynchronous uploads, it will become important for communicating when a local snapshot has ben uploaded successfully. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…trate#1994) I went through a number of models and this seems to be the simplest and lowest overall impact. To keep it simple: * All gates are disabled unless enabled * Only "preview" and "ga" - no "alpha" or "beta" * We only have one gate for now, called "Preview" (later we can break this down, maybe) * All gates can be activated by passing "*" * Package internal/preview has an Init() function which gets passed a list of gates to enable, which is stored as a package-global * Code can call preview.IsEnabled(gatename) Fixes agent-substrate#1690 @juli4n @BenTheElder @msau42
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. #### Breaking change fixes agent-substrate#2065 renames ateom ahead of GA. this is breaking, atelet and the worker images need to be updated together. - images / binaries: ateom-gvisor, ateom-microvm -> ateworker-gvisor, ateworker-microvm - worker pod container: ateom -> worker - RPCs: ateom.Ateom -> ateworker.Worker, atelet.AteomSupport -> atelet.WorkerSupport, atelet.AteomHerder -> atelet.Atelet, MintAteomActorCertificate -> MintWorkerActorCertificate - SPIFFE path: ateom-for-actor -> worker-for-actor - ate.snapshot.phase values: ateom_restore, ateom_checkpoint -> worker_restore, worker_checkpoint - node paths: /var/lib/ate/ateoms/<uid>/ateom.sock -> workers/<uid>/worker.sock, ateom-support.sock -> worker-support.sock - cgroups: the worker's leaf ateom -> worker, the micro-VM guest's /ateomchv -> /actor - authz relation: can_mint_ateom_actor_credential -> can_mint_worker_actor_credential per review, "worker" where any worker implementation fits and "ateworker" for ours. the internal go packages (internal/ateom*) keep their names for now, that's a follow-up. one commit per rename to make review easier. tested with kind e2e (gVisor + micro-VM) on an earlier rebase, plus CI. This PR was AI assisted. I have reviewed the changes. > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…licy object (agent-substrate#2651) This PR reduces memory usage by SNI based egress policies, surface by scalability tests with 1M actors and policies and 1K SNI rules per policy. The cause of memory consumption was two fold: 1. policy proto sent from ext_proc to Envoy was kept in memory, even though it was no longer used after it was converted into internal policy representation. 1. Each CONNECT request kept its own copy of the egress policy, duplicating policy in memory by factor of Actors*Connections_per_actor. This PR makes two high level changes: 1. policy proto sent from the ext_proc is erased after it is parsed into the Envoy internal policy object. 1. policy object is hashed using high quality Rapidhash and stored in thread local store using computed hash key. The policy hash is then communicated to the inner filter chain that fetches policy from the store and enforces SNI rules. With this change, if all actors have exactly the same policy, Envoy will only have 1 copy of the policy object per worker thread. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Yan Avlasov <yavlasov@google.com>
…te#2696) Part of agent-substrate#2069 Adds client-go workqueue of actor UIDs drained by a fixed pool of workers to run background tasks. An actor has at most one background operation in progress. Checkpoint and Terminate never wait for it: commits leave it running, and Forget cancels it. Failed background work backs off exponentially per actor and starts over once the operation is complete. Commits queue the actor for its background work, and Start queues every actor with a snapshot state file, so that work resumes after an atelet restart. No background work is defined yet, so nothing is scheduled. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
) Part of agent-substrate#2069 ateapi assigns each actor snapshot a generation, but never passed it to atelet: every `ateletpb.Snapshot` it builds left `generation` unset. That covers the pause and suspend `Checkpoint`, the paused-actor `PromoteSnapshot`, and both `Restore` paths. This sets it from the snapshot each request names, and narrows the TODO on `Snapshot.generation` to say only that atelet does not check it yet. Atelet requires generation to be set but doesn't use the value yet. Once agent-substrate#2677 lands, atelet can reject a `Checkpoint` or `PromoteSnapshot` whose generation is stale, as the `Snapshot.generation` doc describes. That check came up in review on agent-substrate#2677. > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…2682) On kind, `ate-setup deploy apiserver` doesn't work because the kind settings for `ate-api-server` lived only as an inline patch in the kind system bundle, so this command dropped them. The patch now lives in its own overlay, `manifests/ate-install/kind/ate-api-server`, the way `kind/atelet` already works. Found while running the upgrade E2E on kind, where `ate-setup deploy apiserver` timed out. Part of agent-substrate#1686 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Follow-up to agent-substrate#2254, renames the remaining internal/ateom* go packages. After this change there is no `ateom` left in the repo. 🫡 - Worker only packages move under `internal/ateworker`: register (the registration code, leaving `internal/ateworker` itself for the shared worker code), cgroup, pid1 (was ateominit), actornet (was ateomnet), phaselog, suspend, tunnel - Two packages go flat instead: `internal/workerstats`, which atelet also imports, and `internal/netns`, a generic named netns helper that atunnel's tests also use - The downward API file names atecontroller needs move to `internal/workercapacity`, so it doesn't import the worker registration code - `depguard` (`golangci-lint`) now fails any import of `internal/ateworker` outside `cmd/ateworker-*` and `internal/ateworker` - Drops an unused stopwatch package - Renames cmd/atelet's ateom* files The move commits are only `git mv`, imports and package names are fixed in a separate commit after each round of moves, to make it clear which changes are just file moves and which changes are import rewrites etc. We squash merge, so the intermediate commits don't need to build. This PR was AI assisted. I have reviewed the changes.
agent-substrate#2688) This is a step toward [async suspend](https://github.com/zoez7/substrate/blob/async-suspend-design/docs/dev/async-suspend-en.md). The next PR merges `SuspendActor` and `PauseActor` into one set of shared steps: sync suspend, and async suspend (today's pause). Both APIs stay. The current unit tests call the private step methods (`ensureMarked*`, `ensureAtelet*`, `ensure*Finalized`). But the next PR will rewrite exactly those steps. If the tests stay as they are, they get rewritten in the same PR as the refactor, and nothing shows that the refactor kept the behavior. This PR moves the tests onto the public `SuspendActor` and `PauseActor` entrypoints and checks them against the current implementation. The merge PR then shouldn't need to touch any test file: if this suite still passes unchanged, behavior is preserved. Test-only change. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
) Previous to this errors might be helpful (dumping full help text) but they might also be spartan: ``` $ kubectl-ate dasdsd Error: unknown command "dasdsd" for "kubectl-ate" Run 'kubectl-ate --help' for usage. ``` or they might be useless: ``` $ kubectl-ate -r Error: unknown shorthand flag: 'r' in -r ``` Sometimes they used stdout, and sometimes stderr. In short, Cobra's error handling is a mess. There are bazillions of options and I could not find a combination that did something useful. So now we take control ourselves. Every error should: 1) print an error message telling you what went wrong ("missing flag", "unknown subcommand", etc) 2) dump the closest help message 3) write to stderr exclusively This also cleans up flag descriptions. Gemini helped. Fixes agent-substrate#2463 > It's a good idea to open an issue first for discussion. - [X] Tests pass - [X] Appropriate changes to documentation are included in the PR
Preserve fork integration while adopting the ateworker APIs and required actor resource limits. Carry preview gates through Helm and retain agentgateway validation for both preview modes. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261015
branch
from
October 15, 2026 15:03
10483a3 to
29d9b0b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge 14 upstream commits through agent-substrate/substrate@6edd1fc into the fork. This brings in the worker rename, snapshot tracking and generation checks, preview gates, required actor resource limits, and CLI and installer fixes.
The merge preserves the fork's agentgateway dataplane, Helm integration, optional atelet TLS, and workflow deadlines. It updates release image names and chart CRDs, adds Helm preview configuration, and runs the agentgateway CI matrix with preview features both enabled and disabled.
Breaking change
ateom-gvisor/ateom-microvmtoateworker-gvisor/ateworker-microvm.ExternalVolumeson the server and an opted-in client. Helm supportspreview: [ExternalVolumes];kubectl ateaccepts--preview=ExternalVolumes.Validation passed: race-enabled Go tests (
env -u NO_COLOR go test -race ./...), the apitool module tests, all 10 root-gated test packages,hack/verify-all.sh, Helm lint and 13 chart tests, manifest rendering, and CRD parity.NO_COLORis unset for the existing ANSI-output assertions. Local cluster E2E was not run; the PR workflows cover gVisor and microVM.Merge this PR with a merge commit to preserve upstream ancestry for the next sync.