Repository navigation
Sync upstream asynchronous snapshots and sandbox runtime updates - #89
Merged
Merged
Conversation
Before this change, if we ever forgot to check for len(errors) before calling `resources.ToAPIError`, we risked sending an InvalidArgument for a nil error. As a side change, I migrated some callers to use `resources.ToAPIError` instead of using apierror.InvalidArgument directly
…2679) GKE taints every node it creates for a [ComputeClass](https://cloud.google.com/kubernetes-engine/docs/concepts/about-custom-compute-classes) with `cloud.google.com/compute-class=<class>:NoSchedule`, and adds the matching toleration only to pods that select the class (`nodeSelector: {cloud.google.com/compute-class: <class>}`). A WorkerPool can select a ComputeClass through `template.nodeSelector`. For micro-VM pools that's the natural way to get nested-virtualization nodes (`priorityDefaults.enableNestedVirtualization: true`), with fallback across machine families on stockouts. But the atelet DaemonSet only tolerates `ate.dev/sandboxClass`, so it never runs on those nodes. Without atelet, the node advertises no `ate.dev/kvm`, and the workers stay Pending: ``` 0/3 nodes are available: 1 Insufficient ate.dev/kvm, 2 node(s) didn't match Pod's node affinity/selector. ``` This PR: - adds a toleration for the ComputeClass taint key (any value) to the atelet DaemonSet, - adds a sentence to the API guide. #### How I tested it On GKE Standard 1.37.1-gke.1552000, I used a ComputeClass with `n4` → `c3` → `n2` priorities, nested virtualization, and node pool auto-creation. Substrate was main at e3b707f (this branch is rebased onto a later main with no changes to these files), and the counter-microvm WorkerPool selected the class. - **Before:** GKE created a tainted `c3` node for the class (after `n4` was out of capacity). atelet didn't schedule there: the DaemonSet stayed at the two default-pool nodes. The workers stayed Pending, and `deploy demo counter-microvm` timed out at `0/2 replicas available`. - **With this change** (`hack/install-ate.sh --deploy-atelet`): atelet ran on the ComputeClass node, which then advertised `ate.dev/kvm: 4096`. The workers scheduled and the golden snapshot was built. The counter actor kept its count across suspend/resume, including a resume onto a different worker pod. - `go test ./cmd/ate-setup/internal/steps/` passes. `hack/verify/gofmt.sh`, `boilerplate.sh` and `golangci-lint.sh` pass. #### Related, not addressed here - **Autoscaling.** Cluster Autoscaler can't create a ComputeClass node pool for a micro-VM worker. The pod requests the `ate.dev/kvm` extended resource, which exists only once atelet runs on a node, so node auto-provisioning rejects it (`no.scale.up.nap.pod.zonal.resources.exceeded`). There's no scale-from-zero and no fallback to another ComputeClass priority for micro-VM workers: a pool only grows onto existing nodes, or onto more nodes of an existing node pool that already has a live node. I can open a separate issue to discuss options, for example not requesting the device on nested-virtualization nodes, or documenting pre-provisioning with the ComputeClass `minimumCapacity` field. - **Other taints.** A WorkerPool can set arbitrary `template.tolerations`, so the same mismatch applies to any custom taint. If you'd rather have atelet tolerate all `NoSchedule` taints instead of this GKE-specific key, I'm happy to switch. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…the background after a timer (agent-substrate#2733) Part of agent-substrate#2069 and agent-substrate#2571 Immediate (`UPLOAD_MODE_IMMEDIATE`) checkpoints and PromoteSnapshot now keep a local copy of the snapshot they upload, until the detach deletes it. An immediate checkpoint that uploads its snapshot but fails to keep the local copy still succeeds. Detaching a snapshot that is already durable skips the upload. In the background, atelet detaches an actor that stays suspended on its node for `--snapshot-detach-delay` (default 5m) after its last checkpoint. A detach makes the actor's latest local snapshot durable, uploading it, then deletes the actor's local snapshots. A snapshot without an object storage location, as pause snapshots are today, is not detached until PromoteSnapshot makes it durable. Run and Restore unschedule the actor's detach until its next checkpoint, and fail with Unavailable if atelet cannot save that. Checkpoint, Run, Restore, and PromoteSnapshot wait for a detach in progress, and a detach that comes due during a Checkpoint or PromoteSnapshot is requeued once it ends. Terminate cancels a detach in progress without waiting for it. The detach schedule is kept in the actor's snapshot state file, so it survives an atelet restart. The snapshot tracker's Detach method runs a detach on request, and fails with Aborted while the actor is in use on the node. Nothing calls it until atelet implements the DetachActor RPC from agent-substrate#2725. <img alt="image" src="https://github.com/user-attachments/assets/e0279dbd-5c2e-4dd4-9754-67daeaeb6c3e" /> #### Benchmark Benchmark shows improvement in ResumeActor because we keep the local snapshot now. Glutton 1 user for 3 minutes (-t 3m -u 1), --lifecycle-mode suspend, --mem-target 512Mi, --mem-churn 32Mi, --min-wait-time 1.0, --max-wait-time 1.0, --atelet-lag-s 0. | RPC | Metric | `upstream/main` | `keeping local snapshot` | Change | | :--- | :--- | ---: | ---: | ---: | | **ResumeActor** | Count (`n`) | 25 | 33 | +32% | | | Avg | 2245 ms | 885 ms | **-1360 ms (-61%)** | | | p50 | 2200 ms | 890 ms | **-1310 ms (-60%)** | | | p90 | 2500 ms | 940 ms | **-1560 ms (-62%)** | | | p95 | 2800 ms | 970 ms | **-1830 ms (-65%)** | | | Max | 3115 ms | 1002 ms | **-2113 ms (-68%)** | | **SuspendActor** | Count (`n`) | 26 | 33 | +27% | | | Avg | 3587 ms | 3259 ms | -328 ms (-9%) | | | p50 | 3300 ms | 3000 ms | -300 ms (-9%) | | | p90 | 5400 ms | 4500 ms | -900 ms (-17%) | | | p95 | 5700 ms | 5000 ms | -700 ms (-12%) | | | Max | 5776 ms | 6162 ms | +386 ms (+7%) | #### Notes for reviewers - The three commits are easiest to review one at a time: the detach, keeping the local copy, and the cleanup phase. - atelet does not tell ateapi about a detach yet. A Restore that ateapi still sends to the node reads the snapshot from object storage. - ateapi's suspend workflow still assumes an immediate checkpoint keeps no local copy. - A promote leaves the detach schedule as it is. Promoting a paused actor more than `--snapshot-detach-delay` after the pause detaches it as soon as PromoteSnapshot returns. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…#2665) Changes atelet's default gRPC port (`atelet.DefaultPort`, also its `hostPort`) from 8085 to 21998, so it is less likely to conflict with other processes on the node. atelet is the only Substrate component that binds a port at the node level. Notes for reviewers: - Only node-level ports are in scope. Pod-network listeners keep their defaults, including atunnel's ingress, CONNECT and egress ports on worker pods (443/8443/15001; the router dials `workerIP:443` directly). So do ateapi, the router, the credential providers, and metrics/health. - ateapi dials `atelet.DefaultPort`, so ateapi and atelet need to roll out together. - If Substrate needs more node-level ports later, we recommend 21992 and 21988, in that order. Like 21998, they are not IANA assigned so rarely used. --------- Co-authored-by: Hoda Mohammadi <173203200+hodaaaaaaaaaa@users.noreply.github.com>
Follow up on agent-substrate#587 1. **Pass `SandboxRuntime` on `ateletpb.RunRequest` and `ateletpb.RestoreRequest`**: - Adds `SandboxRuntime sandbox_runtime` to `ateletpb.RunRequest` (field `12`) and `ateletpb.RestoreRequest` (field `17`), along with declarative validation. - Updates `ensureAteletRestored` in `cmd/ateapi/internal/controlapi` to populate `SandboxRuntime` from `WorkerAssignment.SandboxRuntime` on both `Run` and `Restore` requests alongside `SandboxAssets`. This prepares the `ateapi` -> `atelet` wire contract so `atelet` can resolve the selected `SandboxConfig` version directly from the worker assignment's runtime in a follow-up change. 2. **Rename `sandbox_runtime_hash` to `sandbox_bundle_hash`**: - Renames `hardware.AttrSandboxRuntimeHash` (`"sandbox_runtime_hash"`) to `hardware.AttrSandboxBundleHash` (`"sandbox_bundle_hash"`) across `internal/hardware`, `internal/sandboxruntimes`, `internal/ateworker/register`, `cmd/atelet`, `internal/proto/ateletpb`, and `pkg/proto/ateapipb`. - Clarifies that this attribute is the composite SHA-256 hash of a `SandboxVersionConfig`'s runtime bundle (`pauseImage` + per-architecture asset SHA-256 digests), distinguishing it from the full `SandboxRuntime` proto message (which also includes host hardware attributes like `architecture` and `cpu_features`) and from individual `AssetFile.sha256` digests. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
dynconfig had become one struct of every knob every user class ever needed, with a payload mirror, a merge, a validator, and a log line that all need to grow by hand for each new field, and every class saw and was checked against every other class's knobs. The payload is now an arbitrary JSON object, and a worker runs exactly one user class, known at launch. So each class declares the keys it reads as a struct with json tags, wraps it in a dynconfig.Typed codec with its defaults and its validation func, and registers the codec with its userclass.Entry. The worker builds the holder from that codec; the holder stores the typed value, folds each payload into it with one decode over the current value (so an absent or null key keeps what it had), refuses a payload the class cannot run on in favor of the last good config, and hands the class its knobs back through dynconfig.Get. Keys the class does not name are ignored. The wait window and the resume and lifecycle modes are shared pieces a class embeds; durations travel as seconds, the unit the locust flags use. Now we can implement user classes more independently of each other. > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
| Library / Runtime | Blocked Lookup Error (REFUSED → NXDOMAIN) | Search
Domain Traversal (<svc>.<ns>) (REFUSED → NXDOMAIN) | musl Blocked Lookup
Duration (REFUSED → NXDOMAIN) |
| --- | --- | --- | --- |
| glibc (C / C++ / Linux) | EAI_AGAIN ("Temporary failure...") →
EAI_NONAME ("Name or service not known") | ❌ Aborted → ✅ Resolves via
2nd suffix | N/A (glibc: 0.24s) |
| musl libc (Alpine Linux) | EAI_AGAIN ("Try again") → EAI_NONAME ("Name
does not resolve") | ❌ Aborted → ✅ Resolves via 2nd suffix | 1.06s →
0.06s |
| Golang (Pure Go) | server misbehaving (IsTemporary: true) → no such
host (IsNotFound: true) | ✅ Resolved → ✅ Resolves via 2nd suffix | 0.08s
→ 0.08s |
| Golang (cgo) | Temporary failure / Try again → no such host | ❌
Aborted → ✅ Resolves via 2nd suffix | 1.08s → 0.09s |
| Node.js (dns.lookup) | EAI_AGAIN → ENOTFOUND | ❌ Aborted → ✅ Resolves
via 2nd suffix | 2.95s → 1.88s |
| Python (socket / asyncio) | [Errno -3] Temporary failure / Try again →
[Errno -2] Name or service not known / Name does not resolve | ❌ Aborted
→ ✅ Resolves via 2nd suffix | 3.20s → 2.24s |
| Python (dnspython) | dns.resolver.NoNameservers (REFUSED) →
dns.resolver.NXDOMAIN | ❌ Aborted → ✅ Resolves via 2nd suffix | 1.97s →
2.06s |
| Java (OpenJDK InetAddress) | UnknownHostException: Temporary failure /
Try again → UnknownHostException: Name or service not known / Name does
not resolve | ❌ Aborted → ✅ Resolves via 2nd suffix | 2.17s → 1.17s |
| Java (Netty DnsNameResolver) | DnsNameResolverException: REFUSED(5) →
SearchDomainUnknownHostException / NXDOMAIN(3) | ✅ Resolved → ✅ Resolves
via 2nd suffix | 10.84s → 11.10s |
| Rust (std::net::ToSocketAddrs) | Temporary failure / Try again → Name
or service not known / Name does not resolve | ❌ Aborted → ✅ Resolves
via 2nd suffix | 1.05s → 0.05s |
| Rust (hickory-dns / trust-dns) | NoRecordsFound { response_code:
Refused } → NoRecordsFound { response_code: NXDomain } | ✅ Resolved → ✅
Resolves via 2nd suffix | 0.06s → 0.05s |
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
…strate#2764) Follow-up to agent-substrate#2590. This removes redundant `TestActorEgressHTTPSByHostnameMITM`, `TestActorEgressHTTPSByHostnamePassthrough`, and their helpers `hostnamePolicyActor` and `isMitmCert`, introduced in agent-substrate#2524. `TestActorEgressHTTPSByHostname` already covers each of their checks, using in-cluster origins: - **mitm:** the actor's trusted roots include the gateway's CA but not the origin's signer, so a verified fetch proves the gateway minted the certificate. - **passthrough:** the fetch trusts only the origin's CA, so a minted certificate would fail verification. - **unlisted host denied:** a name that only has an HTTP rule is denied on 443. The check also fails on a certificate error, so a leak to passthrough can't count as a denial. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
Part of agent-substrate#1756. - Adds `manifests/ate-install/ate-idp-server.yaml` to the base and kind bundles: two replicas behind the `idp` Service, with a servicedns certificate for `idp.ate-system.svc` and the `actor-id-jwt-pool` Secret ate-api-server signs with, so every install serves the discovery document and key set. - ate-setup sets ate-idp-server's `--issuer` flag from the same setting it gives ate-api-server, so the two can't disagree. - ate-idp-server accepts `--preview`, since ate-setup passes it to every substrate container. - ate-api-server and ate-setup now accept only `https://<host>` issuers through `actoridjwt.ValidateIssuer`, matching what ate-idp-server serves, and `internal/oidcdiscovery` is removed. - Adds `cmd/ate-idp-server` to `images.Components` and to the cordon-control-plane component. - Adds a NetworkPolicy admitting its `https` port from any source, since `ate-system` now denies ingress by default and relying parties fetch the public documents from any namespace and from outside the cluster. For reviewers: a changed issuer reaches ate-idp-server on the next `deploy ate-system`, not `deploy ate-apiserver`. The probes use `/healthz`, with no `/readyz` that turns 503 on SIGTERM, so a rollout can refuse a few connections while endpoints update. An install whose issuer has a path now fails ate-setup's config check. Testing: unit tests for the rendered bundles (the `--issuer` flag, shared pool volume, cordon pinning) and issuer validation. I didn't run it on a cluster. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…-actor snapshot credentials (agent-substrate#2778) Part of agent-substrate#777 A background detach, added in agent-substrate#2733, uploads a paused actor's local snapshot knowing only the actor's UID, so the `PromoteSnapshotRequest` it builds lacks the atespace and name that the same request from ate-api-server carries. The per-actor credential work needs every upload to name the actor it acts for, including a retry whose local copy is already gone and can only probe the remote manifest, which is the one path that cannot read the names from the local checkpoint's own record. The snapshot tracker's `SnapshotRef` now carries the actor, set by the Checkpoint or PromoteSnapshot that commits the snapshot, kept in the snapshot state file across restarts, and passed to the upload. Nothing changes for users today, since atelet does not yet use the two fields on this path. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…density (agent-substrate#2444) ### Summary Each awake actor sandbox creates its own network namespace, veth/tap pair, and inotify watches on the host kernel (even in multi-actor worker pods). At the default COS kernel limits (`net.ipv4.neigh.default.gc_thresh3 = 1024`, `fs.inotify.max_user_instances = 128`, `fs.inotify.max_user_watches = 8192`–`65536`), nodes hit `inotify` exhaustion around ~800 live sandboxes and ARP table overflow (`neighbour: arp_cache: neighbor table overflow!`) around ~1,000 live sandboxes per node, causing new sandbox creation/networking to fail (`100%` ping failure at 1,000 sandboxes/node; `0%` failure up to 1,500 sandboxes/node once raised). This PR: 1. Configures `LinuxNodeConfig.Sysctls` on the default `substrate-node-pool` created by `tools/setup-gcp/cmd/cluster.go`: - `net.ipv4.neigh.default.gc_thresh1 = 4096` - `net.ipv4.neigh.default.gc_thresh2 = 8192` - `net.ipv4.neigh.default.gc_thresh3 = 16384` - `fs.inotify.max_user_instances = 65536` - `fs.inotify.max_user_watches = 1048576` *(Note: GKE creates an IPv4-only VPC cluster by default and rejects `net.ipv6.neigh.default.gc_thresh*` in `LinuxNodeConfig.Sysctls`.)* 2. Adds unit test assertions in `tools/setup-gcp/cmd/cluster_test.go` verifying the node pool's `LinuxNodeConfig.Sysctls`. 3. Documents in `tools/setup-gcp/README.md`: - How to pass `--system-config-from-file=systemconfig.yaml` when creating additional node pools (e.g., bare-metal pools via `gcloud container node-pools create`). - How to apply an in-place `substrate-sysctl-tuner` DaemonSet on an already-running node pool without rolling/recreating nodes. ### Testing - `go test ./tools/setup-gcp/cmd/... ./cmd/ate-setup/...`
…d-trip. (agent-substrate#2094) Some external storage backends (such as Cloud Filestore IP export rules) scope volume access to the Kubernetes worker node currently hosting an actor via CSI `ControllerPublishVolume` and `ControllerUnpublishVolume`. This PR verifies end-to-end NodeId and PublishContext forwarding across the CSI plugin, control plane resume workflow, and functional actor lifecycle (including cross-node migration). ### Changes - CSI plugin: `TestPlugin_AttachVolume` / `TestPlugin_DetachVolume` assert the `NodeId` and `VolumeId` sent on publish and unpublish, including the `Unimplemented` path. - Mock drivers: `MockVolumePlugin` and the functional test's `attachFailVolumePlugin` return a publish context naming the target node and volume, so existing lifecycle tests now carry a non-empty `publishContext` through Restore. - Lifecycle test: builds on the agent-substrate#1581 pause lifecycle test, renamed `TestPauseActor_VolumeLifecycle_DetachResumeAndMigrate`. It checks the Restore request's `publishContext` after each resume, adds a migration leg (suspend → worker on node2 → resume → attach to node2), and verifies detach, volume deletion, and actor removal on delete. `lastRestoreVolumePublishContext` replaces the per-test lookup of the Restore request. - Unit test: `TestEnsureVolumesAttached_ReturnsPublishContext` also asserts that `AttachVolume` is called once, for the mounted volume, with the worker's node. - [x] Tests pass
…default (agent-substrate#2376) Attach a virtio-balloon device with free page reporting to guests by default, returning freed guest memory to the host. ### Motivation Substrate places actors by their declared limits. The agents frequently do not utilize their full memory allocations. [FPR](https://docs.kernel.org/mm/free_page_reporting.html) allows the virtio-balloon driver to report lists of currently unused pages to the host in batches (default reporting duration is 2s, can be tuned via the kernel parameter `page_reporting_delay_ms` ). This allows the host to safely reclaim idle guest memory. #### Impact & use-cases 1. FPR in itself doesnt change much for the per actor metrics as it is just a mechanism for reporting idle pages from inside the host to the guest. Swe-perf comparisons showing this is a safe change to merge: Single actor- | Metric (median) | A: base | B: PR agent-substrate#2376 | | --- | ---: | ---: | | Per-VM memory, single actor (MiB) | 286.0 | 286.1 | | Per-VM memory in the 16-user soak, p50 (MiB) | 287.3 | 287.3 | | Snapshot memory-ranges (MB) | 298 | 298 | | Suspend p50 / p99 (ms) | 3,100 / 12,000 | 3,050 / 10,250 | | Resume p50 / p99 (ms) | 2,150 / 3,250 | 2,150 / 3,200 | | SWE-Perf cycle time, p50 (ms) | 4,500 | 4,800 (+6.7%) | | Cycles completed | 1,264 | 1,272 | 16 users, 15 min, 2 repeats per arm. | Metric | A base | B PR agent-substrate#2376 | | --- | ---: | ---: | | Total /memfd p50 MiB | 2588 | 2588 | | Suspend p50 / p95 / p99 ms | 3100 / 7150 / 12000 | 3050 / 6600 / 10250 | | Resume p50 / p95 / p99 ms | 2150 / 2850 / 3250 | 2150 / 2850 / 3200 | | TaskWallClock p50 / p95 / p99 ms | 43500 / 52500 / 55000 | 43000 / 50000 / 55000 | | CycleCEL p50 ms | 4500 (4400-4600) | 4800 (4700-4900) | | Failures | 0 | 0 | 2. FPR saves memory while an actor keeps running after it frees memory, so its value scales with freed MiB x idle time. FPR is particularly valuable for long-running agents (like RL workloads) where the guest will have some reporting cycles, when the free unused memory can be given back to the host for better density. **RL like glutton arms** | Phase | Arm 1 base (tuned + no FPR) | Arm 2 PR (tuned + FPR) | | --- | ---: | ---: | | peak MiB | 969.16 | 968.10 | | dwell t+4 s MiB | 977.16 | 280.47 | | returned to host | 0.00 (0.0%) | 695.98 (90.6%) | | post-resume /memfd (MiB) | 977.81 | 317.81 | | SuspendActor to GCS (ms) | 7,263.44 | 4,743.10 | Note: Both the arms has some GC settings applied (GOGC=10, GOMEMLIMIT=96MiB), this 'freed' memory comes from Go's heap, but Arm 2 shows that over time FPR is able to give back this to the host. 3. In short duty spans, the suspend path gives back the whole VM back to the host, so actively punching holes in a short running actor's memfd doesnt' dramatically benefit bin-placement. Fixes agent-substrate#2375 > It's a good idea to open an issue first for discussion. - [X] Tests pass - [x] Appropriate changes to documentation are included in the PR
…napshot upload (agent-substrate#2776) Part of agent-substrate#777 The suspend workflow marks a paused actor's local snapshot as a PENDING durable copy in the same write that sets SUSPENDING, then asks the node that holds it to upload it through `PromoteSnapshot`. `HostingNode` trusted `assigned_node` only when the latest generation had no durable copy at all, so that node's `MintWorkerActorJwt` for the upload was refused with `NotFound`. The unit test for this case used a fixture with no durable copy, a shape the workflow never produces, so it passed. `HostingNode` now accepts a durable copy that is absent or PENDING, and still refuses a COMPLETED one, after which the node has nothing left to do for the actor. The fixture carries the PENDING copy, and a direct test pins the six no-worker shapes. Nothing changes for users today, since atelet does not yet mint for its transfers. The node plugin PR that follows depends on this. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…nd finalizes (agent-substrate#2766) Part of Actor Lifecycle v2 agent-substrate#2069. atelet's `Checkpoint` and `PromoteSnapshot` return its snapshot state for the actor (agent-substrate#2677). The suspend workflow discarded it. This applies it when a suspend finalizes, with the same rules as a `ReportActorSnapshotState` report (agent-substrate#2622, agent-substrate#2698). No proto changes. - `ensureAteletSuspended` and `ensurePausedSnapshotUploaded` return the state atelet reports. - `ensureSuspendedFinalized` applies it with `applyCheckpointState` before it frees the worker, and records the result in the same update that commits `SUSPENDED`. - A durable snapshot records the fidelity atelet reports. For `Checkpoint` and `PromoteSnapshot`, that's the one requested. - An upload atelet reports finished for an older generation is marked `COMPLETED`, so finalize keeps it rather than pruning it as `PENDING`. - `applyLatestDurable` refuses a reported durable snapshot with no fidelity, rather than recording `UNSPECIFIED`. atelet echoes the fidelity it was asked for, and reports already require one. #### Notes for reviewers - **The state is never stale.** Every attempt that reaches finalize has just called atelet, so a retried suspend applies the state from its own call. - **A conflicting state fails the suspend.** Finalize applies atelet's state before it frees the worker. If the state conflicts with the record, for example a different upload URI, finalize returns the error and the actor stays `SUSPENDING` with its worker, so a retry can call atelet again. - **The lifecycle-op metric's fidelity label** stays the requested fidelity. - **With today's atelet, the record doesn't change.** The applied state matches what finalize already records. It starts to matter once atelet uploads in the background: an async suspend's response can report an older snapshot as uploaded before atelet's report arrives, and finalize then keeps that snapshot as `COMPLETED` instead of pruning it. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…ation (agent-substrate#2757) Part of agent-substrate#777 The objectstore control plugin's credential exchange looks up the actor a JWT names with `GetActor`, over the loopback, presenting the ate-api-server pod identity. Since agent-substrate#2591 made authorization enforcement the default, that identity holds no role, so the lookup is refused and every exchange fails on a current install. `api_server_sidecar` is a system role like the other components': granted through a contextual tuple for a SPIFFE ID, never through an AccessPolicy. It is named for the identity rather than the program, since authorization sees the certificate a caller presents and every container in the pod presents this one. `--api-server-sidecar-spiffe-id` binds it and defaults to the pod's own identity, `spiffe://cluster.local/ns/<namespace>/sa/ate-api-server`. The role grants `can_get` on actors and nothing else: no listing, no egress policies, no workers, no mints, no writes. The tests pin each refusal. The grant does not widen the pod's reach. The same pod mounts the Postgres client certificate and the JWT signing pool, so anything that can read its volumes already has far more than actor reads. Reading an actor is already granted to `egress_gateway`. The exchange exercises the role only for an actor named by a JWT it has verified. Verified on a GKE dev cluster with enforcement on: without the role the exchange's lookup is refused with `principal "user:spiffe://cluster.local/ns/ate-system/sa/ate-api-server" lacks "can_get" on "actor:..."`, with it the exchange passes end to end. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
images often ship an empty or missing /etc/hosts and neither runtime wrote one, so localhost and the actor's hostname went to DNS and failed. this ~matches kubelet behavior now.
…#2783) Currently the counter demo readme is not well set up for microvm. Added collapsable sections for microvm and gvisor. Allow the instructions for each to have the same flow. User can expand the section pertinent to them. The other sandboxes instructions remain minimized and reduces confusion. #### Breaking change Fixes # - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…rs advertise (agent-substrate#2731) > **Rebased onto main through agent-substrate#2755**, which puts `sandbox_runtime` on Run and Restore. This PR still resolves from `sandbox_assets`; switching to `sandbox_runtime` and removing `sandbox_assets` is the next PR. Four commits: the index, then three rounds of review fixes, kept separate so each thread's change is visible. Step 3 of agent-substrate#2675: atelet keeps an index of the sandbox runtimes its workers advertise, persisted on the node, and resolves an actor's assets through it. **The index.** The worker sandbox syncer already resolves each local worker pod to its pool's SandboxConfig and hashes every enabled version into the worker's `sandbox-runtimes.json`. It now also registers each hash with the asset record it was computed from. Two maps: runtimes by hash, never released (a request may carry a hash a worker advertised earlier; entries are a few URLs; TODO to release stale ones), and what each worker currently advertises by pod UID and sandbox class. A pool runs one class today; the shape admits several. **Order of registration.** The hashes are registered before the worker's file is written, so a hash is resolvable before any worker can register it with the control plane; a SandboxConfig or WorkerPool change registers the affected pools' runtimes from the event handler at once, ahead of the jittered rewrite of the files, which reads the listers only. What a worker advertises is recorded only after its file is written, and only for the runtimes the file names, so the index never credits a worker with a default its file does not carry. Two versions with one hash have the same assets and pause image, so the first registration's record is kept. **Persistence, no startup gate.** The runtimes map is also one file under the node state dir, `sandbox-runtime-index.json`, the map serialized, rewritten durably (`atomicfile.WriteSync`, data then directory) before a new entry enters memory. A worker registers a hash only after atelet wrote the worker's own file, which happens after the index was written, so every hash a worker registered before an atelet restart is on disk and loads before atelet serves. There is therefore no wait on the syncer at startup and no window in which a registered worker's request can miss. On load, an entry whose record does not hash to its key is dropped and a file that does not decode starts the index empty; the syncer writes the current runtimes again either way. A sync that cannot persist a runtime fails and is retried. The file stays small: a node sees a handful of configs with a few versions in its lifetime. Memory stays the read path. **Run and Restore** hash the request's assets with the same function the syncer uses and stage the index's record. A miss fails the request with FailedPrecondition naming the hash: it means the control plane placed the actor on a worker whose pool does not run that config or version. There is deliberately no fallback to the request's own assets. The hashing goes away in the next PR, which resolves from `sandbox_runtime`. **Readiness.** /readyz holds 503 from the health server's first answer until the gRPC port is open, through a check registered with the readiness object, instead of answering 200 during the trust bundle sync and the plugin wait. A syncer that cannot start is fatal: without it no runtime is ever indexed. **Tests.** Index registration with a shared hash across two configs, a re-plan replacing a worker's set while the old runtime stays resolvable, forgetting a gone pod, and early registration without a worker entry. Persistence: register, reopen, resolve with provenance; a mis-keyed entry dropped alone; a corrupt file loads empty; an unwritable path fails the registration. Resolution hit, miss, and unpinned pause image. A Run for an unadvertised runtime is refused with nothing fetched and no actor dir created, then resolves once advertised. The syncer indexes the pods already on the node and retries an unresolvable one in the background; a changed config's runtimes resolve before the jittered sync with both pods queued after it; a worker's entry follows its written file through a failed sync and a partial file; the process test asserts the index entry's fate in every case. Every lifecycle test that sends assets advertises them first. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
… only) (agent-substrate#2779) Adds an optional `suspend_options` field to `SuspendActorRequest`. Its `mode` (`DurabilityMode`) picks when `SuspendActor` returns: - `DURABILITY_MODE_SYNCHRONOUS`: returns once the snapshot is complete in durable storage. - `DURABILITY_MODE_ASYNCHRONOUS`: returns once the snapshot is captured on the node. The upload to durable storage keeps running in the background. `ASYNCHRONOUS` is going to be the default. This PR only adds the API surface and its validation. `ateapi` does not read the field yet, so `SuspendActor` behaves the same whether or not it is set. The server-side handling will follow in a separate PR. Part of agent-substrate#798 - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…te#2786) fixes a TestPlatformMetricsEmitted flake: `missing lifecycle states [crashed]` even though the actor reached CRASHED. The e2e lanes run one after another on the same kind cluster. When the second lane takes its lifecycle baseline about 5 minutes after the first lane's crashes, the collector's Prometheus exporter expires those `crashed` series (default metric_expiration is 5m) right after the baseline. StatesNotAdvanced compared the sum across series, so the expired series dropped out and the new crash couldn't push the total above the baseline. https://github.com/agent-substrate/substrate/blob/d0e16d7f6/internal/e2e/collector_metrics.go#L264 Hit 3 times on 2026-10-16, always in the second lane, with the baseline 4m56s-5m21s after the first lane's last crash: - https://github.com/agent-substrate/substrate/actions/runs/38980427002/job/117239748680 - https://github.com/agent-substrate/substrate/actions/runs/38980976904/job/117241521846 - https://github.com/agent-substrate/substrate/actions/runs/38927528375/job/117062783550 This compares each series to its own baseline instead. A state counts as advanced if any series is new or changed value (a drop means deltatocumulative restarted the stream on a new count), and an expired series is ignored. AI-assisted.
We are deleting the `PauseActor` API and adding sync upload mode to SuspendActor. This PR removes the `can_pause` relation from the authz model. Until the RPC itself is removed, PauseActor is authorized with `can_suspend`.
…vider is slow (agent-substrate#2787) fixes a flake in TestActorEgressCredentialInjection/https on the envoy lane: https://github.com/agent-substrate/substrate/actions/runs/38982841766/job/117247440573 The test restarts k8s-credential-provider and then makes the first injected request. The gateway's FetchSecret to the new pod had no deadline, so a slow first call ran past the ext_proc message_timeout (5s) and envoy answered 504 with an empty body (failed at 5.18s). The test already retries 503 while the gateway reconnects to the provider, but not 504. The policy fetch already handles this by staying under the message timeout (4s): https://github.com/agent-substrate/substrate/blob/cc3354c19520f6c8c65be27699abe93342a4da75/cmd/atenet/internal/router/egress/policycache.go#L38-L41 This puts the same 4s cap on resolving a request's credentials (provider fetch and actor JWT mint), so a hung provider is the documented retryable 503. agentgateway has no outer message timeout, so it waits on a slow provider instead of returning 504 and doesn't hit this, otherwise I would've proposed alignment. AI-assisted.
…_runtime (agent-substrate#2788) > Follows agent-substrate#2755, which put `sandbox_runtime` on Run and Restore. Run and Restore stop hashing the request's `sandbox_assets` to find the runtime to stage, and read the `sandbox_bundle_hash` attribute of the request's `sandbox_runtime` instead. That field is what ate-api-server fills from the worker assignment: the runtime the worker registered from the file this node's atelet wrote, so the hash is in the index by construction. The stopgap hashing and its TODOs go. **Required.** A request without `sandbox_runtime`, or without the attribute, is InvalidArgument. Every worker ate-api-server can place an actor on has registered a default runtime, because `internal/ateworker/register` registers only once the runtimes file names one, so an assignment always has a runtime to send. An unknown hash stays FailedPrecondition. A hash advertised under another sandbox class than the request names is FailedPrecondition too, since the hash does not cover the class. **Not yet.** `sandbox_assets` stays in the request and is no longer consulted. Removing it from the protos, ate-api-server and atelet is the next PR, so this one stays a change to how atelet reads the request. **Tests.** The lifecycle tests' Run and Restore requests carry the runtime ate-api-server would send for an actor placed on the worker that advertises their assets. A request with assets but no runtime is InvalidArgument, and still before any on-node work: the invalid-Run test now proves the detach stays scheduled that way instead of through assets for another architecture, which the resolver no longer reads. The resolver test covers the miss, the hit, four malformed runtimes and the class mismatch. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…agent-substrate#2771) Add two Boomer/Locust flags for `agentsession` benchmarks: - **`--lifecycle-mode none`**: keeps actors resident in memory across steps without calling `PauseActor` or `SuspendActor` (for resident memory density benchmarks). - **`--agentsession-snapshot-steps`** (e.g. `3,10,20,30,40,50,61`): hibernates actors only after the specified 1-based step numbers and keeps them resident between all other steps (for measuring snapshot size across a session without hibernating on every turn).
…gent-substrate#2581) Fixes agent-substrate#1709 Follow-up to agent-substrate#1997, which did the same for atelet's AteworkerHerder RPCs. This brings declarative validation (DV) to the other end of that call: the `Ateworker` service both sandbox runtimes (`ateworker-gvisor`, `ateworker-microvm`) implement. Generated validation lives in a new `internal/ateworkervalidation` package, since both binaries are `package main` and must share one set of rules. Every Ateworker handler calls its validator first thing, before taking any lock, and turns failures into `InvalidArgument` through `resources.ToAPIError`. Rules are the ones atelet already enforces on the same values one hop earlier, shared through `internal/resources` so the two ends cannot drift: short names and UUIDs for identity, `ValidateMountPath` for every mount path, `ValidateHostPort` for the egress gateway, `ValidateHTTPGetPath` for the wakeup probe, `ValidateActorDirs` for the seven actor directories. Bounds mirror the producer's: `containers ≤ 10`, each mount list ≤ 32, `cpu_milli ≤ 999999`, `runtime_asset_paths ≤ 16` entries like atelet's `ArchAssets.files`. Two places ateworker's shape differs from atelet's, and the rules follow the shape: - A container's mounts arrive in four typed lists rather than atelet's one. Each list's key keeps its own paths unique, so a hook on `Container` gathers all four and holds their paths unique and non-nesting across them, reporting each finding on the mount it belongs to. - `WorkloadSpec.containers` is optional on the message, because the fallback spec atelet's Terminate sends for a template-less actor carries no containers. Run and Restore require at least one through a hook on their `spec` field, so neither runtime boots a sandbox around nothing. #### Hand-written checks this replaces - `validateActorDirs` in both binaries, now the `ActorDirs` hook. - The `actor_uid` presence checks in both stats handlers. - The empty-container guards in ateworker-microvm's boot and restore paths, now the `spec` hooks above. - ateworker-microvm's cap of 25 containers, which no accepted spec could reach once `containers` is bounded at 10. `ateworkervalidation.MaxContainers` names the bound. - The `actor UID is required` guard in each binary's `hostActor`, unreachable once every request that hosts an actor requires the UID. #### Removed fields `CheckpointWorkloadRequest.snapshot_uri` and `RestoreWorkloadRequest.snapshot_uri` date from when ateworker moved snapshots to object storage itself. atelet does that now, through the object-store plugin; nothing sets the fields and neither runtime reads them. They are removed rather than validated. #### What is deliberately *not* validated here - **Snapshot fidelity support.** DV only bounds `fidelity` to the enum. Which levels a runtime can capture (gVisor cannot do ROOTFS without MEMORY) stays the per-runtime check in each binary, as designed in agent-substrate#2448: atelet forwards every level and ateworker refuses what it cannot serve. - **`runsc_path` / `runtime_asset_paths` contents.** They must be files under the node's static-files root; that depends on host state and stays a hand check in the binary. DV bounds their length only. - **`container_spec`.** atelet derives it from template fields it already validated, merging the image's ENTRYPOINT, CMD and ENV in, so the template's bounds do not apply. It is marked opaque with that reason. #### Testing - `internal/ateworkervalidation`: one table per request type, plus tables for `WorkloadSpec`, `Container` (which covers the mount rules, including the cross-list cases) and `ActorDirs`. The probe is covered from the Run table. - Each binary's edge test asserts every RPC with a request body rejects an empty one. Existing handler tests that sent minimal requests now build valid ones, so they test the check they are aimed at. - `go vet` for `GOOS=linux` on both binaries at every commit (their tests are Linux-only and run in CI). - `hack/update/codegen.sh` is clean at every commit. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…rate#2386) Part of agent-substrate#1590 Adds `working_set` to server telemetry: - `node`, `worker`, `atelet`: from cAdvisor (`container_memory_working_set_bytes`). - `actor`, `per_actor`: reuse the atelet's existing actor stats metrics, `ate_actor_stats_memory_working_set_bytes` and `ate_actor_stats_sampled_actors`. `actor` is the actors' working set per atelet; `per_actor` divides it by the number of actors that atelet measured. Both sources are already collected (cAdvisor and the atelet), so nothing changes in Substrate or the monitoring setup; this only adds queries. Live check (gVisor, 4 users, 3m, through the telemetry-meter): - `actor` 1.13 GiB vs `worker` 1.22 GiB at the same time - `per_actor` × count = `actor` (0.5628 × 2 = 1.1257) - 74 measurements, none null - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Update default runsc flags configuration Configure additional global flags for runsc sandboxes: - --allow-suid: allow setuid/setgid executables to run properly inside the sandbox (e.g. sudo). This behavior is controlled by the Process.NoNewPrivileges flag. - --host-settings=ignore: avoid redundant host kernel setting checks during startup. *AI disclosure: AI tooling assisted in writing and testing parts of this change.* Updates agent-substrate#2136 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Follow-up to agent-substrate#2660 and agent-substrate#2755. Sandbox runtime and version selection live on the `WorkerPool`: its `sandboxClasses` entry resolves a `SandboxConfig`, atelet reports the enabled versions on `WorkerStatus` as `defaultRuntime`/`restorableRuntimes`, and the chosen runtime is stamped onto `WorkerAssignment` and `Snapshot` and sent to atelet on Run and Restore. The `ActorTemplate` no longer has a say in which sandbox an actor runs in, so its `sandbox_config` field goes. **Breaking:** `ActorTemplate.sandbox_config` and the `SandboxConfig` message are removed from the Control API. ate-api-server rejects unknown fields, so manifests or clients still sending `sandboxConfig` fail with `InvalidArgument`. Stored templates are unaffected (proto bytes; the old field is carried as unknown). `kubectl ate get actortemplates` loses its `SANDBOX CLASS` column. **API** - Remove `sandbox_config` and the `SandboxConfig` message from `ActorTemplate` in `ateapi.proto`, and its validation from ate-api-server. - Remove `sandboxConfig` from `kubectl-ate` output, e2e fixtures, demo and benchmark `ActorTemplate` manifests (sweperf included), and the docs. **ateapi** - `sandbox_assets` on Run/Restore is unchanged on the wire; ateapi now resolves it from the assigned worker's pool instead of the template: `WorkerAssignment` → `WorkerPool` → the controller-confirmed `SandboxConfig` (`WorkerPool.ResolvedSandboxConfig()`, new) → default version. `ActorWorkflow` takes a `WorkerPoolLister` for this; the `sandboxconfigs` RBAC stays. Marked TODO: this lookup, both listers and the RBAC go once atelet resolves the version from `sandbox_runtime` and `sandbox_assets` is dropped (the follow-up noted in agent-substrate#2755). - Scheduler: `Constraints.SandboxClass` is no longer taken from the template. It is the class of the snapshot the actor restores from (local, else latest durable, golden tags included), empty on a cold boot — so a worker of another class is never a candidate, not even as the volumes-only fallback when `RequiredRuntime` is relaxed. A cold-booting actor gets the class of the pools its `workerSelector` matches. - Repointing `actor_template` on a SUSPENDED actor no longer requires the same SandboxConfig; the snapshot's class and runtime checks on resume cover compatibility. - Metrics: `ate.sandbox.class` on lifecycle and scheduler operations is derived from the actor's assignment or snapshot runtime rather than the template, and omitted when neither is known yet; registry updated. **Behavior change to flag:** a template with no `workerSelector` whose actor has no snapshot can be placed on a pool of any class. Previously `sandboxConfig.sandboxClass` alone pinned the class; now the selector has to. **Tests** - `TestResolveSandboxAssets` covers pool-based resolution (resolved, unresolved pool, missing pool, missing config, no default version). - Scheduler `Applies` cases for the snapshot-class pin; `assignWorkerAttempt` proves a microvm worker is never the fallback for a gvisor snapshot at MEMORY or VOLUMES fidelity. - Wire-request tests seed the pool lister; the "missing SandboxConfig" cases now go through the pool. - Functional tests write the pool's `status.sandboxClasses[].configRef` via `UpdateStatus` (status subresource) and wait for it in the informer. - `ReleaseWindowWorkflows` makes the worker ineligible via `workerSelector` instead of a sandbox-class mismatch. No atelet changes beyond two comments. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ent-substrate#2634) When atecontroller both pushes and leaves `:8080` on, `:8080` now also serves the `ate.workerpool.*` instruments. The bridge skips them, so each is still pushed once. Fixes agent-substrate#2402 Written with AI assistance. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Part of agent-substrate#1590 Adds an `oom_events` block to server telemetry: - `node`, `pod`: OOM kills over the run, from cAdvisor's `container_oom_events_total`. `node` counts global OOMs. `pod` counts kills inside a worker pod (actor sandboxes and the `worker` container) and is best effort: a kill whose cgroup is gone before the scrape is missed. - `worker_restarts`, `worker_oomkilled_pods`, `evicted`, `worker_pods_lost`: from the Kubernetes API, read before and after the run. `evicted` comes from `Evicted` events, as `atecontroller` deletes an evicted pod at once. Other changes: - `monitoring.yaml` keeps `container_oom_events_total`. - The runner's `benchmark-workloads` Role can also list events. - The README explains what each field does and doesn't catch. Live check (gVisor, 3m): - Actor limit 256Mi, 2 users: `pod = 6`, matching 6 kernel `oom-kill` lines, one per actor sandbox. Other fields 0; the worker stayed up. - No actor limit, 8 users: the kernel killed the whole worker twice at the node-allocatable limit (`oom_memcg=/kubepods.slice`), and the kubelet evicted it. `evicted = 7` (1 eviction plus 6 replacement pods the kubelet refused to start), `worker_pods_lost = 1`, `node = pod = 0`, as documented. - 80 measurements in the jsonl row. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Implements the benchmark part of the [task goodput design](https://docs.google.com/document/d/1wcO8150RH9bjHulYaAvyW0IrbX2raXCREDYOurQavq4/edit?resourcekey=0-COU9pclBVTxrcs4hGnyI7g&tab=t.0): sweperf logs facts per task, and `tools/goodput` (agent-substrate#2526) decides good/bad from them. The benchmark judges nothing. **New log lines** - `sweperf task record`, one per finished task: `v` (1), `user`, `actor`, `template`, `cycles`, `steps`, `task_start`, `end`, `startup_ms` (CreateActor until `/status`), `cel_ms`, `wall_ms`, `think_ms`, `resume_max_ms`, `ttfe_max_ms`, `retries`, `failed`, `fail_step`, `fail_cycle`. - `Completed sweperf cycle` gains `cel_ms`, `wall_ms`, `ok`, `resume_ms`, `ttfe_ms`. - `ResumeActor failed in step` gains `wall_ms`; `sweperf on_start failed` gains `user`. - `sweperf shutdown started` marks where worker teardown begins (in-flight cycles are canceled without logging a task record, and startup failures logged after it belong to the teardown). **Other changes** - `tracedCallRecord` and `resume` also return the latency they record. - The README documents every line and field. **Example** (from a test run): ``` "msg": "sweperf task record", "v": 1, "user": 184, "template": "sweperf-astropy-7336", "cycles": 4, "steps": 21, "startup_ms": 335.2, "cel_ms": 11150.7, "wall_ms": 14229.2, "think_ms": 771.5, "resume_max_ms": 396.9, "ttfe_max_ms": 406.3, "retries": 0, "failed": false, "fail_step": "", "fail_cycle": 0 ``` Live check (gVisor and microVM, 5 users, 5m): task records = finished tasks = TaskCEL count; 3 SuspendActor per 4-cycle task; cycle and task times add up. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…lans it (agent-substrate#2741) > **Rebased onto main after agent-substrate#2731 merged.** Three commits: the change, then two rounds of review fixes kept separate so each thread's change is visible. Since agent-substrate#2660 the syncer fetches a gVisor version's runsc to probe its CPU features; that fetch now goes through the same asset cache and is bounded by the prewarmer's attempt timeout. Step 4 of agent-substrate#2675: the default runtime of each worker on the node is prewarmed by the syncer that plans the worker, through the runtime index, and the cluster-wide prewarmer goes away. **Before.** `sandbox_prewarm.go` watched every SandboxConfig in the cluster, queued by config name, and fetched each config's default version, gated only by whether the node had `/dev/kvm`. Most of what it fetched no worker on the node would ever run, and nothing related it to what the node's workers advertise. **After.** The worker sandbox syncer already plans each local worker pod, registers its runtimes in the index, and writes its versions file. It now also hands the worker's default runtime hash to a `runtimePrewarmer` right after registering the worker's hashes in the index, before the build probes the versions: the build fetches and probes every restorable version serially, longest on a fresh node, and the file it ends with is what lets the worker take its first actor, so the probes are the prewarm's head start and the file never waits on a download. The prewarmer's queue is keyed by hash, and the prewarmer keeps the hashes whose prewarm succeeded and skips them, so the workers of one pool, every re-plan and every retry of a sync cost one prewarm of a runtime, not one per pod; a failed prewarm is queued again. Its single worker takes the record from the index, stages it into a scratch directory under the base path through the same `ensureSandboxAssets` Run uses, removes the scratch directory, and pulls the pause image alongside. Eight retries with exponential backoff, a ten-minute bound per attempt (a field on the prewarmer, like the hook a test uses to wait for an attempt), and an unknown hash is dropped with an error log since the syncer registers before it enqueues. The syncer's runsc probe keeps its own five-minute backstop rather than borrowing the prewarmer's. Prewarm stays a latency optimization: every failure is logged and left to the on-demand fetch. **Removed.** `sandbox_prewarm.go`, its tests, and `microvmNodeCapable`, which only it used. Locality comes from the pods on the node: a micro-VM pod only lands where it can run. **Scope.** Only default runtimes are prewarmed. The first restore of a non-default version a worker advertises pays its download. **Tests.** `TestRuntimePrewarmProcess` (retry until the cap, then forget; unknown hash forgotten at once), `TestRuntimePrewarmFillsBothCaches` (asset in the cache, scratch dir empty, pause image in the image cache), `TestRuntimePrewarmAssetFailureStillPullsImage`, `TestRuntimePrewarmAttemptTimeout`, `TestSyncPodPrewarmsDefaultRuntime` (exactly the default hash of a multi-version config, handed over before the build fetches for a probe, and still handed over when the build then fails), `TestRuntimePrewarmOnce` (a prewarmed hash is skipped by later enqueues, a failed one is queued again), and `TestDefaultRuntimePrewarmThroughSyncer`, the end-to-end run through fake clientsets and informers that the old download test did. The old enqueue-filter, class-gate, and KVM tests go with the behavior they covered. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…substrate#2791) Failures in the CI step logs have gotten hard to find, the unit test step alone is ~35k lines for a single failure. `junittool report` lists each failed test with just its t.Error / t.Fatal output in the job summary and as annotations. - go test only marks t.Error output in -json without -v, so the gotestsum runs drop -v - the unit test logs print one line per package plus failed test output, the full output is still in the uploaded json - the unit test JUnit files are registered now, so verify actually checks them NOTE: There are some off the shelf tools, but we already had one, so tests aside this isn't much code and avoids taking another third party action dependency, while keeping our existing junit collection etc. It also folds a parent test into its failed subtests, names the tests that never finished on a timeout, and points panic annotations at the right line. Tested locally with sample failures, against the artifacts from a failed main run, and on this PR with a since-dropped commit of failing unit tests. AI-assisted. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…#2740) The system resolver is not always set up to map `localhost` to both IPv4 and IPv6 loopback addresses. For example, this is not the case on stock Ubuntu machines. Because of this TestEnvoyDrainerReachesIPv6OnlyAdmin fails 100% of the time on stock Ubuntu machines.
Checks in the [version skew policy](https://docs.google.com/document/d/11k821WaxAKNhDBvz6_IsiKmp9vZNOJznFrLA34sINfY/edit?usp=sharing) as `docs/version-skew-policy.md`. docs/upgrade.md now links the policy. Because the policy keeps the worker pod template the same within a major, step 4b and its rollback no longer wait for a second worker roll. Fixes agent-substrate#1185
…2795) A pause records no upload location for its snapshot, but atelet uploads it in the background to the location the pause gave it. The next pause's Checkpoint response names that upload, and ate-api-server refused it, leaving the actor SUSPENDING. ate-api-server now records the upload as COMPLETED when its location is the one ate-api-server built for that snapshot. Part of agent-substrate#2069 and agent-substrate#2571 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…ead (agent-substrate#2790) The goal of this PR is to deprecate `PauseActor`. `SuspendActor` with `DURABILITY_MODE_ASYNCHRONOUS` replaces it. The PR has 4 commits: 1. **ateapi: honor SuspendOptions.mode in SuspendActor**: `SuspendActor` now reads `options.mode`. `DURABILITY_MODE_ASYNCHRONOUS` runs the async suspend that `PauseActor` runs, which keeps the snapshot on the worker's node. Any other value, including an unset mode, still runs the sync suspend. `ActorWorkflow.SuspendActor` now takes the mode directly and replaces the separate `PauseActor` workflow entry point. The `PauseActor` RPC still exists and calls the async mode. 2. **Move PauseActor callers to an async SuspendActor**: `kubectl ate pause actor`, the boomer load generators, and the ateapi functional tests now call `SuspendActor` with `DURABILITY_MODE_ASYNCHRONOUS`. The boomer request names (`PauseActor`, `PauseActorRecovery`, `PauseActorWarmup`) stay the same so existing dashboards and result comparisons still line up. 3. **e2e: pause actors with an async SuspendActor**: the e2e demo suites make the same switch. 4. **ateapi: deprecate PauseActor**: marks the RPC `deprecated` in the proto, so staticcheck flags any new caller before `PauseActor` is removed. The two functional tests that still exercise the `PauseActor` handler are exempt from the check. Part of agent-substrate#798 - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
Atelet uploads an actor's latest local snapshot in the background once the oldest snapshot not yet durable has waited --snapshot-upload-delay. Checkpoint, Run, and Restore wait only for a detach, not an upload. Detach waits for an upload in progress rather than repeating it. Terminate cancels the upload without waiting for it. A checkpoint deletes older local snapshots only after the new one is committed, so a failed checkpoint leaves the previous snapshot in place. A checkpoint keeps the snapshot a background upload is still reading, which can be older than the latest snapshot. Part of agent-substrate#2069 and agent-substrate#2571 ### Note to reviewers This PR doesn't update ateapi of the asynchronous upload success. So durable snapshots won't get cleaned up until the actor is deleted. We are addressing this in a followup PR. ### Benchmark Results (glutton, 3m, 1 user, 512Mi RAM, 32Mi churn) Results show we get the same performance as pause, with the benefit of automatically uploading checkpoints to object storage. Out of the 73 local checkpoints, 10 were uploaded to GCS in the background (roughly 1 in 7), and the other 63 were superseded locally before the 15s upload delay fired. | Run | `PauseActor` p50 / p90 / p99 | `ResumeActor` p50 / p90 / p99 | Background Uploads | | :--- | :--- | :--- | :--- | | Baseline (`ac296df67`) | 530 / 1800 / 2600 ms | 820 / 890 / 920 ms | 0 | | Uploads + `snapshot-send-obj` (`cee8dc751`, **15s delay**) | **500 / 540 / 560 ms** | **880 / 940 / 960 ms** | **10** (`2.2–3.0s` each) | - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…trate#2802) Background uploads and detaches shared one pool of workers, so slow uploads could occupy every worker and keep a due detach waiting, leaving the actor's local snapshots on the node. Uploads and detaches now have separate workers, so neither waits behind the other. An actor still runs at most one upload or detach at a time. The new --snapshot-upload-workers and --snapshot-detach-workers flags control the number of workers Part of agent-substrate#2069 and agent-substrate#2571 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…isor (agent-substrate#2673) gVisor ActorTemplates can now set `snapshotConfig.preferredFidelity: SNAPSHOT_FIDELITY_ROOTFS`. Pause and Suspend keep each container's rootfs writes and the DurableDir volumes, without process memory. Resume cold-boots with them back in place. ateworker-gvisor saves container rootfs overlays and DurableDir volumes as two filesystem checkpoint bundles, `rootfs/` and `fs/`: - ROOTFS checkpoints write both bundles with `runsc fscheckpoint`. - MEMORY checkpoints split both bundles out of the process checkpoint, so the rootfs is stored apart from the other state. - Restore passes `--fs-restore-image-path` for each bundle the requested fidelity uses. VOLUMES uses only `fs/`. - `CreateActorTemplate` no longer limits ROOTFS to `sandboxClass: microvm`, and the ROOTFS e2e cases now also run on gVisor. - Removes outdated text that limited gVisor templates to one DurableDir. #### Breaking change - The default gVisor SandboxConfig moves to a newer runsc (`v4`). As with any gVisor version change, existing MEMORY and golden snapshots may not resume at full fidelity. Re-create golden snapshots after upgrading. - ateworker-gvisor now needs a runsc that supports multiple filesystem checkpoint bundles. A SandboxConfig pinned to an older runsc fails MEMORY checkpoints of actors that have a DurableDir. - New gVisor snapshots contain `rootfs/` files, which older atelet and object-store-plugin builds reject. Upgrade atelet and the plugin before, or together with, ateworker-gvisor. Don't roll atelet back once new snapshots exist. Fixes agent-substrate#2435. Completes the gVisor phase of agent-substrate#2447. - [x] Unit tests pass (ateworker-gvisor, internal/resources, ateapi apivalidation) - [x] gVisor e2e passes with the final nightly - [x] Documentation updated
… paused actor's snapshot in the background (agent-substrate#2804) SUSPENDED is now a single state. SuspendActor returns a SUSPENDED actor as is. It no longer uploads a paused actor's node-local snapshot through atelet's PromoteSnapshot. atelet uploads the snapshot in the background. - Both suspend modes record a pending durable copy of the snapshot. - Tagging an actor whose latest snapshot is not uploaded yet still fails. The error message no longer suggests suspending it. #### Breaking change A synchronous SuspendActor on a paused actor no longer waits for the snapshot upload. It returns the actor right away. Callers that need the durable copy should wait for the snapshot's durable copy to complete. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
…e#2721) **Fix the status code returned for CONNECT** - Envoy (`egress/request.go`): refuse a CONNECT on both inner legs with `403 CONNECT egress is not supported`, before identity and policy, like the WebSocket refusal. atunnel's outer CONNECT is handled elsewhere and is unaffected. **Tests for CONNECT** - Unit: `TestRequestLegDeniesConnect` (both legs, with and without a policy). - E2E: `TestActorEgressConnect` (cleartext, proxy on 3128) and `TestActorEgressSecureConnect` (inside TLS on 443). Each sends the CONNECT to an origin that answers any CONNECT 200, so the 403 can't be the origin's, and to the gateway's own Service. - Support: egress demo `POST /connect`; testserver `http --accept-connect`.
…strate#2793) One commit; the review's rounds were squashed into it, and the threads keep their history. `TestWorkerSandboxSyncerJitter`, `TestWorkerSandboxSyncerRegistersAheadOfJitter` and `TestWorkerSandboxSyncerExistingDirs` asserted that an event queued nothing right away, slept past a short real delay, and asserted the delayed pods had arrived. The delay is a random draw and the sleep was real time, so under the race detector the first drain could already find a pod, and a slow runner could miss one after the sleep. The first test failed on a local run. The tests now run under `testing/synctest`. The queue is the ordinary one `newWorkerSandboxSyncer` builds and its timers run on the bubble's clock, so a sleep of the production `syncMaxJitter` or `orphanMinAge` fires every delay without waiting, and `synctest.Wait()` settles the queue before each assertion. The assertions are the original ones: exactly what was queued at once, and exactly what arrived once the delay had passed. The existing-dirs test stamps directory ages from the bubble's clock. `newTestSyncer` shuts its queue down with the test, which the converted tests need and which stops the other syncer tests from leaking the queue's goroutine and ticker. Production code is untouched: `enqueuePool` draws its delay with `rand.N` over `[0, maxJitter)` as before. A draw of exactly zero would queue a pod at once and fail the early assertion; at thirty seconds that is about one in three times ten to the tenth per delayed pod, a property of the function rather than of the tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
…te#2765) fixes agent-substrate#2763 gVisor now writes resolv.conf into the rootfs like micro-VM, so actors can edit it on both and edits survive suspend/resume. This drops the per-actor bind source and its cleanup. this aligns behavior to kubernetes / docker / ateworker-microvm tested on kind with both sandbox classes (identity + `TestActorDNSRequiresPolicy`). AI-assisted.
…trate#2798) #### Breaking change A gvisor SandboxConfig must give every architecture the `gvisor` release tarball asset. A bare `runsc` asset is now rejected. Replace it with the tarball. Fixes agent-substrate#2697 Removes the bare `runsc` asset from atelet and the admission policy, and moves the atelet tests onto the gVisor release tarball. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Have Checkpoint and PromoteSnapshot reject a snapshot whose generation is below the actor's newest tracked snapshot on the node, or equal to it under a different snapshot UUID, with FailedPrecondition before doing any work. Part of agent-substrate#2069 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
> The PR title becomes the release note. Write it for users: what changed for them, not how the code changed. SuspendActor without `options.mode` now runs an asynchronous suspend, as the `SuspendOptions` proto comment already documents: the snapshot stays on the worker's node and atelet uploads it in the background. Callers that need the snapshot in object storage before the call returns pass `DURABILITY_MODE_SYNCHRONOUS`. In-repo callers that relied on the old default now set the mode explicitly: - The template reconciler suspends golden actors synchronously, since it tags the snapshot right after. - `kubectl ate suspend actor` takes `--mode=async|sync` and defaults to `async`, matching the API. Pass `--mode=sync` for a suspend that waits for object storage. - A worker-requested suspend keeps the default, now asynchronous. No worker calls it yet, and a TODO tracks letting the Worker choose the mode. - The boomer and locust "suspend" lifecycle modes are synchronous, so they keep measuring a durable suspend. - Functional and e2e tests keep their existing behavior: they were written expecting SuspendActor to be synchronous, so they ask for a synchronous suspend. Most of them also depend on it, because they tag, revert, repoint the template, or migrate right after suspending. #### Breaking change A SuspendActor call that leaves `options.mode` unset no longer waits for the snapshot to reach object storage. Callers that tag the snapshot (`CreateTag`), move the actor to another node, or otherwise need a durable snapshot right after suspending must set `options.mode` to `DURABILITY_MODE_SYNCHRONOUS`. Until the background upload lands, `CreateTag` fails with `FAILED_PRECONDITION`. `kubectl ate suspend actor` without `--mode` no longer waits for the snapshot to reach object storage either; scripts that tag or migrate right after it need `--mode=sync`. Part of agent-substrate#2069 and agent-substrate#2571 - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
…nt-substrate#2810) atelet uploads and detaches an actor's snapshots on its own timers, and ate-api-server learned of them only from the state a later Checkpoint or PromoteSnapshot returned. atelet now calls WorkerService.ReportActorSnapshotState with the actor's snapshot state after a background upload makes a newer snapshot durable, and after a background detach deletes the actor's local snapshots. A report that fails is retried with a backoff until ate-api-server acknowledges it, refuses it with NotFound, FailedPrecondition, InvalidArgument, or Unimplemented, or the retry window expires. A pending report is kept in the actor's snapshot state file, so it is still sent after an atelet restart. A Checkpoint, PromoteSnapshot, or Detach call returns the latest state in its response, so it cancels any pending report and queues none of its own. Part of agent-substrate#2069 and agent-substrate#2571 - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
PauseActor only forwarded to an asynchronous SuspendActor. Since agent-substrate#2805 made async the SuspendActor default, the RPC is redundant, and every client in the repository already calls SuspendActor. - Remove the `PauseActor` RPC and its request/response messages, the handler, request validation, the authorization rule, and the apitool exemptions for its messages. - Drop the functional test cases that called the RPC. - Rename tests named after it to `TestSuspendActorAsync_*`, and `workflow_pause_test.go` to `workflow_suspend_async_test.go`. - Rename the benchmark request names `PauseActor`, `PauseActorRecovery` and `PauseActorWarmup` to `SuspendActorAsync*`, along with the Grafana dashboard queries and the benchmarking README. Dashboards won't line up with runs from before the rename. - docs/authorization.md listed a `can_pause` relation the model never defined; it now lists only SuspendActor and RevertActor. ## Breaking change - The `PauseActor` RPC is gone. Clients calling it get `Unimplemented`. Call `SuspendActor` instead: it suspends asynchronously by default, which is what `PauseActor` did. - Benchmark request names `PauseActor`, `PauseActorRecovery` and `PauseActorWarmup` are now `SuspendActorAsync`, `SuspendActorAsyncRecovery` and `SuspendActorAsyncWarmup`. Saved queries or alerts on the old names need updating.
…trate#2815) `atelet` now emits metrics for background snapshot detach duration, upload and detach worker delay, and undurable local snapshots waiting on a node. These instruments expose whether background snapshot workers are falling behind and whether a node failure would lose old actor state. - `atelet.snapshot.upload.duration` (histogram): already existing metric, measures hte time spent uploading an actor's snapshot to durable storage, labeled with `ate.snapshot.fidelity` and `error.type` on failure. - `atelet.snapshot.detach.duration` (histogram): time spent deleting an actor's local snapshots during a background detach, excluding any upload that ran first, labeled with `ate.snapshot.fidelity` and `error.type` on failure. - `atelet.snapshot.upload.overdue.duration` and `atelet.snapshot.detach.overdue.duration` (histograms): time between when a background upload or detach came due and when a worker started it. Each stays near zero while a worker is free and grows when the node's workers cannot keep up. - `atelet.snapshot.upload.overdue.actors` (up/down counter): actors on the node with an undurable local snapshot past `--snapshot-upload-delay`, bucketed by `ate.snapshot.upload_overdue_band` (`under_1m`, `1m_to_5m`, `5m_to_15m`, `over_15m`). An actor stays in the count until its upload completes or the tracker drops it, so stuck or retrying uploads move into higher bands. - `atelet.snapshot.undurable.age.max` (gauge): age of the oldest undurable local snapshot on the node, or `0` when all local snapshots are durable. A healthy node stays below `--snapshot-upload-delay` plus one upload. Part of agent-substrate#2069 and agent-substrate#2571 - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Closes agent-substrate#2533. Move Golden actors and tags from the shared `ate-golden` atespace into their ActorTemplate's atespace, named after the template UID. Golden resources use the tenant's ordinary authorization, lifecycle policies, and storage grants. Remove the special handling for the dedicated system atespace. Golden capture follows the template's `preferredFidelity`. MEMORY preserves process state. VOLUMES and ROOTFS restore persisted data and start new processes. Durable snapshot metrics use `ate.snapshot.kind=latest`, including Golden snapshots. Tests cover tenant storage ownership, snapshot inheritance, restore, and cleanup. The VOLUMES Golden fixture matches the template's fidelity. ## Validation Validated `204b8fb3`, rebased onto main `7abc3172`, on Linux with Go 1.27.2. The tested source matches the PR's 13,851 tracked files. - Passed `go test -race -count=1 ./...`: 166 packages passed, including PostgreSQL/envtest functional and authorization tests. - Passed builds of ateapi, atelet, and kubectl-ate, race-enabled compilation of the demo E2E binary, and `tools/apitool` race tests. Local validation and actor-attribute race tests also passed. - Passed `hack/verify-all.sh`, including code generation, lint, modules, licenses, metrics, migrations, and formatting. Regeneration left the source unchanged. - Redeployed the dedicated kind cluster and passed real gVisor `TestGoldenSnapshotLifecycle` and `TestDurableDirLifecycle`: Golden MEMORY/VOLUMES creation, tenant storage ownership, inheritance, resume, template cleanup, and durable-directory lifecycle. JUnit reports zero failures and two ROOTFS skips. All Go compilation and tests ran as a non-root user. The runtime copy uses a host-only Linux 5.14 OverlayFS fallback; source checks use the unmodified PR source. Micro-VM E2E and root-gated tests were not run. ## Operator notes - Follow the pre-1.0 reinstall procedure in the [upgrade guide](https://github.com/agent-substrate/substrate/blob/main/docs/upgrade.md). Existing `ate-golden` resources may still own snapshots borrowed by actors. - Object storage grants must cover actor and tag prefixes in each template's atespace. - Golden actors appear in ordinary Actor lists. Lifecycle and ingress operations can interrupt a Golden build. Deleting a published Golden tag invalidates the template's recorded reference. - Update dashboards that select `ate.snapshot.kind=golden`. The `latest` series now includes Golden and other durable snapshots. AI assistance: Codex helped implement and validate this change.
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
force-pushed
the
sync/upstream-main-20261018
branch
from
October 18, 2026 13:14
15e28af to
8cdb696
Compare
EItanya
marked this pull request as ready for review
October 18, 2026 13:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge 52 upstream commits through
d2b97509984c4dc2df75f128a05f221a7cf98107, including asynchronous snapshot uploads and reporting, worker-selected sandbox runtimes, gVisor filesystem checkpoints, JWT discovery, and networking fixes. One signed-off merge commit preserves both histories and includes the fork compatibility fixes.The conflict resolutions retain the fork's workflow deadlines, worker-cache refresh, local atelet transport opt-in, and credential test fixtures. The Helm chart gains the IdP deployment and ingress policy, a shared actor JWT issuer for the API and discovery service, atelet's new port and ComputeClass toleration, and the updated gVisor assets and CRDs. Agentgateway remains the fork's dataplane. CONNECT rejection tests use the existing dataplane adapter: Envoy returns 403, while agentgateway returns 405; both must reject the tunnel with their exact denial response.
Breaking change
Upgrade clients and components together. Replace
PauseActorwithSuspendActor; its default is now asynchronous, so requestoptions.mode: DURABILITY_MODE_SYNCHRONOUSwhen checkpointing a running actor must finish uploading before returning. RemovesandboxConfigfrom actor templates and select runtimes through WorkerPools. Custom gVisor configs need the release tarball and the new checkpoint support. Update atelet firewall rules from TCP 8085 to 21998. The chart README covers the configuration changes.Validation: all five CI checks passed before folding the follow-up commits into the merge:
The final merge
8cdb69636has exactly the same tree (fe326bcec8a5fcf180a5ab4bdc237ae12617735f) as the tested head15e28af82. GitHub is rerunning checks for the rewritten commit. Locally, 25 Helm tests, chart lint, manifest rendering, CRD parity, and focused Go race tests also passed.Merge this PR with a merge commit to retain upstream ancestry.