Repository navigation
Sync upstream API authorization and worker runtime updates - #85
Merged
Merged
Conversation
…ims (agent-substrate#2671) Part of agent-substrate#777 The objectstore control plugin will verify actor JWTs minted by ate-api-server. No component serves the JWKS yet, ate-idp-server has not landed, so the verifier needs keys from somewhere other than discovery, and it must not be handed the signing keys to get them. `oidcjwt.NewVerifier` now takes a `KeySource`. `HTTPKeySource` is the existing discovery fetch. `FileKeySource` reads a JWK set from a file with the same parser. The `actor-id-jwt-pool` Secret carries that set under `jwks` next to `pool`: ate-setup, `make-jwt-pool`, and `add-jwt-key` write both in one update, and ate-setup brings `jwks` in step with the pool on every deploy, for a Secret from before the key existed or one rewritten by an older tool. The sidecar will mount only the `jwks` item, in the manifests PR. `Claims` gains `SubstrateClaims`, decoded only after the signature check: `Kind`, actor or worker-for-actor, and the claim it names, with a token carrying both refused. Decoding them is opt-in through `WithSubstrateClaims`, which only the actor-id verifier sets, so tokens from other issuers verify as on main whatever they carry under those names. No behavior change for ate-api-server, its call site wraps the discovery client in `HTTPKeySource`.
…odes (agent-substrate#2495) Dense nodes starve the kubelet's heartbeat; after the 300 s toleration the taint manager evicts the node's pods. - `ATE_KIND_RESERVED_CPU=<cpus>` adds `systemReserved.cpu` and `kubeReserved.cpu` to the kind `KubeletConfiguration` patch. Unset, nothing changes. Not verified at the density that triggered it. `shellcheck` clean.
…nt-substrate#2622) Part of Actor Lifecycle v2 agent-substrate#2069. Second PR for agent-substrate#2570, after agent-substrate#2612 added the API. This serves `WorkerService.ReportActorSnapshotState`. With asynchronous suspend, atelet uploads a suspended actor's snapshot and later detaches the actor from its node, on its own timers. The report is how ate-api-server learns about both. Nothing sends reports yet; that comes with atelet's asynchronous upload work. A report lists the snapshots atelet holds for an actor on its node. ate-api-server acts on two things in it: - **`latest_durable` means an upload finished.** If it names a recorded snapshot with the same uuid and upload URI, that snapshot becomes `COMPLETED`, with its manifest's fidelity. - **A report with neither `latest_local` nor `uploading` means atelet deleted its copies (a detach).** It covers this node's snapshots up to the reported `latest_durable` generation. Those localities are cleared, and a suspended actor with no snapshot left on the node is released from it. ate-api-server ignores the rest of the report. <details open> <summary>Flowchart: how a report is applied</summary> ```mermaid flowchart TD A["WorkerService.ReportActorSnapshotState(ReportActorSnapshotStateRequest)<br/>TODO: atelet does not send it yet"] --> B{"Caller presents atelet's certificate?"} B -- no --> X1["Unauthenticated or PermissionDenied"] B -- yes --> C{"Request passes validation?"} C -- no --> X2["InvalidArgument, including a report with none of<br/>latest_local, uploading, and latest_durable"] C -- yes --> E{"Actor exists, actor_uid equals Actor.metadata.uid,<br/>and Actor is on the caller's node?"} E -- no --> X4["NotFound"] E -- yes --> D{"Actor's lease free?"} D -- no --> X3["Aborted, atelet retries"] D -- yes --> E2{"Same check again, under the lease"} E2 -- no --> X4 E2 -- yes --> S{"status.state?"} S -- "CRASHED" --> X9["FailedPrecondition"] S -- "SUSPENDING, RESUMING,<br/>DELETING, REVERTING" --> X10["Aborted, atelet retries"] S -- "any other state" --> X12["Internal"] S -- "SUSPENDED or RUNNING" --> G{"latest_durable set?"} G -- no --> K{"latest_local and uploading both unset?"} G -- yes --> GF{"At or below the latest COMPLETED generation?"} GF -- yes --> K GF -- no --> GR{"Its generation is in status.snapshots?"} GR -- "no, and not newer than<br/>last_assigned_generation" --> K GR -- "no, and newer" --> X5["FailedPrecondition"] GR -- yes --> U{"Same uuid, and an upload URI is recorded?"} U -- no --> X6["FailedPrecondition"] U -- yes --> V{"Reported URI equals the recorded one?"} V -- no --> X7["InvalidArgument"] V -- yes --> H["Mark it COMPLETED"] H --> K K -- yes --> K2{"Latest snapshot on this node, up to<br/>latest_durable's generation, is durable?"} K2 -- no --> X8["FailedPrecondition, nothing recorded"] K2 -- yes --> J["Detach: clear this node's localities up to that generation.<br/>Clear assigned_node if status.state is SUSPENDED, it is this node,<br/>and no snapshot is left on this node"] K -- no --> P{"status changed?"} J --> P P -- no --> Q["Done, nothing written"] P -- yes --> R["Write the actor"] ``` </details> ## Outcomes | Report | Result | | --- | --- | | Any report for a `CRASHED` actor | `FailedPrecondition` | | Any report for a `SUSPENDING`, `RESUMING`, `DELETING`, or `REVERTING` actor | `Aborted`, retried | | `latest_durable` for a recorded snapshot with matching uuid and upload URI | That snapshot becomes `COMPLETED`, with the fidelity its manifest records | | `latest_durable` at or below the latest `COMPLETED` generation | No change | | `latest_durable` for a generation `status.snapshots` no longer has, such as one a revert discarded | No change | | `latest_durable` for an unrecorded generation newer than any assigned | `FailedPrecondition` | | `latest_durable` with another uuid, or for a snapshot with no recorded upload URI | `FailedPrecondition` | | `latest_durable` uploaded to a URI other than the recorded one | `InvalidArgument` | | None of `latest_local`, `uploading`, and `latest_durable` set | `InvalidArgument` | | Neither `latest_local` nor `uploading` set (a detach) | Clears this node's localities up to the `latest_durable` generation. Clears `assigned_node` if the actor is `SUSPENDED` there and no snapshot is left on this node. Refused with `FailedPrecondition` while the latest covered snapshot isn't durable | | A late detach, after a newer snapshot on this node | Leaves the newer snapshot and `assigned_node` alone | | A detach that's refused | Nothing is recorded, including an upload the same report carries | | A report that changes nothing | Nothing is written | ## Notes - **Nothing is pruned or deleted.** The existing workflows prune old snapshots and delete their objects at the next suspend or delete. - **A detach covers only generations up to its `latest_durable`,** so a late or repeated report can't clear a newer snapshot. Repeating any report is safe. - **A detach waits for a durable snapshot.** Clearing the locality of a snapshot that isn't durable would let the actor resume from an older one. - **A report applies as a whole or not at all.** atelet's detach uploads the latest snapshot first, so a refused detach is rare. - **The actor is checked before the lease is taken,** and again under it, so an atelet can't hold the lease of an actor on another node. - **Reports apply only to a `SUSPENDED` or `RUNNING` actor.** A transitioning actor's snapshots belong to its workflow, and a `CRASHED` actor's wait for a revert. - **Every UID or node refusal is `NotFound`,** so an atelet learns nothing about actors on other nodes. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
Nothing in `.github/workflows` or `hack/verify` ran the benchmarking Python tests. The four suites under `benchmarking/analysis`, `automation`, `locust/unit_tests` and `nighthawk-ingress/tests` only ran when someone remembered to, and the egress-tunnels benchmark that follows adds a fifth. A test nothing runs is not a test. `hack/verify/benchmarking-python.sh` gives each suite a venv from its own `requirements.txt` plus pytest and runs it from the suite directory, because the tests import their modules by bare name; pytest rather than unittest because the nighthawk-ingress tests are plain functions. Before running, it stubs `docker`, `gcloud` and `kubectl` on `PATH` to fail and points `KUBECONFIG` at nothing, so a test that escapes its mocks cannot reach a cluster or a registry with the developer's credentials. `hack/verify-all.sh` picks the script up by glob. Testing: the script passes 107 tests across the four suites locally; shellcheck, boilerplate and the rest of `hack/verify/*.sh` are green. Prepared with help from an AI coding agent; reviewed and verified by the author.
Pause and suspend are being merged into one workflow, so PauseActor and SuspendActor become one lifecycle operation. Label both as "suspend" in metrics, state-change events and crash records, and drop the "pause" value from the ate.actor.operation.name registry.
Match the actual behavior of handling unknown fields: reject if server side doesn't know, preserve if client doesn't know. Found this when iterating version skew doc. Part of agent-substrate#1185 - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…trate#2655) Replaces the stock `envoyproxy/envoy` image reference in `manifests/ate-install/atenet-router.yaml` with the `${ENVOY_DATAPLANE_IMAGE}` placeholder so both `atenet-egress` and `atenet-router` deploy the same `envoy-dataplane` image built from `cmd/dataplane/envoy/Dockerfile` (or pinned from a release via `--image-repo`). - Updates `renderAtenetRouterManifest` and `renderSystemManifests` in `cmd/ate-setup` to resolve and substitute `${ENVOY_DATAPLANE_IMAGE}`. - Caches resolved Dockerfile image references on `steps.Env` so deployments rendering multiple manifests (`atenet-router` + `atenet-egress` + system overlay) only build or pin `envoy-dataplane` once. - Extends `TestRenderAtenetEgressManifestPrebuilt` to verify `${ENVOY_DATAPLANE_IMAGE}` substitution across router, egress, and system manifests. Part of agent-substrate#1330 AI was used in the development of this change. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…tity signer (agent-substrate#2389) We should rely on the checks enforced by kube-apiserver. We don't need to double-check whether the pod exists. Upcoming e2e test changes will rely on the fact that non-Kubelet clients are allowed to create arbitary PodCertificateRequests, even if the pod or service account they reference does not exist. This will allow the test code running in Github Actions (or on the developer's machine) to authenticate to the router running within the cluster.
…ogs stale snapshot reports (agent-substrate#2698) Part of Actor Lifecycle v2 agent-substrate#2069, for agent-substrate#2570. atelet's `Checkpoint` response carries the actor's snapshot state. ate-api-server discards it today. This adds what's needed to apply it, with the same rules `ReportActorSnapshotState` uses. It also logs reports that change nothing. - **Conversion.** atelet's `SnapshotReport` nests an atelet `Snapshot`. The control plane's carries the same fields inline, with its own fidelity enum and object type. `snapshotReportFromAtelet` converts one to the other, and `fidelityFromAtelet` maps the fidelity back. - **`applyCheckpointState`** records an upload the state says finished, through `applyLatestDurable`. atelet echoes the `Snapshot` it was asked for, so the fidelity recorded is the one requested. Unlike a report, a state with no local snapshot is not a detach, since the workflow that called `Checkpoint` records where its snapshot lives. - **Stale reports are logged at Info.** A `latest_durable` older than the latest `COMPLETED` generation, or at a generation the actor no longer records, changes nothing. That's expected for a late report, so each case is logged rather than refused. A repeat of the latest isn't logged, since atelet retries reports freely. Nothing calls `applyCheckpointState` yet. A follow-up applies it in the suspend workflow. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
- [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…apshot and frees the node (agent-substrate#2725) Part of Actor Lifecycle v2 agent-substrate#2069. **API change (atelet-internal):** adds `Atelet.DetachActor(DetachActorRequest) returns (DetachActorResponse)` to `internal/proto/ateletpb`. No public API changes. The control plane needs to detach an actor from a node on demand. The first caller is resuming an actor on another node when its latest snapshot is only local: the detach uploads that snapshot first, so the actor can restore from object storage anywhere. `DetachActor` makes the actor's latest local snapshot durable, then deletes the actor's local snapshots on the node. atelet also runs this detach on its own once a suspended actor has stayed on the node for its detach delay. The RPC: - does nothing when atelet holds no local snapshot for the actor; - waits for an operation already in progress on the actor, such as an upload or a detach; - returns `ABORTED` while a Run, Restore, or Checkpoint of the actor is in progress or the actor runs on the node; - returns the actor's `ActorSnapshotState`, with `latest_local` unset and `latest_durable` set to its latest snapshot, if atelet knows one. That's the shape of a detach report, so the control plane can apply it with the `ReportActorSnapshotState` rules. The request names the actor by atespace, name, and UID. Golden actors aren't rejected, since an `IMMEDIATE` checkpoint keeps a local copy too. This PR adds the API and its validation only. atelet embeds `UnimplementedAteletServer`, so it returns `UNIMPLEMENTED` until the snapshot tracker's detach lands. The handler then calls the tracker's detach and returns its state. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…CS tokens (agent-substrate#2672) Part of agent-substrate#777 Two packages the control plugin's credential exchange will use. Nothing calls them yet, the endpoint comes in a follow-up PR. `actorscope.FromActor` reads `status.snapshots` and returns the prefixes an actor may touch. Every durable snapshot URI names the storage location it sits under, and for each location the actor gets read-write on its own prefix there, built with the same helper ate-api-server places snapshots with, so the grant exists from the actor's first generation on. Each completed TAG-owned generation adds read-only on the tag's prefix, taken as is. This parses URIs until `Actor.status` carries a base snapshot URI, as agreed with Julian. An owner that is neither ACTOR nor TAG is an error, so a new kind is never silently dropped. The status is otherwise taken as ate-api-server wrote it. `gcsboundary.FromGrants` turns grants into Credential Access Boundary rules, one per prefix, `objectAdmin` for read-write and `objectViewer` for read-only. `gcsissuer` exchanges the pod's credential at STS through `golang.org/x/oauth2/google/downscope` and caches tokens by actor UID in a bounded LRU. Each entry keeps the grants it was minted for and is served only when they cover the request, so the grants narrowing after the first suspend is a hit and a repoint to a new location mints. That means the exchange derives the grants, `GetActor` included, before it consults the cache. A cached token is served only while it has twenty minutes left, a minted token shorter than that is returned but not cached, and a forced request skips the cache for the node plugin's retry after a 403. The downscoped token expires with the source credential. On GKE the metadata server serves a cached source token until shortly before it expires, so for a stretch at the end of each hour every exchange reaches STS and returns a token shorter than the cache floor, which is handed out but not cached. Minting the source directly is a follow-up. Verified on a GKE cluster as the ate-api-server service account: a token with these rules reads and writes inside its prefixes and gets 403 outside them, and its expiry equals the source token's. `golang.org/x/oauth2` moves from an indirect to a direct dependency and `google/downscope` is vendored.
…e#2021) Initially, it hosts nothing but logging config. A follow-on PR will move the authentication config into the unified config.
…agent-substrate#2678) Part of agent-substrate#2078 (Envoy dataplane, @EItanya @howardjohn for agw) The router accepted any podidentity cert in the trust domain as atunnel, so any pod in the cluster could pose as a worker. It now requires the exact SPIFFE ID of the worker each request is routed to. - atecontroller's syncer records the worker pod's SPIFFE ID on the Worker (new `worker_spiffe_id`, built from the pod's namespace and ServiceAccount), and ateapi copies it into the actor's worker assignment. ateapi rejects a value that isn't a valid SPIFFE ID. - ext_proc passes that ID to Envoy as is and refuses to route without it. Envoy requires that exact SAN, on both the :443 and CONNECT paths. `--upstream-spiffe-prefix` is removed; the cluster still checks the trust domain. - The Worker carries the SPIFFE ID rather than the ServiceAccount, so the router doesn't need to know how the ID is built. A different ID scheme only changes atecontroller. - WorkerPools that don't set serviceAccountName now run as an `ateworker` ServiceAccount the controller creates, instead of `default`, so the worker's ID isn't shared with every other pod in the namespace. Remaining gap: another pod running as the same ServiceAccount in the same namespace can still pose as the worker. Closing it needs a signer that puts the pod in the worker's SPIFFE ID; atecontroller would then record that ID, with no router change. #### Breaking change - The router rejects workers registered before this change until their pods are replaced. Pools on the default ServiceAccount roll automatically; pools that set `spec.template.serviceAccountName` need a restart. Upgrade atecontroller and ate-api-server before atenet-router. - Workers now run as an `ateworker` ServiceAccount by default instead of `default`. RBAC, Workload Identity bindings and imagePullSecrets attached to `default` no longer apply to them; set `serviceAccountName: default` or move those to `ateworker`. - atenet-router no longer has `--upstream-spiffe-prefix`. - atecontroller now needs create and patch on serviceaccounts (included in the generated role). > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…t-substrate#2693) Fixes agent-substrate#2685 The MITM and cleartext chains write one JSON access-log line per actor request. At 2,500 req/s on one replica that is 1.2 MB/s, and with the kubelet's default rotation only about 45 s of history survives, so a failure from a minute ago is already gone. Both chains now log a request only when its status is 400 or above, when it carries a response flag, or as a 1% sample of the rest. At the same rate the log drops to 25 lines/s and the rotation window is over an hour, with every failure still recorded. The threshold and the sample rate are runtime keys, `access_log.min_status` and `access_log.sample`, so an operator can restore full logging at the admin port without a rollout. The log no longer counts successful requests per host; `docs/metrics/substrate.yaml` records that. Testing: the manifest test compares each per-request log's filter against the expected YAML and asserts no other access log carries one. `go test -race ./cmd/atenet/...` and `hack/verify/*.sh` pass. Prepared with help from an AI coding agent; reviewed and verified by the author.
…gent-substrate#2652) Expand the top-level README messaging to give security equal billing with performance. Add a dedicated Security Model section highlighting VM-grade sandboxing, per-agent SPIFFE identity, zero ambient credentials, default-deny networking, and pluggable edge policy. Also some word cleanup on the performance claims. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
… is assigned to (agent-substrate#2702) MintWorkerActorCertificate authenticated the atelet but ignored its node, and atelet did not say which worker was asking. Any node, or any worker on the actor's node, could mint the actor's certificate from its name and UID. This applies the same check MintWorkerActorJwt got in agent-substrate#2668. - ate-api-server mints only when the caller's node hosts the actor and the requesting worker is the one assigned to it. Otherwise it returns NotFound, before the stale-UID check. - `MintWorkerActorCertificateRequest` gains a required `worker` field. atelet fills it from the caller's certificate, as it does for RequestActorSuspend. A paused actor has no worker, so it gets no certificate. - `worker` is required with no fallback; upgrading atelet before ate-api-server, as docs/upgrade.md already orders it, covers the skew. Tested with unit tests for the node, worker, paused, release/rebook and stale-UID cases, plus controlapi tests against a real PostgreSQL store. `go test -race ./cmd/ateapi/internal/... ./cmd/atelet/...` passes.
…hestrator (agent-substrate#2717) The actor CPU limit was never piped through the benchmark deploy path. Now that every ActorTemplate must declare one, the templates default to 1 vCPU, which caps how many actors a worker hosts; this makes it configurable the same way actor memory already is. Fixes #<issue_number_goes_here> > It's a good idea to open an issue first for discussion. - [ X ] Tests pass - [ X ] Appropriate changes to documentation are included in the PR
…strate#2323) Each time an actor finishes its trajectory, sweperf now deletes it and starts a new actor to run the trajectory again. Before, the same actor replayed the trajectory over and over. The last cycle is not suspended: the actor is deleted straight from running with DeleteActor(AnyState). A failed delete is retried once with a fresh timeout, then logged with the actor name. The atespace is created once per worker, and again only if CreateActor reports it missing (FailedPrecondition). Liveness on a new actor is polled at --sweperf-poll-interval-ms, and a failed start is logged with its actor and template. Shutdown: - It logs "sweperf shutdown started", blocks new iterations and cancels the ones in flight, including slow liveness polls. - It waits (bounded) for in-flight iterations, then deletes every remaining actor directly within the pod's termination grace period. - Calls cut short by this cancellation record no failure rows, and no task rows are recorded once shutdown has begun, so the end of a run will not show up as a burst of failures. This also fixes a liveness failure decrementing locust_users twice. Metric changes (README updated): - Cycle 1 runs on a freshly started actor, so its no-op resume is not recorded: one fewer ResumeActor sample per task. - A task of n cycles records n-1 SuspendActor samples. - DeleteActor now times deleting a running actor, including sandbox teardown, so it is not comparable with earlier runs. Follow-ups (since this PR is large): - Bring back ActorStartup that liveness polls at the job interval. - Bound resume retries: a resume that keeps failing retries the same cycle forever. That was already true before this change. - Workers tend to recreate actors around the same time, which can cause bursts of creates. - Record actor startup failures. An actor whose sandbox never answers liveness is deleted and logged, but no failure row is recorded. That was **already true**, but since every task now starts an actor, startup failures can happen per task and are invisible in failure ratios. The ActorStartup row will add both its latency and its failures. Tested with go test -race and on a GKE cluster with the astropy template. - [X] Tests pass - [X] Appropriate changes to documentation are included in the PR
When several agents idle-suspend at the same time, their snapshot uploads allocate compression and storage buffers independently. Ten simultaneous snapshots with eight files each can therefore produce a large memory spike in the objectstore plugin. This adds a shared upload memory budget across requests, with a 512 MiB default configurable through `--upload-memory-budget`. Uploads reserve capacity for compression and S3/GCS buffers before starting, reduce compression concurrency when needed, and wait when the budget is exhausted. Cancellation and failures release capacity after workers finish. Manifests included with data files are uploaded only after those files succeed. The budget limits reserved upload buffers and compression state; it is not a process RSS limit. Queuing trades burst throughput for lower peak memory. The plugin README documents configuration, minimum budgets, and accounting limits. Part of agent-substrate#2553. This addresses simultaneous snapshot uploads and is independent of the small-file allocation optimization in agent-substrate#2633. Validation: - `hack/verify-all.sh` passed. - Object storage, plugin, and affected atelet snapshot/upload tests passed twice with the race detector. Coverage includes concurrent snapshots, cancellation, failures, manifest ordering, and S3/GCS buffering through mocked endpoints. - The full race suite (`go test -race -p 2 ./... -timeout=5m`) hit the existing `TestServerTLSConfigReloadsCACertsWithoutRestart` flake in the benchmarking fake-atelet, reproduced on unchanged upstream (3/5 attempts). The production atelet package passed. - Color-formatting tests failed with this shell's `NO_COLOR=1` and passed twice with that variable unset. - No live cluster or cloud throughput tests were run. One synthetic run with 80 small files and a simulated 20 ms destination stall measured the following tradeoff: | Admission | Sampled peak heap growth | Batch completion | | --- | ---: | ---: | | Shared 512 MiB budget | 279.7 MiB | 225.146 ms | | All 80 uploads admitted | 1,305 MiB | 44.592 ms | These are compression backpressure measurements, not cloud throughput or process RSS measurements. - [ ] Tests pass (affected tests pass; existing TLS flake noted above) - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
… actor cgroup OOM (agent-substrate#1917) ## Summary When a gVisor (`runsc`) sandbox is paused or suspended under a tight memory limit (`ActorMemory`), writing the snapshot (`pages.img`) inside the actor's own cgroup doubles memory pressure (`guest RAM + snapshot page cache`), causing severe swap thrashing or cgroup OOM-kills. This PR decouples snapshot I/O from the actor's memory cgroup and deterministically reclaims snapshot page cache after restore: 1. **Checkpoint Cgroup Isolation**: Run `runsc checkpoint` in a dedicated sibling cgroup(-_ckpt) so snapshot page cache is not charged against the actor's memory limit during pause or suspend. 2. **Post-Restore Cache Reclamation**: After `runsc wait -restore` completes, evict snapshot page cache (guarded by `cachestat(2)` so `FADV_DONTNEED` never forces writeback). Staged files in `restore_dir` are removed only when `preserve_restore_dir` is unset (GCS `restore-state/`); for local resume (agent-substrate#1876) the files are kept and only their cache is evicted. ## Notes Rebased on agent-substrate#1876: file cleanup now keys off `preserve_restore_dir` instead of `Nlink`, since a local restore no longer has a hard-link. `DATA_ON_GOLDEN` case dropped after agent-substrate#2105. ## Benchmark ### 1. Memory-Constrained Local Pause/Resume (`glutton_1user_500mi_gvisor_pause`) 1 user, 500 MiB active guest RAM (64 MiB dirtied before each pause), 600 MiB actor memory cgroup limit (`ActorMemory`), 1m duration. | Metric | Baseline (`main`) | With Fix (agent-substrate#1917) | Improvement | | :--- | :---: | :---: | :---: | | **Completed Cycles (in 60s)** | 6 (1 actor OOM-crashed) | **23** | **3.8x throughput** | | **`PauseActor` Failure Rate** | 25.0% (2/8 failed, `runsc checkpoint` OOM) | **0.0% (0/23)** | **Eliminates checkpoint OOM** | | **`PauseActor` P50 / Avg / P95** | 3,400 ms / 3,245 ms / 5,800 ms | **620 ms / 980 ms / 2,900 ms** | **-82% P50 (-70% Avg, -50% P95)** | | **`ResumeActor` P50 / Avg / P95** | 3,000 ms / 2,957 ms / 3,000 ms | **370 ms / 404 ms / 660 ms** | **-88% P50 (8.1x faster)** | ### 2. Memory-Constrained GCS Suspend/Resume (`glutton_1user_500mi_gvisor_suspend`) 1 user, 500 MiB active guest RAM (64 MiB dirtied before each suspend), 600 MiB actor memory cgroup limit (`ActorMemory`), 1m duration. | Metric | Baseline (`main`) | With Fix (agent-substrate#1917) | Improvement | | :--- | :---: | :---: | :---: | | **Completed Cycles (in 60s)** | 2 (5 actors OOM-crashed) | **9 (10 suspends)** | **4.5x–5x throughput** | | **`SuspendActor` Failure Rate** | 83.3% (10/12 failed, `runsc checkpoint` OOM) | **0.0% (0/10)** | **Eliminates checkpoint OOM** | | **`SuspendActor` Latency (successful)** | 7,700 ms – 8,844 ms | **2,800 ms P50 / 3,105 ms Avg / 4,400 ms P95** | **-64% to -68%** | | **`ResumeActor` P50 / Avg / P95** | 1,800 ms / 1,715 ms / 1,815 ms ($N=2$) | **1,500 ms / 1,605 ms / 2,300 ms ($N=9$)** | **-17% P50 (-6% Avg)** | #### 3. Multi-User Local Pause/Resume (`glutton_mem_512mi_4_user_gvisor_pause`) 4 concurrent users, 512 MiB active guest RAM (64 MiB dirtied before each pause), 768 MiB actor memory cgroup limit (`ActorMemory`), 1m duration. | Metric | Baseline (`main`) | With Fix (`agent-substrate#1917`) | Improvement | | :--- | :---: | :---: | :---: | | **Completed Cycles (in 60s)** | `20` (2 actors OOM-crashed) | **`27`** | **+35% throughput** | | **`PauseActor` Failure Rate** | **`15.4%`** (4/26 failed, `runsc checkpoint` OOM) | **`0.0%`** (0/30) | **Eliminates checkpoint OOM** | | **`PauseActor` P50 / Avg / P95** | `3,100 ms / 3,272 ms / 8,600 ms` | **`1,000 ms / 1,008 ms / 1,400 ms`** | **-68% P50 (-69% Avg, -84% P95)** | | **`ResumeActor` P50 / Avg / P95** | `3,800 ms / 3,698 ms / 6,000 ms` ($N=20$) | **`5,000 ms / 4,365 ms / 5,600 ms`** ($N=27$) | **-7% P95** | > **Note:** `ResumeActor` P50/Avg is slightly higher because `main` OOM-crashes 2 of the 4 actors during `PauseActor`, whereas with `agent-substrate#1917` all 4 actors stay alive and share the worker pod's 2-vCPU limit concurrently (+35% more completed cycles). - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…gent-substrate#2379) Each SWE-perf task is now one line in `manifests/sweperf/sweperf-images.json`: the name, the image pinned by digest, and the step count. `generate_sweperf_templates.py` renders `sweperf-template.yaml.j2` into `manifests/sweperf/generated/`. Pass `'*'` to generate every task, or give a task name to generate just that one. It rejects catalog entries that are incomplete or not pinned by digest. The generated templates are checked in, and `--check` fails if any are missing, out of date, or no longer in the catalog. To add a task, add it to the catalog, run the generator and commit the new file. `deploy.sh` now reads `sweperf-*` templates from `manifests/sweperf/generated/`. Before touching the cluster, it checks that every requested template has a manifest, and if a sweperf one is missing it points you to the generator. Because images are pinned in the catalog, the `${SWEPERF_IMAGE}` substitution is gone. The hand-written `swebench-astropy-7336` template is replaced by the generated `sweperf-astropy-7336`, which is now the default `--sweperf-template`. The client still reads the step count from `--sweperf-total-steps`, so set it to match the catalog. The template keeps the value only for reference. Reading it straight from the template will be a follow-up. Testing: `make verify` and `test_generate_sweperf_templates.py` pass, and `--check`. On GKE I deployed `sweperf-astropy-7336` and `sweperf-django-11099`, both got golden snapshots, and a Locust run created actors from the astropy template. - [X ] Tests pass - [ X] Appropriate changes to documentation are included in the PR
agent-substrate#2707) Fixes agent-substrate#2676. If worker deletion or another crash writer marks an actor CRASHED after checkpoint succeeds but before finalize re-reads it, SuspendActor currently overwrites that crash with SUSPENDED, while PauseActor reports success for the CRASHED actor. Both finalize steps now require the re-read actor to be SUSPENDING before validating snapshots, releasing the worker, or writing the final state. Otherwise they return FailedPrecondition and preserve the existing state, crash reason, and snapshot records. The existing version precondition still rejects a concurrent crash after the read with Aborted. Completed suspend and pause calls retain their existing success paths at workflow entry. The full-workflow regression now checks FailedPrecondition and preservation of the original crash in both modes, updating the behavior pinned by agent-substrate#2688. The already-crashed pause fixture also clears WorkerAssignment, matching a real crash; it previously reached FailedPrecondition only through the missing-node branch. Validation of the production change on Linux with CI=true and real PostgreSQL test containers: - RED: before the production fix, the corrected already-crashed pause test and both modes of the full-workflow regression fail because the calls incorrectly return OK. - GREEN: `go test -race -count=1 -json ./...` passes: 158 test-bearing packages and 9,670 test entries, including all 763 controlapi entries and 169 API functional entries. - `go build ./cmd/ateapi ./cmd/atelet ./cmd/kubectl-ate`, `hack/verify-all.sh`, and the tools/apitool race tests pass. The verifier leaves the source tree unchanged. - The full suite skips 41 tests requiring root privileges, IPv6 loopback, or configured external object storage. The affected PostgreSQL regressions do not skip. E2E follow-up: The original CI passed both gVisor suites but failed both microVM suites in `TestGracefulWorkerTerminationSuspend`. That test unconditionally expected SuspendActor to succeed after worker deletion, even when deletion had already crashed the actor. The test now ties the RPC result to the stored state. OK requires SUSPENDED with no crash record. FailedPrecondition or Aborted requires CRASHED, a crash reason, and the previous completed durable snapshot. Both outcomes require the worker and its assignment to be removed. All other errors fail the test. Follow-up validation on Linux with CI=true and real PostgreSQL and envtest: - The four affected controlapi regressions pass with `-race -count=3`. - All 763 controlapi test entries and all 169 functionaltest entries pass, with no skips. The functional package was rerun after initializing the isolated source directory's Git metadata, which envtest setup requires. - `go test -race -c ./internal/e2e/suites/demo` and `go vet ./internal/e2e/suites/demo` pass. Native compilation, gofmt, and whitespace checks also pass. - Real runtime E2E has not run at this head. The new [pr-workflow run](https://github.com/agent-substrate/substrate/actions/runs/38817919692) is `action_required` and needs maintainer approval. Draft agent-substrate#2694 refactors these same workflows and currently preserves an out-of-band state while returning success. This focused fix also reports FailedPrecondition, as required by agent-substrate#2676; that behavior should carry forward into the refactor. - [x] Local validation passes; runtime E2E awaits workflow approval. - [x] Documentation reviewed; finalize and regression-test comments describe the corrected behavior. No guide change is needed.
…KiB pages and root (agent-substrate#2624) Both unit tests failed on an arm64 host with 64 KiB pages where the test user is root, and neither failure pointed at the code under test. - `TestSetupBundleRootfs_ManyLayers` built 64 production-length lowerdir paths, about 8 KiB: more than a 4 KiB page, less than a 64 KiB one, so its own guard failed it. Each path is now padded to the 255 bytes one fsconfig(2) string value carries, and the layer count follows `os.Getpagesize()`: 17 layers on 4 KiB pages, 257 on 64 KiB pages. - A chain that long makes the overlay's `/proc/self/mountinfo` record longer than `bufio.Scanner`'s 64 KiB default token, so `UnmountAllUnder` failed with "token too long". `mountPointsIn` now reads lines with no length cap, with a regression test. - `TestGetFileToPublishedCopyIsWriteProtected` expects a write to a 0444 file to fail. Root bypasses file permissions, so the test skips as root, like the other permission-based tests. - [x] Tests pass: `go test -race ./internal/imagecache/... ./cmd/atelet/internal/filecache/...` and `golangci-lint` are clean; the root-gated overlay test needs Linux root and runs under `hack/run-root-tests.sh`, which CI does under sudo. - [x] Appropriate changes to documentation are included in the PR: none needed.
…nt-substrate#1906) Durable-dir volumes were created `0700`, so a container running as a non-root user could not write its own volume. Now `0777`, as Kubernetes creates an emptyDir: one actor's containers can run as different users, and the `0700` actor directory above keeps other actors out. A restore keeps the mode the snapshot recorded, as emptyDir keeps a pod's own chmod. Volumes created earlier stay `0700`, fine while every container runs as root. No effect until an image's `USER` is honored (agent-substrate#1918). Cleanup of files a non-root container leaves behind landed in agent-substrate#1910. Tests: `TestPrepareDurableDirVolume`. Exercised on kind with the micro-VM class: uid 65532 writes a durable dir, then suspend and resume. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
…data in trial_summary (agent-substrate#2232) Part of agent-substrate#1590 Adds facts to each run's `trial_summary` row so runs can be compared and traced later, even after the cluster is deleted. **New keys** - `kubernetes_version`: the worker nodes' Kubernetes version, as the kubelet reports it. - `worker_actor_capacity`, `worker_memory_limit_gb`, `sandbox_class`: what each worker pod declares (ateworker `--max-actors`, the container memory limit, and gvisor or microvm from the worker image). Each value is read on its own, so an unreadable one is skipped (and logged once) without dropping the others. - `multi_actor_worker`: `true` when `actors_per_pod_p50` > 1. - `run_start`, `run_end`: Unix seconds bracketing the run, for finding its server-side metrics afterwards. - `metadata_<key>`: one per key of a new free-form `--metadata '<json>'` runner flag, for facts only the caller knows. **Orchestrator** - Passes `--metadata` to locust Jobs with the target cluster's `cluster_name`, `cluster_location` and `project_id`, merged with the `BENCHMARK_METADATA` env var (a JSON object), if it's set. - A test's own `--metadata` in `tests.yaml` is merged in too, its keys winning on conflict, so the runner gets a single `--metadata`. - Fails at startup if `BENCHMARK_METADATA` or a test's `--metadata` is not a valid JSON object. - Nighthawk Jobs are unchanged (that runner has no `--metadata`). **Example** (from a test run): ``` "kubernetes_version": "v1.37.1-gke.1506000", "worker_actor_capacity": "1000", "worker_memory_limit_gb": null, "sandbox_class": "gvisor", "multi_actor_worker": "true", "run_start": "1791245420", "run_end": "1791246036", "metadata_cluster_name": "<cluster>", "metadata_cluster_location": "us-east1-b", "metadata_project_id": "<project>", "metadata_test": "manual", ``` - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
) This is the next step toward [async suspend](https://github.com/zoez7/substrate/blob/async-suspend-design/docs/dev/async-suspend-en.md), following agent-substrate#2688. `SuspendActor` and `PauseActor` now run one set of shared steps behind a `suspendMode`: - `suspendSync` (`SuspendActor`): the snapshot is durable before the workflow returns. - `suspendAsync` (`PauseActor`): the snapshot stays on the worker's node. Both APIs stay for now. `workflow_pause.go` is removed. Behavior is unchanged. The tests agent-substrate#2688 moved testing of ensurePaused* and ensureSuspended* functions onto `PauseActor` and `SuspendActor`. The only test edit is in `workflow_test.go`, which called the removed `ensurePausedFinalized`. The fix for agent-substrate#2676 (finalize not checking that the actor is still `SUSPENDING`) will come in a separate PR. - [x] Tests pass - [ ] Appropriate changes to documentation are included in the PR
…rate (agent-substrate#2181) This commit changes everything except HTTP(s) links. github.com/agent-substrate/substrate -> go.ate.dev/substrate I chose "go.ate.dev" for brevity, but "pkg.ate.dev" would be OK, too. This is implemented as "CloudFlare Pages" on my personal account. If we like it, we need to make it more "community operated". This does not yet convert other repos. This breaks a TON of open PRs, so we should consider if/when we want to merge it. /cc @BenTheElder @rakyll @EItanya
some base images include `/dev/*` device nodes in their layer tars. atelet runs without `CAP_MKNOD` and the runtime mounts `/dev` anyway, so skip char/block device entries when unpacking layers (like rootless containerd does) instead of failing the pull. while here: - if an entry replaces a directory that had subdirectories in the same layer, drop the removed paths from `dirModes` so the post-unpack chmod pass doesn't fail on them - include the tar entry path in `unhandled tar entry typeflag` errors and the image ref in `EnsureImage` errors - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
… for dynamic worker registration (agent-substrate#2660) Part of agent-substrate#587. Follows agent-substrate#2521. agent-substrate#2521 added the initial `atelet` syncer that wrote `sandbox_runtime_hash` per enabled `SandboxConfig` version to each local worker pod's directory. This PR completes the runtime and hardware compatibility pipeline from `atelet` through the worker into `WorkerService.RegisterWorker`: 1. **Renames `sandbox-versions.json` to `sandbox-runtimes.json` (`internal/sandboxruntimes`)**: `atelet` now writes the complete set of `VersionedSandboxCompat` attributes for `defaultRuntime` and `restorableRuntimes` (`architecture`, `sandbox_runtime_hash`, `cpu_features` for `gvisor`, and `cpu_vendor` + `cpu_model` for `microvm`). 2. **Probes CPU identity and gVisor `cpu_features` in `atelet`**: `atelet` owns all hardware and runtime probing on the node — reading `/proc/cpuinfo` for microVM CPU vendor/model and running `<runsc> cpu-features` (memoized by asset SHA-256) for each enabled gVisor version, encoded as a canonical lowercase hexadecimal bitmask whose bit numbers are gVisor's own `Feature` constants (`internal/hardware`). 3. **Watches `sandbox-runtimes.json` in the worker**: `ateworker-gvisor` and `ateworker-microvm` watch `/var/lib/ate/workers/<pod-uid>/sandbox-runtimes.json` via `register.WatchFile` and keep `default_runtime` and `restorable_runtimes` registered through `WorkerSupport.RegisterWorker` (`internal/ateworker/register`). ## End-to-End Flow ```mermaid flowchart LR subgraph Node["Kubernetes Node"] direction TB SC["SandboxConfig CRD"] --> Atelet["atelet (workerSandboxSyncer)"] CPU["Host /proc/cpuinfo\n(microVM vendor & model)"] --> Atelet Runsc["Cached runsc binary\n(<runsc> cpu-features)"] --> Atelet Atelet -->|"Atomic rename"| File["/var/lib/ate/workers/<pod-uid>/\nsandbox-runtimes.json"] File -->|"register.WatchFile (fsnotify)"| Worker["ateworker-gvisor / ateworker-microvm"] Worker -->|"WorkerSupport.RegisterWorker"| Atelet end Atelet -->|"WorkerService.RegisterWorker"| AteAPI["ateapi (Control Plane)"] ``` ## How it works ```mermaid sequenceDiagram participant Atelet as atelet participant FS as /var/lib/ate/workers/<pod-uid>/ participant Watcher as register.WatchFile (fsnotify / poll fallback) participant Reconciler as reconciler.run / sync participant Support as WorkerSupport.RegisterWorker Note over Watcher,FS: Watches parent dir for Create/Write on sandbox-runtimes.json Reconciler->>FS: Initial sync(): Read sandbox-runtimes.json alt File not written yet FS-->>Reconciler: ErrNotExist (wait for watch event or 5m resync) end Atelet->>FS: Probe hardware/runsc & atomic rename -> sandbox-runtimes.json FS-->>Watcher: fsnotify Create/Write event Watcher->>Reconciler: signal on changes channel Reconciler->>FS: Read sandbox-runtimes.json Reconciler->>Reconciler: proto.Equal(req, registered)? alt Changed (or first registration) Reconciler->>Support: RegisterWorker(capacity, default_runtime, restorable_runtimes) alt RPC succeeds Support-->>Reconciler: OK -> record registered = req else InvalidArgument (malformed runtime attributes) Support-->>Reconciler: Log Error, do not retry until file changes else Transient RPC error (e.g. Worker record not in ateapi yet) Support-->>Reconciler: Retry with exponential backoff (500ms..30s) end end ``` - **`atelet` owns all probing**: - For **`gvisor`** workers, `atelet` fetches the version's `gvisor` asset via `ensureSandboxAssets`, executes `<runsc> cpu-features` (memoized in-memory by the asset's SHA-256 digest), and encodes the reported names via `hardware.EncodeCPUFeatures` into a compact lowercase hex bitmask alongside `architecture` and `sandbox_runtime_hash`. - For **`microvm`** workers, `atelet` reads `/proc/cpuinfo` once (memoized after the first successful read) to record `cpu_vendor` and `cpu_model` alongside `architecture` and `sandbox_runtime_hash`. - **`cpu_features` bit layout**: - A feature's bit is the value of its gVisor `Feature` constant: on amd64, `32*block + bit` in that block's CPUID register (`X86FeatureAVX2 = 2*32 + 5 = 69`); on arm64, the value gVisor declares. Intel/AMD never reassign a CPUID bit, so a bit never moves when gVisor adds, reorders, or renames a feature, and a bit with no entry is one the hardware reserves. - `internal/hardware/cpufeatures_table.go` is a committed table, produced once from gVisor's `pkg/cpuid` sources at `gvisor.dev/gvisor@v0.0.0-20261014212059-ddc37c698141`: for every `Feature` constant with an entry in `allFeatures`, the constant's value keyed to the display name `runsc cpu-features` prints. Each entry's comment names the constant and, on amd64, its block and bit. The file header records this and how to add features from a newer gVisor by hand; because a bit is the hardware's, existing entries never change. - **This mapping is temporary.** Once `runsc` can print its feature set as this bitmask directly, `atelet` records that value as printed and the table, `EncodeCPUFeatures`, and `DecodeCPUFeatures` go away (TODO in the table header). The mask format and subset matching stay. No gVisor sources or generator are vendored for it. - **Subset matching**: `hardware.MatchesCompat` decodes `cpu_features` into a stack-allocated `[6]uint64` mask (384 bits; amd64 bits reach 261 today, with room for gVisor blocks through 11) and checks bitwise subset inclusion ($S_{\text{snapshot}} \subseteq S_{\text{worker}}$) while requiring exact equality on all other attributes. `DecodeCPUFeatures` rejects a mask that sets a bit with no name. - **Resilient watching & registration in the worker**: - `register.WatchFile` watches the parent directory for `Create`/`Write` events on the target file (reused by both `register.Report` and `ateworker-gvisor`'s `flagStore`), and falls back to periodic polling (`5s`) with a warning if `fsnotify` initialization fails (e.g. when `fs.inotify.max_user_instances` is exhausted on dense nodes). - `reconciler` runs a `5m` periodic resync tick alongside watch events, deduplicates consecutive registrations via `proto.Equal`, retries transient read/RPC failures with capped exponential backoff (`500ms..30s`), and drops permanent `InvalidArgument` rejections until the file changes. ## Compatibility `cpu_features` values recorded before this PR used a different bit numbering and a narrower mask. They do not compare meaningfully against workers built from this PR: any dev-cluster snapshot carrying one should be retaken rather than restored. No compatibility is guaranteed before v1.0.0, so there is no migration. - [x] Tests pass (`go test -race ./internal/hardware/... ./internal/sandboxruntimes/... ./internal/ateworker/register/... ./cmd/atelet/... ./cmd/ateworker-gvisor/... ./cmd/ateworker-microvm/...`) - [x] Appropriate changes to documentation are included in the PR
…with kubectl ate (agent-substrate#2591) Part of agent-substrate#1563. Fixes agent-substrate#2103, agent-substrate#2106, agent-substrate#2186 ## What - **Authz on by default** in ate-setup, through two settings: `ateapi.authz.enabled` and `ateapi.authz.bootstrapOwners`. The bootstrap owner defaults to the shared `ate-client` ServiceAccount, so demos, benchmarks and CI keep working. ate-setup and `kubectl ate` warn about it, and the docs tell production installs to name their own owners. - **`kubectl ate access-policy`** get/create/update/delete, for an atespace (`-a`) or the global scope (`--global`). - **Tests:** an authz e2e suite (basic flow plus security regressions), `model_test.fga.yaml` run in CI, and Postgres tests that fail rather than skip when CI requires Docker. - **Benchmarks:** the nightly run adds `benchmark-runner` as a bootstrap owner. - **Docs:** new `docs/authorization.md`, covering roles, the checks every RPC makes, managing access and limitations. Follow-up: replace the `ate-client` default with per-caller identities (token options, a `kubectl ate` credential plugin, issuer-qualified principals), then require explicit owners. ## Testing Unit tests and `hack/verify-all.sh` pass. The authz e2e suite runs in the CI e2e lanes.
Integrate the unified suspend workflow and vanity Go module while preserving fork workflow deadlines and immediate worker cache updates. Align Helm API configuration, authorization, controller permissions, and CRDs with the current control plane. Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge 31 upstream commits through agent-substrate/substrate@c4518a1 into the fork. This brings in unified pause/suspend handling, dynamic worker runtime registration, bounded snapshot uploads, API authorization, and the
go.ate.dev/substratemodule path.Preserve the fork's agentgateway dataplane, optional atelet TLS, workflow deadlines, and immediate worker-cache updates after suspend/pause. Update Helm for the required API config, authorization defaults, worker ServiceAccount permissions, and CRDs, and migrate the fork's imports and release build paths.
Breaking change
go.ate.dev/substratefor the new worker and snapshot APIs.--config-file. Helm manages it; replace--log-levelinateApi.extraArgswithateApi.logLevel.ate-clientidentity for development. Shared installs should setateApi.authorization.bootstrapOwnersto operator principals and grant tenant access explicitly. See the chart upgrade notes anddocs/authorization.md.ate-clientand needs an explicit grant. Its current reconciliation callsCreateAtespacewhen creating actor templates, which requires global owner access even if the atespace already exists. Restricting kagent to atespace editor requires pre-provisioning atespaces and adapting that creation path; global worker inventory also needs separate operator access.Please apply the
breaking-changelabel.Validation passed:
env -u NO_COLOR go test -race -p 1 ./...,hack/verify-all.sh, Helm lint, all 18 chart tests, manifest rendering, and CRD parity. The race suite passed on retry with serial package compilation after the first run exhausted the shared 20 GiB temporary filesystem during concurrent compilation. Local runtime E2E has not been run; the PR workflows cover agentgateway with gVisor and microVM.Merge this PR with a merge commit to preserve upstream ancestry for the next sync.