diff --git a/.agents/skills/detect-flaky-tests/SKILL.md b/.agents/skills/detect-flaky-tests/SKILL.md index f66f699c95..09fa827361 100644 --- a/.agents/skills/detect-flaky-tests/SKILL.md +++ b/.agents/skills/detect-flaky-tests/SKILL.md @@ -51,7 +51,7 @@ For each run, you need logs from **two jobs**: | Job name | Coverage | |---|---| | `run-tests` | Unit + integration tests (`go test -race -v ./...`) | -| `e2e-test` | E2E suite, both sandbox classes — two sequential steps in the one job | +| `e2e-test`, `e2e-test (envoy, preview)` | E2E suite, both sandbox classes — two sequential steps in each job | ```bash # List all jobs for a run @@ -62,10 +62,17 @@ gh api "repos/agent-substrate/substrate/actions/runs//jobs" \ gh api "repos/agent-substrate/substrate/actions/jobs//logs" > /tmp/job_.log ``` -The two e2e lanes are NOT a matrix — `e2e-test` is a single job that runs the step +`e2e-test` is a matrix over preview mode: `e2e-test` runs with no preview features, +skipping their tests, and `e2e-test (envoy, preview)` runs with every one on. Other +dataplanes, where they run, are named `e2e-test ()` and `e2e-test +(, preview)`. Match all of them by the `e2e-test` prefix, and keep the +preview mode with each result: a test that fails only without preview features is a +gating bug, not a flake. + +The two sandbox-class lanes are NOT a matrix — each `e2e-test` job runs the step "Run E2E tests (gVisor)" followed by "Run E2E tests (micro-VM)" (same `hack/run-e2e-kind.sh` command, the second with `E2E_SANDBOX_CLASS: microvm`). Split the -one `e2e-test` job log at the "Run E2E tests (micro-VM)" step boundary: PASS/FAIL lines +each `e2e-test` job log at the "Run E2E tests (micro-VM)" step boundary: PASS/FAIL lines before it belong to the gVisor lane, lines after it to the microVM lane. Because the steps are sequential, a gVisor-lane failure means the micro-VM step never ran — record no microVM results for that run rather than counting them as failures. @@ -77,7 +84,8 @@ grep -E '^--- (PASS|FAIL): ' /tmp/job_.log \ ``` Track results per (test_name, job_type) where job_type is one of: -`unit`, `e2e-gvisor`, `e2e-microvm`. +`unit`, `e2e-gvisor`, `e2e-microvm`, `e2e-gvisor-preview`, `e2e-microvm-preview`. The +`-preview` job types come from the preview jobs; the others from the jobs without. --- @@ -167,7 +175,7 @@ gh issue create \ ## Flaky test detected **Test:** `` -**Job:** `` +**Job:** `` ### Evidence (last 7 days) @@ -219,7 +227,7 @@ gh issue create \ ## Recurring infrastructure failure in CI **Pattern:** `` -**Job type:** `` +**Job type:** `` ### Evidence diff --git a/.agents/skills/post-draft-review/SKILL.md b/.agents/skills/post-draft-review/SKILL.md index 11ff542ff7..f6ff72c29c 100644 --- a/.agents/skills/post-draft-review/SKILL.md +++ b/.agents/skills/post-draft-review/SKILL.md @@ -142,7 +142,7 @@ mutation($review: ID!, $path: String!, $line: Int!, $body: String!) { path: $path, line: $line, side: RIGHT, body: $body }) { thread { id } } }' -f review="PRR_kwDO..." \ - -f path="cmd/ateom-microvm/restore.go" -F line=214 \ + -f path="cmd/ateworker-microvm/restore.go" -F line=214 \ -F body=@c1.md ``` diff --git a/.agents/skills/triage-issues/SKILL.md b/.agents/skills/triage-issues/SKILL.md index 91ef940a3c..205dcaf1f0 100644 --- a/.agents/skills/triage-issues/SKILL.md +++ b/.agents/skills/triage-issues/SKILL.md @@ -52,8 +52,8 @@ Apply all areas that are touched. Most issues need one or two; a few need more. | `area/api-machinery` | API server internals: RPC handlers, storage layer, syncer, controllers (`cmd/ateapi/`, `cmd/atecontroller/`) | | `area/network` | Networking: atenet-router, Envoy, ext_proc, DNS, xDS, ingress (`cmd/atenet/`) | | `area/node` | Node agent and worker lifecycle: atelet, sandbox launch, OCI, cgroups (`cmd/atelet/`) | -| `area/gvisor` | gVisor sandbox specifics: runsc integration, GPU in gVisor, gVisor OCI (`cmd/ateom-gvisor/`) | -| `area/microVM` | Micro-VM sandbox specifics: cloud-hypervisor, kata, snapshot/restore (`cmd/ateom-microvm/`) | +| `area/gvisor` | gVisor sandbox specifics: runsc integration, GPU in gVisor, gVisor OCI (`cmd/ateworker-gvisor/`) | +| `area/microVM` | Micro-VM sandbox specifics: cloud-hypervisor, kata, snapshot/restore (`cmd/ateworker-microvm/`) | | `area/storage` | Snapshot storage, image cache, GCS/S3 backends, retention (`internal/imagecache/`, ActorSnapshot) | | `area/scheduling` | WorkerPool sizing, actor placement, HPA, resource allocation | | `area/observability` | Metrics, tracing, logging, OTLP, OTel SDK integration | diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 5c9f6cf216..b320dd3366 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -98,7 +98,7 @@ body: id: logs attributes: label: Relevant Logs and Diagnostic Output - description: Please provide relevant logs from atelet, ateom, ate apiserver, or kubectl describe. + description: Please provide relevant logs from atelet, ateworker, ate apiserver, or kubectl describe. render: shell placeholder: | # Paste relevant logs or command output here diff --git a/.github/SECURITY.md b/.github/SECURITY.md index bbd486ab87..c94264a56e 100644 --- a/.github/SECURITY.md +++ b/.github/SECURITY.md @@ -52,7 +52,7 @@ See the [threat model](../docs/threat-model.md) for the trust boundaries and assumptions this section is based on. In scope: the Agent Substrate control plane (`ateapi`), node supervisor -(`atelet`, `ateom`), networking stack (`atenet`), and CLI (`kubectl-ate`). +(`atelet`, `ateworker`), networking stack (`atenet`), and CLI (`kubectl-ate`). Agent Substrate multiplexes many actors onto a smaller pool of shared workers, so its most important security boundaries are between actors and @@ -70,7 +70,7 @@ valuable, listed roughly from most to least severe: `ateapi` RPCs without valid credentials (mTLS or JWT auth modes), misusing workerpool client certificates, or acting beyond what valid credentials are authorized to do. -- **Actor-to-platform escalation**: an actor compromising `atelet`, `ateom`, +- **Actor-to-platform escalation**: an actor compromising `atelet`, `ateworker`, or the control plane, whether while the actor is still actively running or beyond its own lifecycle. - **Node lateral movement**: using a compromised worker node to reach other diff --git a/.github/workflows/helm-e2e.yaml b/.github/workflows/helm-e2e.yaml index f064231ba6..5af10fa1f9 100644 --- a/.github/workflows/helm-e2e.yaml +++ b/.github/workflows/helm-e2e.yaml @@ -26,6 +26,8 @@ jobs: VERSION: helm-e2e E2E_ATENET_DATAPLANE: agentgateway E2E_CREDENTIAL_PROVIDER: "1" + # Retain external-volume coverage after the preview gate is introduced. + ATE_PREVIEW: ALL ARTIFACTS: ${{ github.workspace }}/_artifacts steps: - name: Checkout @@ -97,6 +99,7 @@ jobs: --set postgres.clientCertificates.enabled=true \ --set image.registry=localhost:5001 \ --set image.tag=helm-e2e \ + --set 'preview[0]=ALL' \ --set 'atelet.extraArgs[0]=--localhost-registry-replacement=kind-registry:5000' \ --set otel.endpoint=http://opentelemetry-collector.otel-system.svc:4317 \ --wait --timeout=10m @@ -124,20 +127,20 @@ jobs: # tests keep a stable trust bundle and gateway signing certificate. env: E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/helm-gvisor.xml - run: hack/run-e2e-kind.sh -v -skip '^(TestActorIdentity_|TestActorEgressCredentialInjection$)' -args --no-color + run: hack/run-e2e-kind.sh -v -skip '^(TestActorIdentity_|TestActorEgressCredentialInjection$)' -args --no-color --preview="${ATE_PREVIEW}" - name: Run E2E tests (micro-VM) env: E2E_SANDBOX_CLASS: microvm E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/helm-microvm.xml - run: hack/run-e2e-kind.sh ./internal/e2e/suites/demo -v -args --no-color + run: hack/run-e2e-kind.sh ./internal/e2e/suites/demo -v -args --no-color --preview="${ATE_PREVIEW}" - name: Run E2E tests (egress credential injection) env: E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/helm-egress-credentials.xml - run: hack/run-e2e-kind.sh ./internal/e2e/suites/egresscredinject -v -args --no-color + run: hack/run-e2e-kind.sh ./internal/e2e/suites/egresscredinject -v -args --no-color --preview="${ATE_PREVIEW}" - name: Run E2E tests (identity and trust rotation) env: E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/helm-identity.xml - run: hack/run-e2e-kind.sh ./internal/e2e/suites/identity -v -args --no-color + run: hack/run-e2e-kind.sh ./internal/e2e/suites/identity -v -args --no-color --preview="${ATE_PREVIEW}" - name: Require every lane to have run tests if: always() run: | diff --git a/.github/workflows/pr-workflow.yaml b/.github/workflows/pr-workflow.yaml index 6159da5b65..da91d9e414 100644 --- a/.github/workflows/pr-workflow.yaml +++ b/.github/workflows/pr-workflow.yaml @@ -90,8 +90,8 @@ jobs: ~/.cargo/registry/index ~/.cargo/registry/cache ~/.cargo/git/db - cmd/dataplane/envoy/dynamic-modules/*/target - key: cargo-${{ runner.os }}-${{ hashFiles('cmd/dataplane/envoy/dynamic-modules/*/Cargo.lock') }} + cmd/dataplane/envoy/dynamic-modules/target + key: cargo-${{ runner.os }}-${{ hashFiles('cmd/dataplane/envoy/dynamic-modules/Cargo.lock') }} restore-keys: cargo-${{ runner.os }}- - name: dynamic module tests run: | @@ -122,8 +122,10 @@ jobs: # One kind cluster exercises BOTH runtimes. Free x86-64 ubuntu-latest runners # expose /dev/kvm (with a udev rule), so create-kind-cluster.sh mounts it and the # micro-VM (kata + cloud-hypervisor) sandbox class works alongside gVisor. + # Each preview mode runs on a fresh cluster with agentgateway. + # The default install retains the required e2e-test check name. e2e-test: - name: e2e-test + name: ${{ matrix.preview == 'off' && 'e2e-test' || 'e2e-test (preview)' }} runs-on: ubuntu-latest # Cluster bring-up, image builds and two sequential lanes; generous, but far # short of the six-hour platform default a hung lane would otherwise consume. @@ -131,8 +133,8 @@ jobs: strategy: fail-fast: false matrix: - include: - - dataplane: agentgateway + dataplane: ['agentgateway'] + preview: ['off', 'on'] env: E2E_ATENET_DATAPLANE: ${{ matrix.dataplane }} # The installer requires a credential provider selection, and deploys the @@ -140,6 +142,14 @@ jobs: # because every ate-setup invocation reads it, including the control # plane redeploy in run-microvm-demo-kind.sh. ATE_CREDENTIAL_PROVIDER: '{"name":"k8s.io"}' + # Job-wide for the same reason. Empty is an explicit "no preview + # features", not an unset value. The e2e steps pass it to the suites as + # --preview, so tests of a disabled gate skip themselves (see + # internal/e2e/preview.go). + ATE_PREVIEW: ${{ matrix.preview == 'on' && 'ALL' || '' }} + # Appended to the lane and artifact names: e2e-gvisor runs without + # preview features, e2e-gvisor-preview with them. + LANE_SUFFIX: ${{ matrix.preview == 'on' && '-preview' || '' }} # E2E_JUNIT_FILE is set per step, not here: the lanes below share this # job, so one job-level path would have each overwrite the last. ARTIFACTS: ${{ github.workspace }}/_artifacts @@ -147,7 +157,7 @@ jobs: # share one copy instead of each storing their own. DOCKER_BUILD_FLAGS: >- --cache-from type=gha,scope=envoy-dataplane - ${{ github.ref == 'refs/heads/main' && matrix.dataplane == 'envoy' + ${{ github.ref == 'refs/heads/main' && matrix.preview == 'off' && '--cache-to type=gha,mode=max,scope=envoy-dataplane,ignore-error=true' || '' }} steps: - name: Checkout @@ -209,8 +219,10 @@ jobs: run: hack/create-kind-cluster.sh - name: Install Agent Substrate (${{ matrix.dataplane }}) # The dataplane selection applies to both the ingress router and egress - # gateway. - run: hack/install-ate-kind.sh --deploy-ate-system --atenet-dataplane=${{ matrix.dataplane }} --rollout-timeout=300s + # gateway. ATE_PREVIEW above selects the preview features. + env: + ATE_ATENET_DATAPLANE: ${{ matrix.dataplane }} + run: hack/install-ate-kind.sh --deploy-ate-system --rollout-timeout=300s - name: Enable NFS # Load NFS kernel modules so in-cluster NFS server and CSI driver can run. run: | @@ -240,8 +252,8 @@ jobs: - name: Run E2E tests (gVisor) env: E2E_PARALLELISM: 8 - E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/e2e-gvisor.xml - run: hack/run-e2e-kind.sh -v -args --no-color + E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/e2e-gvisor${{ env.LANE_SUFFIX }}.xml + run: hack/run-e2e-kind.sh -v -args --no-color --preview="${ATE_PREVIEW}" - name: Focus the node on micro-VM run: hack/e2e-focus-sandbox-class.sh microvm - name: Run E2E tests (micro-VM) @@ -250,8 +262,8 @@ jobs: env: E2E_SANDBOX_CLASS: microvm E2E_PARALLELISM: 8 - E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/e2e-microvm.xml - run: hack/run-e2e-kind.sh -v -args --no-color + E2E_JUNIT_FILE: ${{ github.workspace }}/_artifacts/e2e-microvm${{ env.LANE_SUFFIX }}.xml + run: hack/run-e2e-kind.sh -v -args --no-color --preview="${ATE_PREVIEW}" # Every lane must have reported tests; a `-run` filter that stops matching # otherwise leaves its lane green and empty. The set comes from the manifest # each lane registers itself in, so adding a lane needs no change here, and @@ -262,7 +274,7 @@ jobs: - name: Save Go build cache # Main only, so PRs restore the latest main build rather than each # storing its own copy. - if: ${{ !cancelled() && github.ref == 'refs/heads/main' }} + if: ${{ !cancelled() && github.ref == 'refs/heads/main' && matrix.preview == 'off' }} uses: actions/cache/save@0057852bfaa89a56745cba8c7296529d2fc39830 # v4.3.0 with: path: | @@ -273,7 +285,7 @@ jobs: if: always() uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: - name: test-results-e2e-${{ matrix.dataplane }} + name: test-results-e2e-${{ matrix.dataplane }}${{ env.LANE_SUFFIX }} path: ${{ env.ARTIFACTS }} if-no-files-found: error - name: Dump diagnostics on failure @@ -289,7 +301,7 @@ jobs: done # Every worker pod in any namespace: the demo pools plus the e2e suites' # randomly-named per-test namespaces, which the suites keep on failure. - # The failing actor runs in one of these, so this is where its ateom logs + # The failing actor runs in one of these, so this is where its worker logs # (and, for a micro-VM worker, the guest console tail) live. kubectl --context kind-kind get pods -A -l ate.dev/worker-pool \ -o 'custom-columns=:.metadata.namespace,:.metadata.name' --no-headers 2>/dev/null \ diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml index bc764a539a..37393f7087 100644 --- a/.github/workflows/release.yaml +++ b/.github/workflows/release.yaml @@ -88,7 +88,7 @@ jobs: run: | set -o errexit -o nounset -o pipefail - for component in ateapi atecontroller atelet ateom-gvisor ateom-microvm podcertcontroller atenet objectstore-plugin credential-provider/kubernetes-secrets; do + for component in ateapi atecontroller atelet ateworker-gvisor ateworker-microvm podcertcontroller atenet objectstore-plugin credential-provider/kubernetes-secrets; do KO_DOCKER_REPO="${IMAGE_REPOSITORY}/${component##*/}" \ ./hack/run-tool.sh ko build \ --tags "${IMAGE_TAGS}" \ diff --git a/.golangci.yaml b/.golangci.yaml index b7a075d9fa..96c1f42822 100644 --- a/.golangci.yaml +++ b/.golangci.yaml @@ -27,10 +27,25 @@ linters: default: standard enable: + # Import restrictions, see depguard settings below. + - depguard # Spelling errors in comments, strings, and identifiers. - misspell settings: + depguard: + rules: + # internal/ateworker is the ateworker binaries' shared code. Anything + # else should import a flat package under internal/ instead. + ateworker: + files: + - "$all" + - "!**/cmd/ateworker-*/**" + - "!**/internal/ateworker/**" + deny: + - pkg: github.com/agent-substrate/substrate/internal/ateworker + desc: only the ateworker binaries may import internal/ateworker + errcheck: # Functions where ignoring the return is idiomatic in this codebase. # Note: errcheck's exclude-functions matches on the static receiver diff --git a/AGENTS.md b/AGENTS.md index 1e5634bce0..ab3eaf7953 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,7 +31,7 @@ tools/ # Standalone Go tools (go run ./tools/) for Dev/CI | Shared across binaries, not for external import | `internal/` | | Public API for external consumers | `pkg/` | | Public proto (control-plane gRPC API) | `pkg/proto/` | -| Internal proto (atelet / ateom) | `internal/proto/` | +| Internal proto (atelet / ateworker) | `internal/proto/` | | Dev/CI scripts | `hack/` | | Standalone Go dev/CI tools | `tools/` with its own `go.mod` | | Self-contained plugin that nothing in the repo depends on | `internal/plugins/` with its own `go.mod`, importing only `pkg/`; see `internal/plugins/README.md` | diff --git a/README.md b/README.md index 6432c37fb3..41b9cf60bf 100644 --- a/README.md +++ b/README.md @@ -243,7 +243,7 @@ We provide several sample applications demonstrating Agent Substrate's capabilit * [Architecture](docs/architecture.md): How the control plane, node supervisor, and networking stack fit together. * [API Configuration Guide](docs/api-guide.md): Detailed reference for configuring WorkerPools, ActorTemplates, Secrets, and Volumes. * [Full CLI Documentation](cmd/kubectl-ate/README.md): Installation and usage for `kubectl-ate`. -* [Glossary](docs/glossary.md): Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateom) and how they relate. +* [Glossary](docs/glossary.md): Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateworker) and how they relate. * [Integration Repositories](docs/integration-repos.md): Where integrations live, how their repositories are named, and how fixes flow back to core. * [Observability Guide](docs/observability.md): Guide to actor logging, metrics, and distributed tracing. * [Authentication Guide](docs/authentication.md): Configure trusted JWT providers and human credentials. @@ -263,8 +263,8 @@ We provide several sample applications demonstrating Agent Substrate's capabilit * `cmd/atelet`: A node-level DaemonSet that supervises physical worker pods, coordinates snapshotting, and manages state transfers. * `cmd/atecontroller`: A Kubernetes controller that reconciles WorkerPool custom resources. * `cmd/atenet`: A combined networking controller providing Envoy routing and proxy sidecars. -* `cmd/ateom-gvisor`: An interior-pod helper running inside sandboxed worker pods to execute `runsc` checkpoint and restore commands. -* `cmd/ateom-microvm`: The micro-VM peer of `ateom-gvisor`, running actors as cloud-hypervisor VMs. +* `cmd/ateworker-gvisor`: An interior-pod helper running inside sandboxed worker pods to execute `runsc` checkpoint and restore commands. +* `cmd/ateworker-microvm`: The micro-VM peer of `ateworker-gvisor`, running actors as cloud-hypervisor VMs. * `cmd/podcertcontroller`: A "polyfill" that provides Pod Certificate signers that will eventually ship in upstream Kubernetes (with different names). * `cmd/kubectl-ate`: A CLI tool for managing Agent Substrate resources. See its [README](cmd/kubectl-ate/README.md). diff --git a/benchmarking/README.md b/benchmarking/README.md index 8c45b2c7ea..393e0c33e1 100644 --- a/benchmarking/README.md +++ b/benchmarking/README.md @@ -462,7 +462,7 @@ fakes from `cmd/benchmarking/isolate/` instead of worker pods: up and down with the pool, draining a Worker before deleting it. Each Worker reports what a real worker would: cpu and memory from the pool's `template.resources.limits`, or the node's allocatable without one, and 1000 - actors, ateom's default, unless the pool's `ate.dev/fake-max-actors` + actors, ateworker's default, unless the pool's `ate.dev/fake-max-actors` annotation says otherwise. It writes the pool's status, so `kubectl get workerpool` and `kubectl scale workerpool` work. A node keeps its Workers for `--node-grace`, five minutes by default, after its @@ -472,11 +472,11 @@ fakes from `cmd/benchmarking/isolate/` instead of worker pods: would run on, and answers every lifecycle call with success after its `--delay`, without running or saving any workload. It writes one placeholder object per snapshot, because - ate-api-server refuses to tag an empty one. Like ateom, it mints the actor's + ate-api-server refuses to tag an empty one. Like ateworker, it mints the actor's certificate from ate-api-server on every run and restore that names an egress gateway, which the default install does; the mint is ate-api-server's time, so it adds to `--delay`, the data plane's share of the call. - It also relays fake-workersync's capacity reports, as atelet relays ateom's: + It also relays fake-workersync's capacity reports, as atelet relays ateworker's: ate-api-server accepts a report only from the atelet on the Worker's node. The relay admits only fake-workersync's pod identity. @@ -647,7 +647,7 @@ still succeeds. A `null` means the value was not measured. It never means zero. `SuspendActor` and `ResumeActor` latency can be attributed to the phases inside them (object-storage transfer, hypervisor snapshot/restore, rootfs -assembly, ...) from the structured log records atelet and ateom-microvm +assembly, ...) from the structured log records atelet and ateworker-microvm emit. `analysis/collect_logs.sh` dumps the node logs of a run and `analysis/phase_report.py` aggregates them into per-phase percentiles and waterfalls of the slowest operations. See diff --git a/benchmarking/analysis/README.md b/benchmarking/analysis/README.md index e5c0ca043c..91076b3cfb 100644 --- a/benchmarking/analysis/README.md +++ b/benchmarking/analysis/README.md @@ -3,7 +3,7 @@ This directory turns the node side's developer-facing timing logs into the percentiles a benchmark run needs, without making the phases metric API. The `ate.actor.{restore,checkpoint}.duration` histograms stop at -`ateom_restore` / `ateom_checkpoint` / `persist` / `download` on purpose: +`worker_restore` / `worker_checkpoint` / `persist` / `download` on purpose: phases inside those are implementation details of a runtime, and metrics are a contract. The finer breakdown is emitted as structured log records instead, and this is the consumer that aggregates them. @@ -12,8 +12,8 @@ Two joinable JSON log records feed it, each written by both layers: | record (`msg`) | emitter | keys | |---|---|---| -| `Restore timing breakdown` | atelet, ateom-microvm and ateom-gvisor | `ate.actor.restore.duration.` / `ateom.actor.restore.duration.` | -| `Checkpoint timing breakdown` | atelet, ateom-microvm and ateom-gvisor | `ate.actor.checkpoint.duration.` / `ateom.actor.checkpoint.duration.` | +| `Restore timing breakdown` | atelet, ateworker-microvm and ateworker-gvisor | `ate.actor.restore.duration.` / `ateworker.actor.restore.duration.` | +| `Checkpoint timing breakdown` | atelet, ateworker-microvm and ateworker-gvisor | `ate.actor.checkpoint.duration.` / `ateworker.actor.checkpoint.duration.` | Every record carries the full actor identity (`ate.actor.uid`, name, atespace, template) and the snapshot scope, which the histograms are barred @@ -40,7 +40,7 @@ A failed atelet operation still writes its record, marked with `error.type` scripts. `--since-time` (or a relative `--since 30m`) should cover the run and nothing before it. `--namespace` (default `ate-system`) selects the atelet pods and `--worker-namespace` (default `benchmark-workloads`) the - worker pods; the ateom records land in the worker pod's stdout. `kubectl logs` + worker pods; the ateworker records land in the worker pod's stdout. `kubectl logs` returns only a container's current log file, so on a long or busy run kubelet's rotation can drop earlier records — collect soon after the run. A container that restarted mid-run is dumped twice (`.previous.log`). @@ -75,33 +75,33 @@ A failed atelet operation still writes its record, marked with `error.type` scope and phase: count, p50, p90, p95 and max. A `golden` restore downloads the golden image and a `latest` one the actor's own, so they are separate rows, as are gVisor and micro-VM operations. The atelet rows split a -checkpoint between `sandbox_assets`, `ateom_checkpoint` and `persist`, and a +checkpoint between `sandbox_assets`, `worker_checkpoint` and `persist`, and a restore between `volume_mount`, `manifest_fetch`, `sandbox_assets`, -`download`, `oci_unpack` and `ateom_restore`. The ateom rows split the -`ateom_*` phase further with the runtime's own phases: for micro-VM +`download`, `oci_unpack` and `worker_restore`. The ateworker rows split the +`worker_*` phase further with the runtime's own phases: for micro-VM `prep` / `pause` / `snapshot` / `durable_dir` / `rootfs_upper` / `teardown` on a checkpoint and `prep` / … / `vm_restore` / `wakeup_probe` on a restore; for gVisor `pause` / `checkpoint` / `durable_dir` / `resume` / `teardown` and `prep` / `net_setup` / … / `pause_restore` / `app_restore` / `wakeup_probe` / -`activate` (the full lists are in `cmd/ateom-*/phaselog.go`). The ateom +`activate` (the full lists are in `cmd/ateworker-*/phaselog.go`). The ateworker records carry no sandbox class or kind of their own; a paired one takes both from its atelet record, so both layers split into the same rows. Concurrency matters when reading them: the atelet restore phases overlap (the download runs alongside the asset fetch and OCI unpack), the three -ateom checkpoint captures run concurrently on the paused guest (the paused -window costs their max), and the ateom restore phases are sequential. For +ateworker checkpoint captures run concurrently on the paused guest (the paused +window costs their max), and the ateworker restore phases are sequential. For the two checkpoint layers the report derives an `unattributed` row: the total minus what the logged phases account for, counting the concurrent captures -once. It is the time the instrumentation does not yet name. The ateom +once. It is the time the instrumentation does not yet name. The ateworker records carry no snapshot kind of their own; a paired one takes its kind from the atelet record, so both layers split into the same rows. -**Waterfalls.** The slowest operations, with the ateom record of the same -actor nested under the atelet `ateom_*` phase and that phase's gap to the -ateom total (RPC and queueing between the layers). The ateom record is the +**Waterfalls.** The slowest operations, with the ateworker record of the same +actor nested under the atelet `worker_*` phase and that phase's gap to the +ateworker total (RPC and queueing between the layers). The ateworker record is the one written inside that phase's window (for a checkpoint, before `persist` -began), and each ateom record pairs with at most one operation, so an +began), and each ateworker record pairs with at most one operation, so an operation whose own record is missing prints without one rather than with another cycle's. Failed operations (`error.type` set) are left out of both the percentiles and the waterfalls. Tail outliers that blow up in one phase diff --git a/benchmarking/analysis/collect_logs.sh b/benchmarking/analysis/collect_logs.sh index e0555ca15b..fc483fdc62 100755 --- a/benchmarking/analysis/collect_logs.sh +++ b/benchmarking/analysis/collect_logs.sh @@ -15,7 +15,7 @@ # Dump the node-side logs a benchmark run needs for phase_report.py: every # atelet pod (the atelet-side timing breakdowns) and every worker pod -# (ateom-microvm writes its records to the worker pod's stdout). Point +# (ateworker-microvm writes its records to the worker pod's stdout). Point # --since-time (RFC 3339) or --since at the run's start so the report covers # exactly one run. # diff --git a/benchmarking/analysis/phase_report.py b/benchmarking/analysis/phase_report.py index 3d0e4d2f92..21d3860fdb 100755 --- a/benchmarking/analysis/phase_report.py +++ b/benchmarking/analysis/phase_report.py @@ -22,13 +22,13 @@ side emits, each written by both layers: - "Restore timing breakdown" atelet (ate.actor.restore.duration.*) - and ateom (ateom.actor.restore.duration.*) + and ateworker (ateworker.actor.restore.duration.*) - "Checkpoint timing breakdown" atelet (ate.actor.checkpoint.duration.*) - and ateom (ateom.actor.checkpoint.duration.*) + and ateworker (ateworker.actor.checkpoint.duration.*) The report answers "where does the SuspendActor / ResumeActor time go": per-phase percentiles at each layer, and the slowest operations as nested -waterfalls (the ateom record joined under the atelet record of the same +waterfalls (the ateworker record joined under the atelet record of the same actor). Usage: @@ -57,12 +57,12 @@ # The duration-key prefixes. Source of truth: cmd/atelet/metrics.go # (restoreDurationMetric, checkpointDurationMetric) and -# cmd/ateom-microvm/phaselog.go. +# cmd/ateworker-microvm/phaselog.go. BREAKDOWN_PREFIXES = { ("atelet", "restore"): "ate.actor.restore.duration.", ("atelet", "checkpoint"): "ate.actor.checkpoint.duration.", - ("ateom", "restore"): "ateom.actor.restore.duration.", - ("ateom", "checkpoint"): "ateom.actor.checkpoint.duration.", + ("ateworker", "restore"): "ateworker.actor.restore.duration.", + ("ateworker", "checkpoint"): "ateworker.actor.checkpoint.duration.", } BREAKDOWN_MSGS = {"Restore timing breakdown", "Checkpoint timing breakdown"} @@ -74,8 +74,8 @@ TEMPLATE_KEY = "ate.template.name" ERROR_TYPE_KEY = "error.type" -# The ateom records carry no sandbox class or snapshot kind of their own; -# both ateoms write the same two messages, and the atelet record of the same +# The ateworker records carry no sandbox class or snapshot kind of their own; +# both ateworkers write the same two messages, and the atelet record of the same # operation says which runtime it was. join_layers copies them across. MICROVM = "microvm" GVISOR = "gvisor" @@ -90,30 +90,30 @@ PHASE_ORDER = [ # atelet restore "volume_mount", "manifest_fetch", "sandbox_assets", "download", - "oci_unpack", "ateom_restore", - # ateom restore: micro-VM (cmd/ateom-microvm/phaselog.go) and gVisor - # (cmd/ateom-gvisor/phaselog.go), each in its own sequential order + "oci_unpack", "worker_restore", + # ateworker restore: micro-VM (cmd/ateworker-microvm/phaselog.go) and gVisor + # (cmd/ateworker-gvisor/phaselog.go), each in its own sequential order "prep", "egress_prepare", "net_setup", "bundles", "upper_join", "lowers", "pause_rootfs", "pause_create", "pause_restore", "app_rootfs", "app_create", "app_restore", "tap", "vmm_launch", "vm_restore", "resume", "wakeup_probe", "activate", - # atelet + ateom checkpoint + # atelet + ateworker checkpoint "pause", "checkpoint", "snapshot", "durable_dir", "rootfs_upper", - "ateom_checkpoint", "teardown", "persist", + "worker_checkpoint", "teardown", "persist", UNATTRIBUTED, TOTAL, ] _PHASE_RANK = {name: i for i, name in enumerate(PHASE_ORDER)} -# Which atelet phase wraps the ateom record. -INNER_PHASE = {"restore": "ateom_restore", "checkpoint": "ateom_checkpoint"} +# Which atelet phase wraps the ateworker record. +INNER_PHASE = {"restore": "worker_restore", "checkpoint": "worker_checkpoint"} # The micro-VM checkpoint captures that run concurrently on the paused guest: # the paused window costs their max, not their sum. gVisor's checkpoint phases # are sequential. CONCURRENT_CAPTURES = ("snapshot", "durable_dir", "rootfs_upper") -# Slack on the window in which the ateom record of an operation must land: -# atelet does a little work after the ateom call returns before it writes its +# Slack on the window in which the ateworker record of an operation must land: +# atelet does a little work after the ateworker call returns before it writes its # own record (unmounting, resetting the actor dirs), and the two clocks are # the same node's but not the same goroutine's. JOIN_SLACK_S = 1.0 @@ -122,7 +122,7 @@ @dataclass class Breakdown: """One parsed timing-breakdown record.""" - source: str # atelet | ateom + source: str # atelet | ateworker op: str # restore | checkpoint time: str ts: float | None # epoch seconds parsed from time, None when unparseable @@ -165,24 +165,24 @@ def parse_time(s: str) -> float | None: def unattributed(source: str, op: str, sandbox_class: str, phases: dict[str, float]) -> float | None: """The total minus what the logged phases account for, where that is well-defined: the atelet checkpoint phases are sequential; the micro-VM - ateom checkpoint is prep, pause, the concurrent captures (max), then - teardown; the gVisor ateom checkpoint phases are all sequential. The + ateworker checkpoint is prep, pause, the concurrent captures (max), then + teardown; the gVisor ateworker checkpoint phases are all sequential. The atelet restore phases overlap by design (download runs alongside the - asset fetch and OCI unpack) and both ateoms' restore phases partition + asset fetch and OCI unpack) and both ateworkers' restore phases partition their total by construction, so neither gets a residual; nor does an - ateom checkpoint whose runtime is unknown (its atelet record is missing). + ateworker checkpoint whose runtime is unknown (its atelet record is missing). The phases are sub-intervals of the total, so a negative result is a bug in the emitter or in this formula; it is not clamped, so that it shows.""" total = phases.get(TOTAL) if total is None: return None if (source, op) == ("atelet", "checkpoint"): - spent = sum(phases.get(p, 0.0) for p in ("sandbox_assets", "ateom_checkpoint", "persist")) - elif (source, op, sandbox_class) == ("ateom", "checkpoint", MICROVM): + spent = sum(phases.get(p, 0.0) for p in ("sandbox_assets", "worker_checkpoint", "persist")) + elif (source, op, sandbox_class) == ("ateworker", "checkpoint", MICROVM): spent = (phases.get("prep", 0.0) + phases.get("pause", 0.0) + max(phases.get(p, 0.0) for p in CONCURRENT_CAPTURES) + phases.get("teardown", 0.0)) - elif (source, op, sandbox_class) == ("ateom", "checkpoint", GVISOR): + elif (source, op, sandbox_class) == ("ateworker", "checkpoint", GVISOR): spent = sum(v for k, v in phases.items() if k != TOTAL) else: return None @@ -331,12 +331,12 @@ def report_phases(breakdowns: list[Breakdown], writer) -> list[dict]: return rows -def ateom_window(op: Breakdown) -> tuple[float, float] | None: - """When, relative to the atelet record's time, the ateom record of the - same operation was written. ateom writes it as its RPC returns, i.e. at - the end of the atelet ateom_* phase: for a checkpoint that is before +def ateworker_window(op: Breakdown) -> tuple[float, float] | None: + """When, relative to the atelet record's time, the ateworker record of the + same operation was written. ateworker writes it as its RPC returns, i.e. at + the end of the atelet worker_* phase: for a checkpoint that is before persist runs, for a restore it is the last phase. An operation that never - reached the ateom call has no window and nothing to pair with.""" + reached the ateworker call has no window and nothing to pair with.""" total = op.phases.get(TOTAL) if op.ts is None or total is None or INNER_PHASE[op.op] not in op.phases: return None @@ -347,21 +347,21 @@ def ateom_window(op: Breakdown) -> tuple[float, float] | None: def pair_records(breakdowns: list[Breakdown]) -> dict[int, Breakdown]: - """Match each atelet record to the ateom record of the same operation: - same actor and op, written inside the atelet ateom_* phase's window, and - not already claimed by another operation. Returns id(atelet) -> ateom. + """Match each atelet record to the ateworker record of the same operation: + same actor and op, written inside the atelet worker_* phase's window, and + not already claimed by another operation. Returns id(atelet) -> ateworker. - One cycle emits exactly one of each. An ateom record outside the window + One cycle emits exactly one of each. An ateworker record outside the window belongs to another cycle whose partner is missing (pod gone, log rotated, window cut); claiming it would print a gap that never happened, so an operation without a match prints without one. Matching in time order with a consumed set keeps rapid cycles of one actor from sharing or - swapping records. A paired ateom record also inherits the snapshot kind + swapping records. A paired ateworker record also inherits the snapshot kind and sandbox class its layer does not log, so the two layers' percentiles split alike and the residual knows which runtime's rule applies.""" by_actor: dict[tuple, list[Breakdown]] = defaultdict(list) for b in breakdowns: - if b.source == "ateom" and b.ts is not None: + if b.source == "ateworker" and b.ts is not None: by_actor[(b.actor_uid, b.op)].append(b) for v in by_actor.values(): v.sort(key=lambda b: b.ts) @@ -371,7 +371,7 @@ def pair_records(breakdowns: list[Breakdown]) -> dict[int, Breakdown]: atelet = sorted((b for b in breakdowns if b.source == "atelet" and b.ts is not None), key=lambda b: b.ts) for op in atelet: - window = ateom_window(op) + window = ateworker_window(op) if window is None: continue lo, hi = window @@ -391,7 +391,7 @@ def pair_records(breakdowns: list[Breakdown]) -> dict[int, Breakdown]: def join_layers(breakdowns: list[Breakdown]) -> dict[int, Breakdown]: """Pair the layers' records, then derive each record's unattributed - residual, which for an ateom checkpoint depends on the runtime the pairing + residual, which for an ateworker checkpoint depends on the runtime the pairing just identified. Returns the pairs, as pair_records does.""" pairs = pair_records(breakdowns) for b in breakdowns: @@ -403,8 +403,8 @@ def join_layers(breakdowns: list[Breakdown]) -> dict[int, Breakdown]: def report_waterfalls(breakdowns: list[Breakdown], writer, slowest: int, pairs: dict[int, Breakdown] | None = None) -> None: - """The slowest operations: the ateom record nested under the atelet - ateom_* phase, and the gap between that phase and the ateom total (RPC + """The slowest operations: the ateworker record nested under the atelet + worker_* phase, and the gap between that phase and the ateworker total (RPC and queueing between the layers).""" if pairs is None: pairs = join_layers(breakdowns) @@ -427,7 +427,7 @@ def report_waterfalls(breakdowns: list[Breakdown], writer, slowest: int, for iname in sorted(inner.phases, key=phase_sort_key): if iname == TOTAL: continue - writer(f" ateom {iname:13} {fmt_s(inner.phases[iname])}") + writer(f" ateworker {iname:13} {fmt_s(inner.phases[iname])}") if TOTAL in inner.phases: gap = b.phases[name] - inner.phases[TOTAL] writer(f" (gap) {'rpc/queueing':13} {fmt_s(gap)}") @@ -489,7 +489,7 @@ def main() -> int: print("no matching records; are these the atelet and worker pod logs?", file=sys.stderr) return 1 - # Pair before the percentiles: a paired ateom record takes its kind and + # Pair before the percentiles: a paired ateworker record takes its kind and # sandbox class from the atelet record, so both layers' rows split the # same way and each checkpoint's residual uses its runtime's rule. pairs = join_layers(parsed.breakdowns) diff --git a/benchmarking/analysis/test_phase_report.py b/benchmarking/analysis/test_phase_report.py index 0ad518f77b..386bbb4f72 100644 --- a/benchmarking/analysis/test_phase_report.py +++ b/benchmarking/analysis/test_phase_report.py @@ -49,18 +49,18 @@ def render(fn, *args, **kwargs): "ate.actor.uid": "uid-1", "ate.template.name": "swebench-astropy-7336", "ate.snapshot.scope": "full", "ate.snapshot.kind": "latest", "ate.actor.restore.duration.download": 2.4, - "ate.actor.restore.duration.ateom_restore": 1.1, + "ate.actor.restore.duration.worker_restore": 1.1, "ate.actor.restore.duration.total": 3.9, } -ATEOM_RESTORE = { +ATEWORKER_RESTORE = { "time": "2026-09-23T10:00:05.400000000Z", "level": "INFO", "msg": "Restore timing breakdown", "ate.actor.uid": "uid-1", "ate.template.name": "swebench-astropy-7336", "ate.snapshot.scope": "full", - "ateom.actor.restore.duration.vm_restore": 0.8, - "ateom.actor.restore.duration.wakeup_probe": 0.2, - "ateom.actor.restore.duration.total": 1.05, + "ateworker.actor.restore.duration.vm_restore": 0.8, + "ateworker.actor.restore.duration.wakeup_probe": 0.2, + "ateworker.actor.restore.duration.total": 1.05, } ATELET_CHECKPOINT = { @@ -69,27 +69,27 @@ def render(fn, *args, **kwargs): "ate.actor.uid": "uid-1", "ate.template.name": "swebench-astropy-7336", "ate.snapshot.scope": "full", "ate.snapshot.kind": "latest", "ate.actor.checkpoint.duration.sandbox_assets": 0.01, - "ate.actor.checkpoint.duration.ateom_checkpoint": 1.14, + "ate.actor.checkpoint.duration.worker_checkpoint": 1.14, "ate.actor.checkpoint.duration.persist": 4.48, "ate.actor.checkpoint.duration.total": 6.0, } -ATEOM_CHECKPOINT = { +ATEWORKER_CHECKPOINT = { "time": "2026-09-23T10:00:55.000000000Z", "level": "INFO", "msg": "Checkpoint timing breakdown", "ate.actor.uid": "uid-1", "ate.template.name": "swebench-astropy-7336", "ate.snapshot.scope": "full", - "ateom.actor.checkpoint.duration.prep": 0.04, - "ateom.actor.checkpoint.duration.pause": 0.01, - "ateom.actor.checkpoint.duration.snapshot": 0.5, - "ateom.actor.checkpoint.duration.rootfs_upper": 0.9, - "ateom.actor.checkpoint.duration.teardown": 0.2, - "ateom.actor.checkpoint.duration.total": 1.2, + "ateworker.actor.checkpoint.duration.prep": 0.04, + "ateworker.actor.checkpoint.duration.pause": 0.01, + "ateworker.actor.checkpoint.duration.snapshot": 0.5, + "ateworker.actor.checkpoint.duration.rootfs_upper": 0.9, + "ateworker.actor.checkpoint.duration.teardown": 0.2, + "ateworker.actor.checkpoint.duration.total": 1.2, } -# The same actor's ateom restore record from an hour earlier: a different +# The same actor's ateworker restore record from an hour earlier: a different # cycle, whose own atelet partner is missing from the dump. -ATEOM_RESTORE_STALE = dict(ATEOM_RESTORE, time="2026-09-23T09:00:05.400000000Z") +ATEWORKER_RESTORE_STALE = dict(ATEWORKER_RESTORE, time="2026-09-23T09:00:05.400000000Z") # One entry as `gcloud logging read --format json` writes it, trimmed to the # fields that matter: the record under jsonPayload, slog's time promoted to @@ -97,7 +97,7 @@ def render(fn, *args, **kwargs): CLOUD_LOGGING_ENTRY = { "insertId": "m9xljnzmzppup1vj", "jsonPayload": { - "ate.actor.checkpoint.duration.ateom_checkpoint": 1.024788329, + "ate.actor.checkpoint.duration.worker_checkpoint": 1.024788329, "ate.actor.checkpoint.duration.persist": 4.18451256, "ate.actor.checkpoint.duration.sandbox_assets": 2.4358e-05, "ate.actor.checkpoint.duration.total": 5.385861508, @@ -126,9 +126,9 @@ def render(fn, *args, **kwargs): class ParseTest(unittest.TestCase): def test_parses_both_layers_of_the_same_msg(self): - out = parse(ATELET_RESTORE, ATEOM_RESTORE) + out = parse(ATELET_RESTORE, ATEWORKER_RESTORE) self.assertEqual([(b.source, b.op) for b in out.breakdowns], - [("atelet", "restore"), ("ateom", "restore")]) + [("atelet", "restore"), ("ateworker", "restore")]) self.assertEqual(out.breakdowns[0].phases["download"], 2.4) self.assertEqual(out.breakdowns[1].phases["vm_restore"], 0.8) self.assertEqual(out.breakdowns[1].actor_uid, "uid-1") @@ -140,9 +140,9 @@ def test_nanosecond_timestamps_parse(self): self.assertAlmostEqual(out.breakdowns[0].ts % 1, 0.5, places=6) self.assertIsNone(phase_report.parse_time("yesterday")) - def test_ateom_checkpoint_parses(self): - b = parse(ATEOM_CHECKPOINT).breakdowns[0] - self.assertEqual((b.source, b.op), ("ateom", "checkpoint")) + def test_ateworker_checkpoint_parses(self): + b = parse(ATEWORKER_CHECKPOINT).breakdowns[0] + self.assertEqual((b.source, b.op), ("ateworker", "checkpoint")) self.assertEqual(b.phases["rootfs_upper"], 0.9) def test_cloud_logging_entry_parses_with_the_entry_timestamp(self): @@ -162,15 +162,15 @@ def test_cloud_logging_export_file_is_a_json_array(self): out = phase_report.parse_files([path]) self.assertEqual((out.lines_seen, len(out.breakdowns)), (2, 2)) - def test_ateom_records_take_class_and_kind_from_the_paired_atelet_record(self): + def test_ateworker_records_take_class_and_kind_from_the_paired_atelet_record(self): atelet = dict(ATELET_RESTORE, **{"ate.sandbox.class": "gvisor"}) - out = parse(ATEOM_RESTORE, atelet) - self.assertEqual(out.breakdowns[0].sandbox_class, "") # not on the ateom record + out = parse(ATEWORKER_RESTORE, atelet) + self.assertEqual(out.breakdowns[0].sandbox_class, "") # not on the ateworker record phase_report.join_layers(out.breakdowns) self.assertEqual((out.breakdowns[0].sandbox_class, out.breakdowns[0].kind), ("gvisor", "latest")) - def test_an_unpaired_ateom_record_keeps_no_class(self): - out = parse(ATEOM_RESTORE) + def test_an_unpaired_ateworker_record_keeps_no_class(self): + out = parse(ATEWORKER_RESTORE) phase_report.join_layers(out.breakdowns) self.assertEqual(out.breakdowns[0].sandbox_class, "") rows = phase_report.report_phases(out.breakdowns, lambda _: None) @@ -202,7 +202,7 @@ def test_bracket_prefixed_dump_is_read_line_by_line(self): path = os.path.join(d, "pod.log") with open(path, "w") as f: f.write("[INFO] " + json.dumps(ATELET_RESTORE) + "\n" - "[2026-09-23 10:00:05] " + json.dumps(ATEOM_RESTORE) + "\n") + "[2026-09-23 10:00:05] " + json.dumps(ATEWORKER_RESTORE) + "\n") err = io.StringIO() with contextlib.redirect_stderr(err): out = phase_report.parse_files([path]) @@ -212,7 +212,7 @@ def test_bracket_prefixed_dump_is_read_line_by_line(self): def test_parse_lines_feeds_an_existing_parse(self): out = phase_report.parse_lines([json.dumps(ATELET_RESTORE), "not json", ""]) self.assertEqual((out.lines_seen, len(out.breakdowns)), (1, 1)) - phase_report.parse_lines(["pod/x " + json.dumps(ATEOM_RESTORE)], out) + phase_report.parse_lines(["pod/x " + json.dumps(ATEWORKER_RESTORE)], out) self.assertEqual(len(out.breakdowns), 2) def test_prefixed_kubectl_lines_still_parse(self): @@ -230,53 +230,53 @@ def test_atelet_checkpoint_residual_is_total_minus_sequential_phases(self): out = parse(ATELET_CHECKPOINT) phase_report.join_layers(out.breakdowns) self.assertAlmostEqual(out.breakdowns[0].phases["unattributed"], 6.0 - (0.01 + 1.14 + 4.48)) - def test_microvm_ateom_checkpoint_counts_the_concurrent_captures_once(self): + def test_microvm_ateworker_checkpoint_counts_the_concurrent_captures_once(self): # prep + pause + max(snapshot, rootfs_upper) + teardown; the two # captures overlapped, so only the slower one is spent wall time. - phases = {k[len("ateom.actor.checkpoint.duration."):]: v - for k, v in ATEOM_CHECKPOINT.items() if k.startswith("ateom.actor.checkpoint.duration.")} - got = phase_report.unattributed("ateom", "checkpoint", "microvm", phases) + phases = {k[len("ateworker.actor.checkpoint.duration."):]: v + for k, v in ATEWORKER_CHECKPOINT.items() if k.startswith("ateworker.actor.checkpoint.duration.")} + got = phase_report.unattributed("ateworker", "checkpoint", "microvm", phases) self.assertAlmostEqual(got, 1.2 - (0.04 + 0.01 + 0.9 + 0.2)) - def test_gvisor_ateom_checkpoint_phases_are_sequential(self): + def test_gvisor_ateworker_checkpoint_phases_are_sequential(self): phases = {"prep": 0.01, "pause": 0.02, "checkpoint": 0.15, "durable_dir": 0.03, "resume": 0.02, "teardown": 0.05, "total": 0.30} - got = phase_report.unattributed("ateom", "checkpoint", "gvisor", phases) + got = phase_report.unattributed("ateworker", "checkpoint", "gvisor", phases) self.assertAlmostEqual(got, 0.30 - (0.01 + 0.02 + 0.15 + 0.03 + 0.02 + 0.05)) - def test_ateom_checkpoint_of_unknown_runtime_gets_no_residual(self): - out = parse(ATEOM_CHECKPOINT) # no atelet partner, so no class + def test_ateworker_checkpoint_of_unknown_runtime_gets_no_residual(self): + out = parse(ATEWORKER_CHECKPOINT) # no atelet partner, so no class phase_report.join_layers(out.breakdowns) self.assertNotIn("unattributed", out.breakdowns[0].phases) def test_gvisor_pair_splits_both_layers_by_class(self): atelet = dict(ATELET_CHECKPOINT, **{"ate.sandbox.class": "gvisor"}) - ateom = { + ateworker = { "time": "2026-09-23T10:00:55.000000000Z", "msg": "Checkpoint timing breakdown", "ate.actor.uid": "uid-1", "ate.template.name": "swebench-astropy-7336", "ate.snapshot.scope": "full", - "ateom.actor.checkpoint.duration.pause": 0.02, - "ateom.actor.checkpoint.duration.checkpoint": 0.15, - "ateom.actor.checkpoint.duration.teardown": 0.05, - "ateom.actor.checkpoint.duration.total": 0.25, + "ateworker.actor.checkpoint.duration.pause": 0.02, + "ateworker.actor.checkpoint.duration.checkpoint": 0.15, + "ateworker.actor.checkpoint.duration.teardown": 0.05, + "ateworker.actor.checkpoint.duration.total": 0.25, } - out = parse(ateom, atelet) + out = parse(ateworker, atelet) phase_report.join_layers(out.breakdowns) rows = phase_report.report_phases(out.breakdowns, lambda _: None) - ateom_rows = {r["phase"]: r for r in rows if r["layer"] == "ateom"} - self.assertEqual({r["class"] for r in ateom_rows.values()}, {"gvisor"}) - self.assertEqual({r["kind"] for r in ateom_rows.values()}, {"latest"}) - self.assertAlmostEqual(ateom_rows["unattributed"]["p50_ms"], (0.25 - 0.22) * 1000) + ateworker_rows = {r["phase"]: r for r in rows if r["layer"] == "ateworker"} + self.assertEqual({r["class"] for r in ateworker_rows.values()}, {"gvisor"}) + self.assertEqual({r["kind"] for r in ateworker_rows.values()}, {"latest"}) + self.assertAlmostEqual(ateworker_rows["unattributed"]["p50_ms"], (0.25 - 0.22) * 1000) # gVisor's phases sort in their own order, before the shared tail. - order = [r["phase"] for r in rows if r["layer"] == "ateom"] + order = [r["phase"] for r in rows if r["layer"] == "ateworker"] self.assertEqual(order, ["pause", "checkpoint", "teardown", "unattributed", "total"]) def test_restore_layers_get_no_residual(self): - out = parse(ATELET_RESTORE, ATEOM_RESTORE) + out = parse(ATELET_RESTORE, ATEWORKER_RESTORE) phase_report.join_layers(out.breakdowns) for b in out.breakdowns: self.assertNotIn("unattributed", b.phases) def test_negative_residual_is_not_clamped(self): # Cannot happen on a well-formed record; if it does, it must show. - phases = {"sandbox_assets": 0.01, "ateom_checkpoint": 1.14, "persist": 4.48, "total": 1.0} + phases = {"sandbox_assets": 0.01, "worker_checkpoint": 1.14, "persist": 4.48, "total": 1.0} self.assertAlmostEqual(phase_report.unattributed("atelet", "checkpoint", "microvm", phases), 1.0 - (0.01 + 1.14 + 4.48)) @@ -299,37 +299,37 @@ def test_percentiles_split_by_sandbox_class_and_kind(self): downloads = {(r["class"], r["kind"]): r["p50_ms"] for r in rows if r["phase"] == "download"} self.assertEqual(downloads, {("gvisor", "golden"): 9000.0, ("gvisor", "latest"): 2400.0}) - def test_paired_ateom_record_inherits_the_atelet_kind(self): - out = parse(ATEOM_RESTORE, ATELET_RESTORE) + def test_paired_ateworker_record_inherits_the_atelet_kind(self): + out = parse(ATEWORKER_RESTORE, ATELET_RESTORE) pairs = phase_report.pair_records(out.breakdowns) self.assertEqual(len(pairs), 1) self.assertEqual(out.breakdowns[0].kind, "latest") rows = phase_report.report_phases(out.breakdowns, lambda _: None) - self.assertEqual({r["kind"] for r in rows if r["layer"] == "ateom"}, {"latest"}) + self.assertEqual({r["kind"] for r in rows if r["layer"] == "ateworker"}, {"latest"}) def test_checkpoint_pairing_window_ends_where_persist_starts(self): - # The ateom record is written when ateom_checkpoint ends, before the + # The ateworker record is written when worker_checkpoint ends, before the # 4.48 s persist; a record from inside the persist window belongs to # a later cycle of a rapidly cycling actor. - during_persist = dict(ATEOM_CHECKPOINT, time="2026-09-23T10:00:59.000000000Z") - out = parse(ATEOM_CHECKPOINT, during_persist, ATELET_CHECKPOINT) + during_persist = dict(ATEWORKER_CHECKPOINT, time="2026-09-23T10:00:59.000000000Z") + out = parse(ATEWORKER_CHECKPOINT, during_persist, ATELET_CHECKPOINT) pairs = phase_report.pair_records(out.breakdowns) atelet = next(b for b in out.breakdowns if b.source == "atelet") self.assertIs(pairs[id(atelet)], out.breakdowns[0]) - def test_an_ateom_record_pairs_with_at_most_one_operation(self): + def test_an_ateworker_record_pairs_with_at_most_one_operation(self): second = dict(ATELET_RESTORE, time="2026-09-23T10:00:05.900000000Z") - out = parse(ATEOM_RESTORE, ATELET_RESTORE, second) + out = parse(ATEWORKER_RESTORE, ATELET_RESTORE, second) pairs = phase_report.pair_records(out.breakdowns) self.assertEqual(len(pairs), 1) first = next(b for b in out.breakdowns if b.source == "atelet") self.assertIn(id(first), pairs) - def test_waterfall_ignores_an_ateom_record_from_another_cycle(self): - out = parse(ATEOM_RESTORE_STALE, ATELET_RESTORE) + def test_waterfall_ignores_an_ateworker_record_from_another_cycle(self): + out = parse(ATEWORKER_RESTORE_STALE, ATELET_RESTORE) text = render(phase_report.report_waterfalls, out.breakdowns, slowest=1) - self.assertIn("atelet ateom_restore", text) - self.assertNotIn("ateom vm_restore", text) + self.assertIn("atelet worker_restore", text) + self.assertNotIn("ateworker vm_restore", text) self.assertNotIn("(gap)", text) def test_stats_rows_take_the_runner_jsonl_shape(self): @@ -345,15 +345,15 @@ def test_stats_rows_take_the_runner_jsonl_shape(self): "scope": "full", "phase": "download", "count": "1", "p50_ms": "2400.0", "p90_ms": "2400.0", "p95_ms": "2400.0", "max_ms": "2400.0"}) - def test_waterfall_nests_ateom_and_gap_under_atelet(self): - out = parse(ATEOM_RESTORE, ATELET_RESTORE) + def test_waterfall_nests_ateworker_and_gap_under_atelet(self): + out = parse(ATEWORKER_RESTORE, ATELET_RESTORE) text = render(phase_report.report_waterfalls, out.breakdowns, slowest=1) lines = [line.strip() for line in text.splitlines()] - self.assertTrue(any(line.startswith("atelet ateom_restore") for line in lines), text) - self.assertTrue(any(line.startswith("ateom vm_restore") for line in lines), text) + self.assertTrue(any(line.startswith("atelet worker_restore") for line in lines), text) + self.assertTrue(any(line.startswith("ateworker vm_restore") for line in lines), text) gap = [line for line in lines if line.startswith("(gap)")] self.assertEqual(len(gap), 1, text) - self.assertIn("50.0", gap[0]) # 1.1s atelet ateom_restore - 1.05s ateom total + self.assertIn("50.0", gap[0]) # 1.1s atelet worker_restore - 1.05s ateworker total if __name__ == "__main__": diff --git a/benchmarking/automation/README.md b/benchmarking/automation/README.md index a8f8aa6003..9a152df4f0 100644 --- a/benchmarking/automation/README.md +++ b/benchmarking/automation/README.md @@ -46,7 +46,7 @@ the router capacity benchmark — see Each entry in `tests.yaml` may set `sandboxClass: gvisor | microvm` (default `gvisor`). This controls both `spec.sandboxClasses[0].name` on the benchmark WorkerPool -and its `workerImage` (`ateom-gvisor` vs `ateom-microvm`). +and its `workerImage` (`ateworker-gvisor` vs `ateworker-microvm`). For `microvm` tests the target cluster must have KVM-capable nodes and the object store bucket named in its `.ate-dev-env.sh` must be writable by the diff --git a/benchmarking/automation/tests.yaml b/benchmarking/automation/tests.yaml index d254ed1223..b59f3734a8 100644 --- a/benchmarking/automation/tests.yaml +++ b/benchmarking/automation/tests.yaml @@ -44,7 +44,7 @@ # workerWaitTimeout # Optional (default 300, set by workloads/deploy.sh). # Whole seconds to wait after the WorkerPool deploy for -# the ateom worker pods to become ready. Raise it for +# the worker pods to become ready. Raise it for # large workerCount fleets whose image pulls or scheduling # outrun the default. # ateArgs Optional. List of strings appended verbatim to diff --git a/benchmarking/deploy_locust.sh b/benchmarking/deploy_locust.sh index a51d1e6ee5..a431870a76 100755 --- a/benchmarking/deploy_locust.sh +++ b/benchmarking/deploy_locust.sh @@ -54,7 +54,7 @@ usage() { echo " --storage-class-name S Forwarded to workloads/deploy.sh. StorageClass for the glutton-storage" echo " external volume; setting it deploys glutton-storage." echo " --wait-timeout SECONDS Forwarded to workloads/deploy.sh. The timeout in seconds for" - echo " waiting for the ateom workers to be ready (default: 300)" + echo " waiting for the workers to be ready (default: 300)" echo " --skip-build Skip locust image build/push (use the existing :latest image)" echo " -h|--help Show this help message" echo "" diff --git a/benchmarking/locust/phase_breakdown.py b/benchmarking/locust/phase_breakdown.py index ea2953e67c..ec914e9a7f 100644 --- a/benchmarking/locust/phase_breakdown.py +++ b/benchmarking/locust/phase_breakdown.py @@ -14,7 +14,7 @@ """Suspend/resume phase percentiles for stats.jsonl, read from the node logs. -atelet and ateom-microvm write one `Checkpoint timing breakdown` / `Restore +atelet and ateworker-microvm write one `Checkpoint timing breakdown` / `Restore timing breakdown` record per operation (see benchmarking/analysis/README.md). The orchestrator deletes the worker and ate-system pods right after a test, so the runner reads those pod logs itself, through the Kubernetes API it diff --git a/benchmarking/locust/server_telemetry.py b/benchmarking/locust/server_telemetry.py index 3db4837dcc..5ecd93ad00 100644 --- a/benchmarking/locust/server_telemetry.py +++ b/benchmarking/locust/server_telemetry.py @@ -222,9 +222,9 @@ def _query_window_quantile( ) -# Each ateom pod's whole cgroup, systemd or cgroupfs; drops the pause container. +# Each worker pod's whole cgroup, systemd or cgroupfs; drops the pause container. POD_PSI_SELECTOR = ( - 'pod=~"benchmark-ateom.*", container="", ' + 'pod=~"benchmark-worker.*", container="", ' 'id=~".*[/-]pod[^/]+(\\\\.slice)?"' ) @@ -314,7 +314,7 @@ def _harvest_cluster_packing( ) -> dict[str, Any]: """Workers with an actor over all workers, per sample and as percentiles.""" packing_query = ( - 'ate_workerpool_workers{ate_workerpool_name="benchmark-ateom"}' + 'ate_workerpool_workers{ate_workerpool_name="benchmark-worker"}' ) packing_series = query_prometheus_range( prom_url, packing_query, start_ts, end_ts, step="10s" diff --git a/benchmarking/locust/unit_tests/test_phase_breakdown.py b/benchmarking/locust/unit_tests/test_phase_breakdown.py index 52063ac0d2..cf93138817 100644 --- a/benchmarking/locust/unit_tests/test_phase_breakdown.py +++ b/benchmarking/locust/unit_tests/test_phase_breakdown.py @@ -40,14 +40,14 @@ "ate.actor.uid": "uid-1", "ate.template.name": "glutton", "ate.sandbox.class": "microvm", "ate.snapshot.scope": "full", "ate.snapshot.kind": "latest", "ate.actor.restore.duration.download": 2.4, - "ate.actor.restore.duration.ateom_restore": 1.1, + "ate.actor.restore.duration.worker_restore": 1.1, "ate.actor.restore.duration.total": 3.9, }) -ATEOM = json.dumps({ +ATEWORKER = json.dumps({ "time": "2026-09-23T10:00:05.400000000Z", "msg": "Restore timing breakdown", "ate.actor.uid": "uid-1", "ate.template.name": "glutton", "ate.snapshot.scope": "full", - "ateom.actor.restore.duration.vm_restore": 0.8, - "ateom.actor.restore.duration.total": 1.05, + "ateworker.actor.restore.duration.vm_restore": 0.8, + "ateworker.actor.restore.duration.total": 1.05, }) STATS_CSV = ("Type,Name,Request Count,Failure Count\n" "grpc,SuspendActor,0,0\ngrpc,ResumeActor,1,0\n,Aggregated,1,0\n") @@ -90,13 +90,13 @@ def append(api, stats=STATS_CSV): class PhaseBreakdownTest(unittest.TestCase): def test_rows_are_appended_after_the_locust_rows(self): - api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}) + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEWORKER}) n, entries, _ = append(api) self.assertEqual(entries[0]["metric"], "grpc_ResumeActor") # untouched metrics = {e["metric"] for e in entries[1:]} self.assertIn("phase_atelet_restore_microvm_latest_full_download", metrics) - # The ateom row took its kind from the paired atelet record. - self.assertIn("phase_ateom_restore_microvm_latest_full_vm_restore", metrics) + # The ateworker row took its kind from the paired atelet record. + self.assertIn("phase_ateworker_restore_microvm_latest_full_vm_restore", metrics) self.assertEqual(n, len(entries) - 1) for e in entries[1:]: self.assertEqual((e["timestamp"], e["tag"], e["test_name"]), @@ -107,7 +107,7 @@ def test_rows_are_appended_after_the_locust_rows(self): self.assertEqual((summary["measurements"]["pods_read"], summary["measurements"]["pods_failed"]), ("2", "0")) def test_fewer_records_than_successful_requests_is_warned(self): - api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}) + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEWORKER}) _, entries, output = append(api, stats=STATS_CSV.replace("ResumeActor,1,0", "ResumeActor,5,0")) self.assertIn("Warning: locust made 5 successful ResumeActor requests but the atelet logs hold 1", output) self.assertIn("a node log was rotated", output) @@ -116,18 +116,18 @@ def test_fewer_records_than_successful_requests_is_warned(self): def test_requests_that_never_reached_atelet_are_not_owed_a_record(self): # 5 requests, 4 failed client-side: only 1 successful one reached atelet. - api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}) + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEWORKER}) _, _, output = append(api, stats=STATS_CSV.replace("ResumeActor,1,0", "ResumeActor,5,4")) self.assertNotIn("Warning: locust made", output) def test_missing_records_blame_unreadable_pods_when_there_are_any(self): - api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}, broken_pod="ate-system-pod") + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEWORKER}, broken_pod="ate-system-pod") _, _, output = append(api) self.assertIn("1 pod(s) could not be read", output) self.assertNotIn("rotated", output) def test_a_missing_previous_log_keeps_the_current_one(self): - api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}, restarted=True, previous_missing=True) + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEWORKER}, restarted=True, previous_missing=True) _, entries, output = append(api) self.assertIn("no previous log for", output) self.assertTrue(any(e["metric"].startswith("phase_atelet_restore") for e in entries)) @@ -135,12 +135,12 @@ def test_a_missing_previous_log_keeps_the_current_one(self): self.assertEqual((summary["measurements"]["pods_read"], summary["measurements"]["pods_failed"]), ("2", "0")) def test_an_unreadable_pod_is_skipped_not_fatal(self): - api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEOM}, + api = fake_api({"ate-system": ATELET, "benchmark-workloads": ATEWORKER}, broken_pod="benchmark-workloads-pod") n, entries, output = append(api) self.assertIn("could not read logs of benchmark-workloads/benchmark-workloads-pod", output) self.assertTrue(any(e["metric"].startswith("phase_atelet_") for e in entries)) - self.assertFalse(any(e["metric"].startswith("phase_ateom_") for e in entries)) + self.assertFalse(any(e["metric"].startswith("phase_ateworker_") for e in entries)) def test_no_records_still_writes_the_summary_row(self): # A failed configuration must not look like an idle run. @@ -151,7 +151,7 @@ def test_no_records_still_writes_the_summary_row(self): self.assertIn("Warning: no timing breakdown records", output) def test_a_restarted_container_is_read_twice(self): - api = fake_api({"ate-system": "", "benchmark-workloads": ATEOM}, restarted=True, + api = fake_api({"ate-system": "", "benchmark-workloads": ATEWORKER}, restarted=True, previous_logs={"ate-system": ATELET}) _, entries, _ = append(api) self.assertTrue(any(e["metric"].startswith("phase_atelet_restore") for e in entries)) @@ -160,7 +160,7 @@ def test_a_restarted_container_is_read_twice(self): def test_failed_requests_do_not_trigger_the_rotation_warning(self): failed = ATELET.replace('"ate.snapshot.kind": "latest",', '"ate.snapshot.kind": "latest", "error.type": "DeadlineExceeded",') - api = fake_api({"ate-system": failed, "benchmark-workloads": ATEOM}) + api = fake_api({"ate-system": failed, "benchmark-workloads": ATEWORKER}) _, _, output = append(api) self.assertNotIn("Warning: locust made", output) diff --git a/benchmarking/locust/unit_tests/test_server_telemetry.py b/benchmarking/locust/unit_tests/test_server_telemetry.py index 7d52929d37..811a1b32c8 100644 --- a/benchmarking/locust/unit_tests/test_server_telemetry.py +++ b/benchmarking/locust/unit_tests/test_server_telemetry.py @@ -47,7 +47,7 @@ def ranges(packing, node=(), pod=(), active=()): regex (and would also sum the pause container) gets nothing back. Node PSI only answers when kept to nodes hosting a worker pod. """ - pod_slice = ('{pod=~"benchmark-ateom.*", container="", ' + pod_slice = ('{pod=~"benchmark-worker.*", container="", ' 'id=~".*[/-]pod[^/]+(\\\\.slice)?"}') worker_nodes = "and on (instance) sum by (instance) (rate(" @@ -238,7 +238,7 @@ def test_packing_and_checkpoint_math(self, mock_instant, mock_range): "values": [[100, "1.0"], [105, "1.0"]]} quiet = [{"metric": {"instance": f"node-{n}"}, "values": [[100, "0.0"], [105, "0.0"]]} for n in "ab"] - pods = [{"metric": {"pod": f"benchmark-ateom-{p}"}, + pods = [{"metric": {"pod": f"benchmark-worker-{p}"}, "values": [[100, "0.5"], [105, "0.5"]]} for p in "ab"] mock_range.side_effect = ranges( [partial, full, idle], node=quiet, pod=pods, @@ -452,7 +452,7 @@ def test_steady_window_bounds_packing_and_psi(self, mock_instant, mock_range): "values": [[100, "1.0"], [105, "1.0"], [110, "5.0"]]}], node=[{"metric": {"instance": "node-a"}, "values": [[100, "0.5"], [110, "9.0"]]}], - pod=[{"metric": {"pod": "benchmark-ateom-a"}, + pod=[{"metric": {"pod": "benchmark-worker-a"}, "values": [[100, "0.5"], [110, "9.0"]]}], ) mock_instant.return_value = [] diff --git a/benchmarking/nighthawk-egress/README.md b/benchmarking/nighthawk-egress/README.md index 2a14dfc04c..a0796bae9f 100644 --- a/benchmarking/nighthawk-egress/README.md +++ b/benchmarking/nighthawk-egress/README.md @@ -10,7 +10,7 @@ flowchart LR invoker["traffic-driver-invoker
(:19980, adds ate-target-actor)"] router["atenet-router
(:80)"] - subgraph worker["WorkerPod (ateom-gvisor)"] + subgraph worker["WorkerPod (ateworker-gvisor)"] atunnel-ingress["ingress atunnel"] actor["Actor: nighthawk/tester
(nighthawk_service :80)"] atunnel-egress["egress atunnel"] diff --git a/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl b/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl index 0c607fad69..7dc71a7ea5 100644 --- a/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl +++ b/benchmarking/nighthawk-egress/traffic-driver-template.yaml.tmpl @@ -58,6 +58,13 @@ volumes: - egress-mitm.ate.dev - system-roots.ate.dev path: egress-trust-bundle/trust-bundle.pem +# Sandbox size: headroom so the load generator is not the bottleneck. +resources: + limits: + - name: cpu + quantity: "2" + - name: memory + quantity: 1Gi sandboxConfig: sandboxClass: gvisor configName: gvisor-default diff --git a/benchmarking/nighthawk-egress/traffic-driver.yaml.tmpl b/benchmarking/nighthawk-egress/traffic-driver.yaml.tmpl index 155df4dc9a..3e08db74af 100644 --- a/benchmarking/nighthawk-egress/traffic-driver.yaml.tmpl +++ b/benchmarking/nighthawk-egress/traffic-driver.yaml.tmpl @@ -33,6 +33,6 @@ metadata: workload: traffic-driver spec: replicas: 1 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor diff --git a/benchmarking/telemetry/README.md b/benchmarking/telemetry/README.md index 18d3c27a5d..c5aa2c431b 100644 --- a/benchmarking/telemetry/README.md +++ b/benchmarking/telemetry/README.md @@ -43,7 +43,7 @@ METER=http://telemetry-meter.benchmarking.svc.cluster.local:4317 `--otlp-endpoint` patches the `ate-otel-config` ConfigMap and restarts the workloads that read it. One patch is sufficient for `ateapi`, `ate-controller`, `atelet`, and `atenet-router`, because each one reads the ConfigMap through -`envFrom`. `ate-controller` also copies the values to the ateom worker pods +`envFrom`. `ate-controller` also copies the values to the worker pods that it creates. The actor containers are different. Substrate puts no OTLP configuration in diff --git a/benchmarking/workloads/deploy.sh b/benchmarking/workloads/deploy.sh index 113094967f..9fe68389bc 100755 --- a/benchmarking/workloads/deploy.sh +++ b/benchmarking/workloads/deploy.sh @@ -48,8 +48,8 @@ WORKER_COUNT=1 SANDBOX_CLASS="gvisor" # Actor memory limit (ActorTemplate resources.limits.memory). The default # is the smallest size microvm admits (128Mi VMM reserve + 128Mi guest floor), -# so benchmark actors do not inherit the 2 GiB kata default and drag its page -# cache into every memory snapshot. Raise it for RAM-consuming suites. +# since a bigger guest drags more page cache into every memory snapshot. Raise +# it for RAM-consuming suites. ACTOR_MEMORY="256Mi" # Worker pod memory (WorkerPool spec.template.resources), set as both request # and limit. Empty leaves the pod unsized. The limit is also the memory the @@ -62,7 +62,7 @@ STORAGE_CLASS_NAME="${STORAGE_CLASS_NAME:-}" # --otlp-endpoint sets it. Without the flag, resolve_otlp_endpoint reads the # address that the control plane uses. OTLP_ENDPOINT="" -# The timeout, in whole seconds, for waiting for the ateom worker pods to be +# The timeout, in whole seconds, for waiting for the worker pods to be # ready. WAIT_TIMEOUT_SECS=300 # fake-atelet's tuning on the fake data plane, as FAKE_ATELET_=VALUE @@ -87,7 +87,7 @@ usage() { echo " --otlp-endpoint URL The address to which an instrumented actor container" echo " sends telemetry (default: the endpoint in the" echo " ate-otel-config ConfigMap)" - echo " --wait-timeout SECONDS The timeout in seconds for waiting for the ateom workers to be ready (default: 300)" + echo " --wait-timeout SECONDS The timeout in seconds for waiting for the workers to be ready (default: 300)" echo " --fake-atelet-env NAME=VALUE" echo " On a substrate installed with --fake-data-plane, set a fake-atelet flag:" echo " FAKE_ATELET_DELAY=500ms sets --delay, FAKE_ATELET_DELAY_RESTORE=2s sets" @@ -257,13 +257,13 @@ deploy() { # ready once its capacity is reported, and writes that to the pool's # status. if ((WORKER_COUNT > 0)); then - kubectl wait --for=jsonpath='{.status.readyReplicas}'="${WORKER_COUNT}" workerpool/benchmark-ateom \ + kubectl wait --for=jsonpath='{.status.readyReplicas}'="${WORKER_COUNT}" workerpool/benchmark-worker \ --namespace=benchmark-workloads --timeout="${WAIT_TIMEOUT_SECS}s" fi else - kubectl wait --for=create deployment/benchmark-ateom \ + kubectl wait --for=create deployment/benchmark-worker \ --namespace=benchmark-workloads --timeout="${WAIT_TIMEOUT_SECS}s" - kubectl rollout status deployment/benchmark-ateom \ + kubectl rollout status deployment/benchmark-worker \ --namespace=benchmark-workloads --timeout="${WAIT_TIMEOUT_SECS}s" fi diff --git a/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl index 8ba59e96b2..d77db84139 100644 --- a/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-durdir-data-template.yaml.tmpl @@ -21,7 +21,7 @@ metadata: name: glutton-durdir-data workerSelector: matchLabels: - workload: benchmark-ateom + workload: benchmark-worker containers: - name: glutton image: ko://github.com/agent-substrate/substrate/cmd/benchmarking/glutton @@ -43,6 +43,13 @@ containers: volumes: - name: data durableDir: {} +# Sandbox size: see the note on the sleep template. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: ${ACTOR_MEMORY} snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton-durdir-data/ diff --git a/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl index febfad850d..a4860bfc47 100644 --- a/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-durdir-full-template.yaml.tmpl @@ -20,7 +20,7 @@ metadata: name: glutton-durdir-full workerSelector: matchLabels: - workload: benchmark-ateom + workload: benchmark-worker containers: - name: glutton image: ko://github.com/agent-substrate/substrate/cmd/benchmarking/glutton @@ -40,6 +40,13 @@ containers: volumes: - name: data durableDir: {} +# Sandbox size: see the note on the sleep template. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: ${ACTOR_MEMORY} snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/glutton-durdir-full/ diff --git a/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl index 92e776603b..fee943693e 100644 --- a/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-storage-template.yaml.tmpl @@ -21,7 +21,7 @@ metadata: name: glutton-storage workerSelector: matchLabels: - workload: benchmark-ateom + workload: benchmark-worker volumes: - name: external-storage externalVolumeTemplate: @@ -52,6 +52,8 @@ containers: port: 80 resources: limits: + - name: cpu + quantity: "1" - name: memory quantity: ${ACTOR_MEMORY} snapshotConfig: diff --git a/benchmarking/workloads/manifests/glutton-template.yaml.tmpl b/benchmarking/workloads/manifests/glutton-template.yaml.tmpl index 8babdd0516..dc249366db 100644 --- a/benchmarking/workloads/manifests/glutton-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/glutton-template.yaml.tmpl @@ -21,12 +21,12 @@ metadata: name: glutton workerSelector: matchLabels: - workload: benchmark-ateom + workload: benchmark-worker containers: - name: glutton image: ko://github.com/agent-substrate/substrate/cmd/benchmarking/glutton # Port 80 is the only inbound port nftables forwards from the worker pod - # into the actor sandbox (see cmd/ateom-gvisor/main.go). HTTP/1.1 mode + # into the actor sandbox (see cmd/ateworker-gvisor/main.go). HTTP/1.1 mode # lets the atenet router (which forwards HTTP/1.1 upstream) reach Ping. command: - "/ko-app/glutton" @@ -53,11 +53,12 @@ containers: httpGet: path: /readyz port: 80 -# See the note on the sleep template: unset means the 2 GiB microvm default, -# which the benchmarks should not silently inherit. Raise ${ACTOR_MEMORY} -# for suites that exercise glutton's WriteRAM path. +# See the note on the sleep template. Raise ${ACTOR_MEMORY} for suites that +# exercise glutton's WriteRAM path. resources: limits: + - name: cpu + quantity: "1" - name: memory quantity: ${ACTOR_MEMORY} snapshotConfig: diff --git a/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl b/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl index 013d5667af..b39d7bb26a 100644 --- a/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/kernelmem-template.yaml.tmpl @@ -16,20 +16,28 @@ # benchmark workload (locust/tests/kernelmem.py): fills kernel memory with # dentry cache and sleeping processes to size kernel-state snapshots. Not # part of deploy.sh's default template set; deploy with -# WORKLOAD_TEMPLATES="kernelmem ...". No memory limit: the 50000 sleeping -# processes need more than the 256Mi default --actor-memory the other -# templates take. The sandbox class must match the WorkerPool's so snapshots -# stay within a class. +# WORKLOAD_TEMPLATES="kernelmem ...". Its memory limit is fixed rather than +# --actor-memory: the 50000 sleeping processes need far more than the 256Mi +# default. The sandbox class must match the WorkerPool's so snapshots stay +# within a class. metadata: atespace: benchmark-workloads name: kernelmem workerSelector: matchLabels: - workload: benchmark-ateom + workload: benchmark-worker containers: - name: kernelmem image: containerstack/alpine-stress:latest command: ["sh", "-c", "find /; for i in $(seq 50000); do sleep 10000 & done; sleep 10000"] +# Sandbox size: the 50000 sleeping processes take ~6.3GiB on Linux (~130KB +# each, mostly page tables and kernel state). +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 8Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/kernelmem/ diff --git a/benchmarking/workloads/manifests/sleep-template.yaml.tmpl b/benchmarking/workloads/manifests/sleep-template.yaml.tmpl index 99e5c7f352..72becd7126 100644 --- a/benchmarking/workloads/manifests/sleep-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/sleep-template.yaml.tmpl @@ -21,17 +21,19 @@ metadata: name: sleep workerSelector: matchLabels: - workload: benchmark-ateom + workload: benchmark-worker containers: - name: sleep image: busybox@sha256:1487d0af5f52b4ba31c7e465126ee2123fe3f2305d638e7827681e7cf6c83d5e command: ["/bin/sleep", "1000000"] -# Without a limit a microvm actor gets the kata default 2 GiB guest, and -# anything the guest kernel caches rides along in the memory snapshot — -# bigger guest, bigger snapshot, slower suspend/resume. microvm requires at -# least 256Mi (128Mi VMM reserve + 128Mi guest floor). +# Keep the memory small: anything the micro-VM guest kernel caches rides +# along in the memory snapshot, so a bigger guest means slower +# suspend/resume. microvm requires at least 256Mi (128Mi VMM reserve + +# 128Mi guest floor). resources: limits: + - name: cpu + quantity: "1" - name: memory quantity: ${ACTOR_MEMORY} snapshotConfig: diff --git a/benchmarking/workloads/manifests/swebench-astropy-7336-template.yaml.tmpl b/benchmarking/workloads/manifests/swebench-astropy-7336-template.yaml.tmpl index ab3c93561b..5f8455216e 100644 --- a/benchmarking/workloads/manifests/swebench-astropy-7336-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/swebench-astropy-7336-template.yaml.tmpl @@ -17,7 +17,7 @@ metadata: name: swebench-astropy-7336 workerSelector: matchLabels: - workload: benchmark-ateom + workload: benchmark-worker containers: - name: astropy # The task image is built from the SWE-Perf repo @@ -46,6 +46,14 @@ containers: env: - name: SWEPERF_DISABLE_INTERNAL_SLEEP value: "1" +# Sandbox size: enough for the astropy test runs the trace replays. The cpu +# limit caps the measured execution time, so keep it fixed across runs. +resources: + limits: + - name: cpu + quantity: "2" + - name: memory + quantity: 4Gi sandboxConfig: sandboxClass: ${SANDBOX_CLASS} configName: ${SANDBOX_CONFIG_NAME} diff --git a/benchmarking/workloads/manifests/usermem-template.yaml.tmpl b/benchmarking/workloads/manifests/usermem-template.yaml.tmpl index d8d5ef5403..75553f0120 100644 --- a/benchmarking/workloads/manifests/usermem-template.yaml.tmpl +++ b/benchmarking/workloads/manifests/usermem-template.yaml.tmpl @@ -15,20 +15,27 @@ # ActorTemplate (protojson-shaped ateapipb.ActorTemplate) for the usermem # benchmark workload (locust/tests/usermem.py): stress holds 1 GiB of user # memory to size memory snapshots. Not part of deploy.sh's default template -# set; deploy with WORKLOAD_TEMPLATES="usermem ...". No memory limit: the -# workload allocates 1 GiB, well above the 256Mi default --actor-memory the -# other templates take. The sandbox class must match the WorkerPool's so +# set; deploy with WORKLOAD_TEMPLATES="usermem ...". Its memory limit is +# fixed rather than --actor-memory: the workload allocates 1 GiB, well above +# the 256Mi default. The sandbox class must match the WorkerPool's so # snapshots stay within a class. metadata: atespace: benchmark-workloads name: usermem workerSelector: matchLabels: - workload: benchmark-ateom + workload: benchmark-worker containers: - name: usermem image: containerstack/alpine-stress:latest command: ["stress", "--vm", "1", "--vm-bytes", "1G", "--vm-keep", "--vm-hang", "0"] +# Sandbox size: the 1 GiB stress allocation plus headroom. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 1536Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/benchmark-workloads/usermem/ diff --git a/benchmarking/workloads/manifests/workloads.yaml.tmpl b/benchmarking/workloads/manifests/workloads.yaml.tmpl index 022014bd10..d8043c4d76 100644 --- a/benchmarking/workloads/manifests/workloads.yaml.tmpl +++ b/benchmarking/workloads/manifests/workloads.yaml.tmpl @@ -27,15 +27,15 @@ metadata: apiVersion: ate.dev/v1alpha1 kind: WorkerPool metadata: - name: benchmark-ateom + name: benchmark-worker namespace: benchmark-workloads labels: - workload: benchmark-ateom + workload: benchmark-worker spec: replicas: ${WORKER_COUNT} sandboxClasses: - name: ${SANDBOX_CLASS} - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-${SANDBOX_CLASS} + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-${SANDBOX_CLASS} template: # Lets gvisor and microvm workers share a node tainted ate.dev/sandboxClass=shared. tolerations: diff --git a/charts/substrate-crds/templates/ate.dev_workerpools.yaml b/charts/substrate-crds/templates/ate.dev_workerpools.yaml index d62d252e25..4c32fcb044 100644 --- a/charts/substrate-crds/templates/ate.dev_workerpools.yaml +++ b/charts/substrate-crds/templates/ate.dev_workerpools.yaml @@ -482,7 +482,7 @@ spec: x-kubernetes-list-type: atomic type: object workerImage: - description: WorkerImage is the ateom container image to deploy as + description: WorkerImage is the worker container image to deploy as workers. minLength: 1 type: string diff --git a/charts/substrate/README.md b/charts/substrate/README.md index d996c673e4..d326d71a65 100644 --- a/charts/substrate/README.md +++ b/charts/substrate/README.md @@ -56,6 +56,7 @@ See `values.yaml` for the full set; the important keys: | `postgres.ownerConnectionStringSecretRef` | `substrate-postgres-owner` | Read the owner connection from a pre-created Secret | | `postgres.readWriteRole` | `substrate_readwrite` | Role assumed by read/write connections | | `postgres.ownerRole` | `substrate_owner` | Role assumed by owner connections | +| `preview` | `[]` | Preview gates for the API server, controller and workers, and atelet; set `[ExternalVolumes]` to enable external volume APIs and mounts | | `postgres.schema` | `substrate` | Store the Substrate tables in this PostgreSQL schema | | `postgres.clientCertificates.enabled` | `false` | Project separate owner and runtime login certificates into the API server | | `.resources` | see `values.yaml` | Requests and limits for each container, e.g. `ateApi.resources` and `atenetRouter.agentgateway.resources` | @@ -75,11 +76,17 @@ See `values.yaml` for the full set; the important keys: | `otel.traces.samplingRatio` | `0.01` | Fraction of parentless requests that start a trace, applied to the Go components and the router | | `otel.metrics.enabled` | `true` | Set to `false` to disable OTLP metrics push; Prometheus scraping remains available | | `otel.metrics.endpoint` | `""` | OTLP endpoint for metrics, overriding `otel.endpoint` | -| `otel.logs.enabled` | `true` | Enable OTLP actor events from ateapi and the ateoms, plus the router access log. Actor events go to stdout when OTLP logs are disabled | +| `otel.logs.enabled` | `true` | Enable OTLP actor events from ateapi and the ateworkers, plus the router access log. Actor events go to stdout when OTLP logs are disabled | | `otel.logs.endpoint` | `""` | OTLP endpoint for logs, overriding `otel.endpoint` | ## Upgrading sandbox and snapshot configuration +External volumes now require `preview: [ExternalVolumes]` and a client that opts +in to the same gate (for `kubectl ate`, pass `--preview=ExternalVolumes`). +Actor and ActorTemplate container resource limits must include positive CPU and +memory values. Worker images are now named `ateworker-gvisor` and +`ateworker-microvm`; update custom WorkerPools and image mirrors accordingly. + Upgrade `substrate-crds` before the application chart. Custom `SandboxConfig` resources now put assets and `pauseImage` under `spec.versions[]`, with a named `spec.defaultVersion`. WorkerPools use `spec.sandboxClasses[]`, and actor @@ -112,7 +119,7 @@ API server or atelet itself; select it through `atelet.storageBackend`. RustFS is wired automatically. Private S3 asset URLs must use `s3://`. The default gVisor config now uses the 2026-10-10 nightly (`v3`) for split -filesystem checkpoints. Upgrade the sandbox config together with ateom-gvisor; +filesystem checkpoints. Upgrade the sandbox config together with ateworker-gvisor; custom configs must pin a release with the same checkpoint support. The chart now denies ingress to the control-plane namespace by default and diff --git a/charts/substrate/templates/ate-api-server.yaml b/charts/substrate/templates/ate-api-server.yaml index af72b6d50e..d1b6f81595 100644 --- a/charts/substrate/templates/ate-api-server.yaml +++ b/charts/substrate/templates/ate-api-server.yaml @@ -115,6 +115,7 @@ spec: resources: {{ toYaml .Values.ateApi.resources | indent 10 }} args: + - {{ printf "--preview=%s" (join "," .Values.preview) | quote }} - --objectstore-plugin-socket=/run/objectstore-plugin/control.sock - "--grpc-listen-addr=0.0.0.0:443" - "--grpc-server-cred-bundle=/run/servicedns.podcert.ate.dev/credential-bundle.pem" diff --git a/charts/substrate/templates/ate-controller.yaml b/charts/substrate/templates/ate-controller.yaml index 1db603bb1e..69789defd8 100644 --- a/charts/substrate/templates/ate-controller.yaml +++ b/charts/substrate/templates/ate-controller.yaml @@ -74,6 +74,7 @@ spec: resources: {{ toYaml .Values.ateController.resources | indent 10 }} args: + - {{ printf "--preview=%s" (join "," .Values.preview) | quote }} # The atecontroller binary defaults --ateapi-conn-spec to # dns:///api.ate-system.svc:443, which is correct only for the # canonical render (release name "substrate" in namespace diff --git a/charts/substrate/templates/ate-system-networkpolicies.yaml b/charts/substrate/templates/ate-system-networkpolicies.yaml index 197d28f66e..6c8517918f 100644 --- a/charts/substrate/templates/ate-system-networkpolicies.yaml +++ b/charts/substrate/templates/ate-system-networkpolicies.yaml @@ -19,7 +19,7 @@ # so a port is reachable exactly when some policy here (or another one that # selects the pod, such as k8s-credential-provider's) admits it. Ports that no # policy admits -- the router's /statusz port (4040), the bundled postgres from -# anything but ate-api-server, atelet's AteomHerder from anything but +# anything but ate-api-server, atelet's Atelet from anything but # ate-api-server -- are reachable only through `kubectl port-forward`, which # NetworkPolicy does not filter. # diff --git a/charts/substrate/templates/atelet.yaml b/charts/substrate/templates/atelet.yaml index 733ec6533d..66c0f86a01 100644 --- a/charts/substrate/templates/atelet.yaml +++ b/charts/substrate/templates/atelet.yaml @@ -167,7 +167,7 @@ spec: {{- . | trim | nindent 8 }} {{- end }} volumeMounts: - - name: run-ateom + - name: ate-base mountPath: /var/lib/ate - name: objectstore-plugin mountPath: /run/objectstore-plugin @@ -177,6 +177,7 @@ spec: - name: atelet image: {{ include "substrate.componentImage" (list "atelet" .) }} args: + - {{ printf "--preview=%s" (join "," .Values.preview) | quote }} - --objectstore-plugin-socket=/run/objectstore-plugin/node.sock {{- if $credentialProviderConfig }} - --image-credential-provider-config=/run/image-credential-provider/config.yaml @@ -249,7 +250,7 @@ spec: volumeMounts: - name: objectstore-plugin mountPath: /run/objectstore-plugin - - name: run-ateom + - name: ate-base mountPath: /var/lib/ate - name: podidentity mountPath: /run/podidentity.podcert.ate.dev @@ -282,7 +283,7 @@ spec: - name: asset-staging emptyDir: sizeLimit: 9Gi - - name: run-ateom + - name: ate-base hostPath: path: /var/lib/ate type: DirectoryOrCreate diff --git a/charts/substrate/templates/sandboxconfig-gvisor.yaml b/charts/substrate/templates/sandboxconfig-gvisor.yaml index 6962c80635..055016f758 100644 --- a/charts/substrate/templates/sandboxconfig-gvisor.yaml +++ b/charts/substrate/templates/sandboxconfig-gvisor.yaml @@ -33,7 +33,7 @@ spec: versions: # gVisor nightly 2026-10-10, for the split filesystem checkpoint # (runsc fscheckpoint, -fs-checkpoint-paths and -fs-restore-image-path) that - # ateom-gvisor uses to save and restore durable-dir volumes. + # ateworker-gvisor uses to save and restore durable-dir volumes. - name: v3 state: Enabled # The root sandbox container's image. On GCP, prefer the in-project mirror diff --git a/charts/substrate/values.yaml b/charts/substrate/values.yaml index 4efb91aca0..b4ee86f3fe 100644 --- a/charts/substrate/values.yaml +++ b/charts/substrate/values.yaml @@ -23,6 +23,10 @@ # manifests/ate-install/ install path (kubectl apply). createNamespace: false +# Preview gates for the API server, controller (including its workers), and atelet. +# ExternalVolumes is required for external volume APIs and mounts. +preview: [] + postgres: # Project database login certificates only when the connection strings use them. clientCertificates: @@ -168,7 +172,7 @@ ateApiServerEnvVarsConfigMap: ate-api-server-envvars # # metrics.enabled disables OTLP metrics push from the Go components and router. # traces.enabled takes effect on the router; Go components do not read it yet. -# logs.enabled gates OTLP actor events from ateapi and the ateoms, plus the +# logs.enabled gates OTLP actor events from ateapi and the ateworkers, plus the # router's access log. Actor events go to stdout when OTLP logs are disabled # or no endpoint resolves. otel: @@ -216,5 +220,5 @@ image: images: rustfs: rustfs/rustfs:1.0.0-beta.3@sha256:378642b05b7dcb4849fb77ebe6aca4ced1c3f66e7e504247df95a5c9018d3358 awsCli: amazon/aws-cli:2.17.0@sha256:643507c10ada7964ca6157b3d799f030b90577643da9955d319a77399ed80d73 - agentgateway: ghcr.io/kagent-dev/substrate/agentgateway:9557280b@sha256:59602f697f2ab16346c18e0a5a57044e5ff37e56bbed1b7f9de72445e29aee0c + agentgateway: ghcr.io/kagent-dev/substrate/agentgateway:3554db38a0b2@sha256:309f6811560635471958db67b0f68dd083d84048dffc681967599d159ebfc83d busybox: busybox:1.36 diff --git a/cmd/ate-setup/commands.md b/cmd/ate-setup/commands.md index 1c88b0832b..b7fde07919 100644 --- a/cmd/ate-setup/commands.md +++ b/cmd/ate-setup/commands.md @@ -34,6 +34,7 @@ a pre-scan pass, so they may appear anywhere on its command line. | `--experimental-additional-egress-extproc-service NS/SVC:PORT` | `--experimental-additional-egress-extproc-service NS/SVC:PORT` | External processor authorization filter | | `--credential-provider JSON` | `--credential-provider JSON`, or `ATE_CREDENTIAL_PROVIDER` | Required by `deploy ate-system` and `deploy atenet`. A JSON object: `{"name":"k8s.io"}` deploys and uses the bundled Kubernetes Secrets provider, with a NetworkPolicy that admits only the egress gateway; `{"enabled":false}` turns egress credential injection off; `{"name":"","address":":"}` uses a provider you deploy yourself. See [`docs/egress-credential-injection.md`](../../docs/egress-credential-injection.md) | | `--otlp-endpoint URL` | `--otlp-endpoint URL`, or `ATE_OTLP_ENDPOINT=URL` | Send control plane telemetry to `URL` instead of the cluster default (see [`benchmarking/telemetry/README.md`](../../benchmarking/telemetry/README.md)) | +| `--preview GATES` | `--preview GATES`, or `ATE_PREVIEW=GATES` | Preview gates passed as `--preview` to every substrate component: comma-separated gate names, or `ALL` for all. `ATE_PREVIEW` or `config.preview` when the flag is absent. An explicit `--preview=''` turns them all off, even when `ATE_PREVIEW` is set | | `--context NAME` | `KUBECTL_CONTEXT=NAME` | Kubeconfig context; still defaults to `KUBECTL_CONTEXT` | | `--kubeconfig PATH` | `KUBECONFIG=PATH` | Explicit kubeconfig path | | `--no-dev-env` | `NO_DEV_ENV=1` | Skip `.ate-dev-env.sh` at the repository root | @@ -121,9 +122,9 @@ a running cluster. | `publish worker-images` | (no shell equivalent) | | `publish release-images` | (no shell equivalent) | -`publish worker-images` builds and pushes the ateom worker images for the +`publish worker-images` builds and pushes the worker images for the checked-out build and prints their refs; a WorkerPool points `spec.workerImage` -to a build to use the ateom. +to a build to use the ateworker. `publish release-images` builds and pushes every image a pre-built install needs, all tagged with the build version; see [Installing a @@ -193,7 +194,7 @@ See | `ate-setup` | `hack/install-ate.sh` | Description | |---|---|---| | `deploy demo counter` | `--deploy-demo-counter` | A counter actor exercising snapshot, resume, and atenet ingress | -| `deploy demo counter --with-external-volume [--storage-class NAME]` | `--deploy-demo-counter-with-external-volume` (`STORAGE_CLASS=NAME`) | The same, plus an external volume and a pre-seeded file to validate. Run `setup csi` first and name the class it created, e.g. `csi-nfs-sc`; defaults to `standard` | +| `deploy demo counter --with-external-volume [--storage-class NAME]` | `--deploy-demo-counter-with-external-volume` (`STORAGE_CLASS=NAME`) | The same, plus an external volume and a pre-seeded file to validate (requires `--preview=ExternalVolumes`). Run `setup csi` first and name the class it created, e.g. `csi-nfs-sc`; defaults to `standard` | | `deploy demo counter-microvm` | `--deploy-demo-counter-microvm` | The counter demo on micro-VM workers. Run `hack/install-microvm-deps.sh --install` first | | `deploy demo egress` | `--deploy-demo-egress` | Egress policy enforcement through atenet; the actors trust the egress gateway's CA | | `deploy demo egress-microvm` | `--deploy-demo-egress-microvm` | The same on micro-VM workers. Run `hack/install-microvm-deps.sh --install` first | diff --git a/cmd/ate-setup/differences.md b/cmd/ate-setup/differences.md index 832ca52fea..20f60de36c 100644 --- a/cmd/ate-setup/differences.md +++ b/cmd/ate-setup/differences.md @@ -111,7 +111,7 @@ Manifests carry `ko://` references either way. With `--image-repo` each is looked up in `images.Components` and rewritten to `/:`, which is ko's own `--base-import-paths` naming and so is how the release images are already -published — `ateapi`, `atelet`, `atenet`, `ateom-gvisor`, and the rest. +published — `ateapi`, `atelet`, `atenet`, `ateworker-gvisor`, and the rest. `--image-tag` supplies the tag. Each of the two requires the other: a tag with no repository to pull from would be dropped, leaving a build from source that looks like the release the tag names. diff --git a/cmd/ate-setup/internal/cmd/deploy.go b/cmd/ate-setup/internal/cmd/deploy.go index 8ca21b06d3..08ab2d7bb6 100644 --- a/cmd/ate-setup/internal/cmd/deploy.go +++ b/cmd/ate-setup/internal/cmd/deploy.go @@ -60,7 +60,7 @@ ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING or the ATE_API_POSTGRES_CLOUDSQL_* select an external database. Shape the install with the global --atenet-dataplane, --cluster-size, ---cordon-control-plane, and --fake-data-plane flags.`, +--cordon-control-plane, --fake-data-plane, and --ingress-auth-mode flags.`, // Args runs before the root command loads the configuration, so only the // flag is readable here. Checking it keeps an unusable value from costing // a credential fetch, which is the common case; a value arriving from the diff --git a/cmd/ate-setup/internal/cmd/publish.go b/cmd/ate-setup/internal/cmd/publish.go index 48d7718440..8f9ce0bef8 100644 --- a/cmd/ate-setup/internal/cmd/publish.go +++ b/cmd/ate-setup/internal/cmd/publish.go @@ -25,13 +25,13 @@ var publishCmd = &cobra.Command{ var publishWorkerImagesCmd = &cobra.Command{ Use: "worker-images", - Short: "Build and push the ateom worker images for this build and print their refs", - Long: `Build and push the ateom worker images (one per sandbox class) for the + Short: "Build and push the worker images for this build and print their refs", + Long: `Build and push the worker images (one per sandbox class) for the checked-out build and print their pushed references, one ": " line per image. A WorkerPool moves to a build by pointing spec.workerImage at that build's -ateom ref.`, +worker ref.`, Args: cobra.NoArgs, RunE: func(cmd *cobra.Command, _ []string) error { return env.PublishWorkerImages(cmd.Context(), cmd.OutOrStdout()) diff --git a/cmd/ate-setup/internal/config/config.go b/cmd/ate-setup/internal/config/config.go index 249f61bea1..6e43606974 100644 --- a/cmd/ate-setup/internal/config/config.go +++ b/cmd/ate-setup/internal/config/config.go @@ -50,6 +50,11 @@ const ( // the podcertificate controller's API rate limits to match. ClusterSizeSize0 = "size0" ClusterSizeSize10 = "size10" + + // Ingress router authentication modes; see atenet router's + // --ingress-auth-mode. + IngressAuthDeprecatedInsecure = "deprecated-insecure" + IngressAuthStaticMTLS = "static-mtls" ) // DefaultRolloutTimeout is the default wait timeout for workload rollouts. @@ -199,6 +204,11 @@ type Config struct { // (ATE_INSTALL_FAKE_DATA_PLANE), for control-plane benchmarks. FakeDataPlane bool + // IngressAuthMode is how the ingress router authenticates its clients + // (ATE_INGRESS_AUTH_MODE): IngressAuthDeprecatedInsecure or + // IngressAuthStaticMTLS. + IngressAuthMode string + // AdditionalEgressExtprocService is the optional NS/SVC:PORT external processor filter. AdditionalEgressExtprocService string @@ -212,6 +222,9 @@ type Config struct { // OtlpEndpoint is where the control plane ships telemetry // (ATE_OTLP_ENDPOINT). Benchmark actors are pointed at it too. OtlpEndpoint string + // PreviewGates is the --preview value (ATE_PREVIEW) every substrate + // component is deployed with: gate names, or "ALL" for all of them. + PreviewGates []string // BenchmarkActorMemory is the memory limit for benchmark actors // (BENCHMARK_ACTOR_MEMORY). Empty leaves the workload default in place. BenchmarkActorMemory string @@ -433,6 +446,10 @@ func (c *Config) ScriptEnv() []string { if c.ClusterSize != "" && c.ClusterSize != ClusterSizeSize0 { merged["ATE_INSTALL_CLUSTER_SIZE"] = c.ClusterSize } + delete(merged, "ATE_PREVIEW") + if len(c.PreviewGates) > 0 { + merged["ATE_PREVIEW"] = strings.Join(c.PreviewGates, ",") + } delete(merged, "ATE_INSTALL_CORDON_CONTROL_PLANE") if c.CordonControlPlane { merged["ATE_INSTALL_CORDON_CONTROL_PLANE"] = "true" @@ -441,6 +458,10 @@ func (c *Config) ScriptEnv() []string { if c.FakeDataPlane { merged["ATE_INSTALL_FAKE_DATA_PLANE"] = "true" } + delete(merged, "ATE_INGRESS_AUTH_MODE") + if c.IngressAuthMode != "" && c.IngressAuthMode != IngressAuthDeprecatedInsecure { + merged["ATE_INGRESS_AUTH_MODE"] = c.IngressAuthMode + } if c.AdditionalEgressExtprocService != "" { merged["ATE_ADDITIONAL_EGRESS_EXTPROC_SERVICE"] = c.AdditionalEgressExtprocService } diff --git a/cmd/ate-setup/internal/config/config_test.go b/cmd/ate-setup/internal/config/config_test.go index b9b786299c..20a2abd650 100644 --- a/cmd/ate-setup/internal/config/config_test.go +++ b/cmd/ate-setup/internal/config/config_test.go @@ -19,11 +19,15 @@ import ( "path/filepath" "runtime" "slices" + "strconv" "strings" "testing" "time" + "github.com/spf13/pflag" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" + "github.com/agent-substrate/substrate/internal/preview" ) // loadEnv isolates Load from the ambient environment. @@ -56,6 +60,7 @@ func loadEnv(t *testing.T) { "ATE_CREDENTIAL_PROVIDER", "ATE_IMAGE_REPO", "ATE_IMAGE_TAG", + "ATE_INGRESS_AUTH_MODE", "ATE_INSTALL_CLUSTER_SIZE", "ATE_INSTALL_CORDON_CONTROL_PLANE", "ATE_INSTALL_FAKE_DATA_PLANE", @@ -63,6 +68,7 @@ func loadEnv(t *testing.T) { "ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER", "ATE_INSTALL_ROLLOUT_TIMEOUT", "ATE_OTLP_ENDPOINT", + "ATE_PREVIEW", "BENCHMARK_ACTOR_MEMORY", "BUCKET_NAME", "CLUSTER_LOCATION", @@ -121,6 +127,9 @@ func TestLoadDefaults(t *testing.T) { if cfg.CordonControlPlane { t.Error("CordonControlPlane = true, want false") } + if cfg.IngressAuthMode != IngressAuthDeprecatedInsecure { + t.Errorf("IngressAuthMode = %q, want %q", cfg.IngressAuthMode, IngressAuthDeprecatedInsecure) + } } func TestLoadClusterSize(t *testing.T) { @@ -236,6 +245,39 @@ func TestLoadFakeDataPlane(t *testing.T) { }) } } +func TestLoadIngressAuthMode(t *testing.T) { + for _, tc := range []struct { + name string + opts Options + env string + want string + wantExport bool + }{ + {name: "flag static-mtls", opts: Options{IngressAuthMode: IngressAuthStaticMTLS}, want: IngressAuthStaticMTLS, wantExport: true}, + {name: "environment static-mtls", env: IngressAuthStaticMTLS, want: IngressAuthStaticMTLS, wantExport: true}, + {name: "environment deprecated-insecure", env: IngressAuthDeprecatedInsecure, want: IngressAuthDeprecatedInsecure}, + {name: "flag outranks environment", opts: Options{IngressAuthMode: IngressAuthDeprecatedInsecure}, env: IngressAuthStaticMTLS, want: IngressAuthDeprecatedInsecure}, + } { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + if tc.env != "" { + t.Setenv("ATE_INGRESS_AUTH_MODE", tc.env) + } + cfg, err := Load(tc.opts) + if err != nil { + t.Fatalf("Load() error = %v", err) + } + if cfg.IngressAuthMode != tc.want { + t.Errorf("IngressAuthMode = %q, want %q", cfg.IngressAuthMode, tc.want) + } + got, exported := scriptEnvMap(t, cfg)["ATE_INGRESS_AUTH_MODE"] + if exported != tc.wantExport || (exported && got != tc.want) { + t.Errorf("ScriptEnv() ATE_INGRESS_AUTH_MODE = %q (exported %v), want %q (exported %v)", got, exported, tc.want, tc.wantExport) + + } + }) + } +} func TestLoadDockerBuildFlags(t *testing.T) { loadEnv(t) @@ -250,6 +292,94 @@ func TestLoadDockerBuildFlags(t *testing.T) { } } +// The preview gates arrive from any channel as one comma-separated value. An +// explicitly empty one turns every gate off, even over a lower channel's. +func TestLoadPreviewGates(t *testing.T) { + preview.InitForTestFake(t, []preview.Gate{"A", "B"}) + + // nil leaves the channel out; a pointer to "" supplies it, empty. + val := func(s string) *string { return &s } + for _, tc := range []struct { + name string + flag, file, env *string + want []string + }{ + {name: "default"}, + {name: "flag all", flag: val("ALL"), want: []string{"ALL"}}, + {name: "flag list", flag: val("A,B"), want: []string{"A", "B"}}, + {name: "file", file: val("A, B"), want: []string{"A", "B"}}, + {name: "environment", env: val("ALL"), want: []string{"ALL"}}, + {name: "empty items dropped", env: val(",A,,"), want: []string{"A"}}, + {name: "file beats environment", file: val("A"), env: val("ALL"), want: []string{"A"}}, + {name: "flag beats file", flag: val("B"), file: val("A"), want: []string{"B"}}, + {name: "empty flag beats environment", flag: val(""), env: val("ALL")}, + {name: "empty file beats environment", file: val(""), env: val("ALL")}, + } { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + if tc.env != nil { + t.Setenv("ATE_PREVIEW", *tc.env) + } + fs := pflag.NewFlagSet("test", pflag.ContinueOnError) + BindFlags(fs) + if tc.flag != nil { + if err := fs.Set("preview", *tc.flag); err != nil { + t.Fatal(err) + } + } + var lo LoadOptions + if tc.file != nil { + lo.ConfigPath = filepath.Join(t.TempDir(), "install.yaml") + body := "apiVersion: " + FileAPIVersion + "\nkind: " + FileKind + "\n" + + "preview: " + strconv.Quote(*tc.file) + "\n" + if err := os.WriteFile(lo.ConfigPath, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + } + cfg, err := LoadFlags(fs, lo) + if err != nil { + t.Fatalf("LoadFlags() error = %v", err) + } + if !slices.Equal(cfg.PreviewGates, tc.want) { + t.Errorf("PreviewGates = %q, want %q", cfg.PreviewGates, tc.want) + } + got, exported := scriptEnvMap(t, cfg)["ATE_PREVIEW"] + if want := strings.Join(tc.want, ","); got != want || exported != (want != "") { + t.Errorf("ScriptEnv() ATE_PREVIEW = %q (exported %v), want %q", got, exported, want) + } + }) + } +} + +func TestLoadRejectsUnknownPreviewGates(t *testing.T) { + preview.InitForTestFake(t, []preview.Gate{"Fake"}) + for _, tc := range []struct { + name string + env string + wantErr string + }{ + {name: "all", env: "ALL"}, + {name: "known gate", env: "Fake"}, + {name: "unknown gate", env: "NoSuchGate", wantErr: "ATE_PREVIEW"}, + {name: "unknown among known", env: "ALL,NoSuchGate", wantErr: "ATE_PREVIEW"}, + } { + t.Run(tc.name, func(t *testing.T) { + loadEnv(t) + t.Setenv("ATE_PREVIEW", tc.env) + _, err := Load(Options{}) + if tc.wantErr == "" { + if err != nil { + t.Fatalf("Load() error = %v", err) + } + return + } + if err == nil || !strings.Contains(err.Error(), tc.wantErr) || !strings.Contains(err.Error(), "NoSuchGate") { + t.Fatalf("Load() error = %v, want one naming NoSuchGate and %s", err, tc.wantErr) + } + }) + } +} + func TestLoadFlagsBeatEnvironment(t *testing.T) { loadEnv(t) t.Setenv("ATE_ATENET_DATAPLANE", RouterEnvoy) @@ -696,6 +826,9 @@ func TestLoadRejectsInvalidValues(t *testing.T) { {"kubernetes address without a port", Options{CredentialProvider: `{"name":"k8s.io","address":"secrets.ate-system.svc"}`}}, {"other provider without an address", Options{CredentialProvider: `{"name":"vault.example.com"}`}}, {"other provider address without a port", Options{CredentialProvider: `{"name":"vault.example.com","address":"vault.ate-system.svc"}`}}, + {"ingress auth mode", Options{IngressAuthMode: "mtls"}}, + {"old insecure ingress auth mode", Options{IngressAuthMode: "insecure"}}, + {"static-mtls agentgateway", Options{Router: RouterAgentgateway, IngressAuthMode: IngressAuthStaticMTLS}}, } { t.Run(tc.name, func(t *testing.T) { if _, err := Load(tc.opts); err == nil { diff --git a/cmd/ate-setup/internal/config/coverage_test.go b/cmd/ate-setup/internal/config/coverage_test.go index 287c192027..f8a45bba42 100644 --- a/cmd/ate-setup/internal/config/coverage_test.go +++ b/cmd/ate-setup/internal/config/coverage_test.go @@ -90,6 +90,8 @@ func valueFor(key string) string { return RouterEnvoy case "clusterSize": return ClusterSizeSize0 + case "ingressAuthMode": + return IngressAuthStaticMTLS case "ateapi.postgres.cloudsql.ipType": return CloudSQLIPTypePSC case "atenet.egress.additionalExtprocService": @@ -102,6 +104,8 @@ func valueFor(key string) string { return "https://idp.example.com" case "csi.setup": return "none" + case "preview": + return "ALL" } return "set" } diff --git a/cmd/ate-setup/internal/config/load.go b/cmd/ate-setup/internal/config/load.go index 47a945be29..9ce6e53b47 100644 --- a/cmd/ate-setup/internal/config/load.go +++ b/cmd/ate-setup/internal/config/load.go @@ -26,6 +26,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ate-setup/internal/images" "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/oidcdiscovery" + "github.com/agent-substrate/substrate/internal/preview" ) // ConfigPathEnv names the configuration document when --config is not given. @@ -260,11 +261,13 @@ func buildConfig(root string, env map[string]string, r *Resolved) (*Config, erro ClusterSize: r.String("clusterSize"), CordonControlPlane: r.Bool("cordonControlPlane"), FakeDataPlane: r.Bool("fakeDataPlane.enabled"), + IngressAuthMode: r.String("ingressAuthMode"), AdditionalEgressExtprocService: r.String("atenet.egress.additionalExtprocService"), CredentialProviderJSON: r.String("atenet.egress.credentialProvider"), AnthropicAPIKey: r.String("demo.anthropicAPIKey"), OtlpEndpoint: r.String("otlpEndpoint"), BenchmarkActorMemory: r.String("benchmark.actorMemory"), + PreviewGates: splitList(r.String("preview")), kubeconfigEnv: kubeconfigEnv, shellEnv: env, resolved: r, @@ -276,6 +279,18 @@ func buildConfig(root string, env map[string]string, r *Resolved) (*Config, erro return cfg, nil } +// splitList splits a comma-separated value, dropping empty items, so an empty +// value is no items rather than one empty one. +func splitList(raw string) []string { + var out []string + for _, item := range strings.Split(raw, ",") { + if item = strings.TrimSpace(item); item != "" { + out = append(out, item) + } + } + return out +} + // splitKubeconfig divides the setting into the path handed to client-go and the // value exported to the shell scripts. // @@ -376,6 +391,18 @@ func validateResolved(cfg *Config, r *Resolved) error { return &InvalidError{Value: size, Want: ClusterSizeSize0 + " or " + ClusterSizeSize10} } + authMode, _ := r.Value("ingressAuthMode") + switch cfg.IngressAuthMode { + case IngressAuthDeprecatedInsecure: + case IngressAuthStaticMTLS: + if cfg.Router != RouterEnvoy { + return &ConflictError{A: authMode, B: dataplane, + Why: IngressAuthStaticMTLS + " requires dataplane " + RouterEnvoy} + } + default: + return &InvalidError{Value: authMode, Want: IngressAuthDeprecatedInsecure + " or " + IngressAuthStaticMTLS} + } + switch cfg.ActorJWTAlgorithm { case "ES256", "RS256": default: @@ -396,6 +423,12 @@ func validateResolved(cfg *Config, r *Resolved) error { Why: "an additional ext_proc filter requires dataplane " + RouterEnvoy} } } + // A gate the components do not know makes each of them refuse to start, so + // catch it here rather than as a failed rollout. + if err := preview.Verify(cfg.PreviewGates...); err != nil { + gates, _ := r.Value("preview") + return fmt.Errorf("%w (from %s)", err, gates.From.Describe(gates.Setting)) + } // Parsed here so a malformed provider is rejected with the rest of the // configuration rather than part-way through a deploy. Whether one is // required at all is the deploy's question, not this one's. diff --git a/cmd/ate-setup/internal/config/options_test.go b/cmd/ate-setup/internal/config/options_test.go index 3e8fba4ffc..4aac4df5ea 100644 --- a/cmd/ate-setup/internal/config/options_test.go +++ b/cmd/ate-setup/internal/config/options_test.go @@ -40,6 +40,7 @@ type Options struct { ClusterSize string CordonControlPlane bool FakeDataPlane bool + IngressAuthMode string AdditionalEgressExtprocService string CredentialProvider string OtlpEndpoint string @@ -88,6 +89,7 @@ func optionsFlagSet(opts Options) (*pflag.FlagSet, error) { {"atenet-dataplane", opts.Router}, {"rollout-timeout", opts.RolloutTimeout}, {"cluster-size", opts.ClusterSize}, + {"ingress-auth-mode", opts.IngressAuthMode}, {"experimental-additional-egress-extproc-service", opts.AdditionalEgressExtprocService}, {"credential-provider", opts.CredentialProvider}, {"otlp-endpoint", opts.OtlpEndpoint}, diff --git a/cmd/ate-setup/internal/config/setting.go b/cmd/ate-setup/internal/config/setting.go index 2dcfc6c894..1de019c875 100644 --- a/cmd/ate-setup/internal/config/setting.go +++ b/cmd/ate-setup/internal/config/setting.go @@ -279,11 +279,27 @@ var Registry = []Setting{ "atelet, and fake-workersync in place of ate-controller. Actors do not exist; never " + "use it on a cluster that serves real actors", }, + { + Key: "ingressAuthMode", Env: "ATE_INGRESS_AUTH_MODE", Flag: "ingress-auth-mode", Kind: KindString, + Default: IngressAuthDeprecatedInsecure, + Usage: "How the ingress router authenticates clients: deprecated-insecure or static-mtls. Both check a " + + "client certificate on the router's TLS ports against the podidentity CA and the SPIFFE IDs listed in " + + "manifests/ate-install/atenet-router.yaml; deprecated-insecure also lets through TLS clients without one " + + "and serves plaintext ports, while static-mtls requires the certificate and --atenet-dataplane=envoy", + }, { Key: "otlpEndpoint", Env: "ATE_OTLP_ENDPOINT", Flag: "otlp-endpoint", Kind: KindString, Usage: "Send control plane telemetry to this OTLP collector instead of the cluster default", }, + { + // Comma-separated, as the components' own --preview flag is. The + // registry has no list kind, so the value is carried as written and + // split where it is used. + Key: "preview", Env: "ATE_PREVIEW", Flag: "preview", Kind: KindString, + Usage: "Preview gates every substrate component runs with, comma-separated; ALL enables all", + }, + { Key: "benchmark.actorMemory", Env: "BENCHMARK_ACTOR_MEMORY", Flag: "benchmark-actor-memory", Kind: KindString, diff --git a/cmd/ate-setup/internal/images/images.go b/cmd/ate-setup/internal/images/images.go index fb5b392620..0cfa2e4354 100644 --- a/cmd/ate-setup/internal/images/images.go +++ b/cmd/ate-setup/internal/images/images.go @@ -44,8 +44,8 @@ var Components = []string{ "cmd/atecontroller", "cmd/atelet", "cmd/atenet", - "cmd/ateom-gvisor", - "cmd/ateom-microvm", + "cmd/ateworker-gvisor", + "cmd/ateworker-microvm", "cmd/benchmarking/isolate/fake-atelet", "cmd/benchmarking/isolate/fake-workersync", "cmd/credential-provider/kubernetes-secrets", diff --git a/cmd/ate-setup/internal/images/prebuilt.go b/cmd/ate-setup/internal/images/prebuilt.go index 9f3a5403bb..63656b29ad 100644 --- a/cmd/ate-setup/internal/images/prebuilt.go +++ b/cmd/ate-setup/internal/images/prebuilt.go @@ -22,8 +22,6 @@ import ( "regexp" "slices" "strings" - - "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" ) // koRef matches a whole ko:// reference: the prefix plus everything up to the @@ -68,19 +66,6 @@ func NewPrebuilt(src Source, digest Digester) *Prebuilt { return &Prebuilt{src: src, digest: digest, pinned: make(map[string]string)} } -// ResolvePath rewrites the manifests a path covers. -func (p *Prebuilt) ResolvePath(ctx context.Context, path string) ([]byte, error) { - manifest, err := kube.ReadPath(path) - if err != nil { - return nil, err - } - out, err := p.rewrite(ctx, manifest) - if err != nil { - return nil, fmt.Errorf("in %s: %w", path, err) - } - return out, nil -} - // ResolveBytes rewrites an in-memory manifest, such as kustomize output. func (p *Prebuilt) ResolveBytes(ctx context.Context, manifest []byte) ([]byte, error) { return p.rewrite(ctx, manifest) diff --git a/cmd/ate-setup/internal/images/prebuilt_test.go b/cmd/ate-setup/internal/images/prebuilt_test.go index 5811fbfd85..cbf6acc74c 100644 --- a/cmd/ate-setup/internal/images/prebuilt_test.go +++ b/cmd/ate-setup/internal/images/prebuilt_test.go @@ -90,8 +90,8 @@ func TestPrebuiltResolveBytes(t *testing.T) { // The reference is a plain CRD field here, not a pod spec, which is // why the rewrite cannot be driven off a Kubernetes schema. name: "CRD field", - in: "spec:\n workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor\n", - want: "spec:\n workerImage: example.com/substrate/ateom-gvisor:v1.2.3" + digestSuffix + "\n", + in: "spec:\n workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor\n", + want: "spec:\n workerImage: example.com/substrate/ateworker-gvisor:v1.2.3" + digestSuffix + "\n", }, { // A tag can carry its own digest, and then there is nothing to look @@ -133,8 +133,8 @@ func TestPrebuiltResolveBytes(t *testing.T) { // message and nothing else: the truncation cannot collide with a // listed package, because every one of those ends before the "$". name: "unexpanded placeholder", - in: "image: ko://github.com/agent-substrate/substrate/cmd/ateom-${SANDBOX_CLASS}\n", - error: "cmd/ateom-${SANDBOX_CLASS has no published image", + in: "image: ko://github.com/agent-substrate/substrate/cmd/ateworker-${SANDBOX_CLASS}\n", + error: "cmd/ateworker-${SANDBOX_CLASS has no published image", }, { // A reference extending a known one must not inherit its image. A @@ -217,7 +217,7 @@ func TestPrebuiltResolveBytes(t *testing.T) { // Every unmappable reference is reported at once, so a manifest with several // problems takes one install attempt to diagnose rather than several. func TestPrebuiltReportsEveryFailure(t *testing.T) { - in := "a: ko://github.com/agent-substrate/substrate/cmd/ateom-${SANDBOX_CLASS}\n" + + in := "a: ko://github.com/agent-substrate/substrate/cmd/ateworker-${SANDBOX_CLASS}\n" + "b: ko://github.com/example/other/cmd/thing\n" + "c: ko://github.com/agent-substrate/substrate/cmd/nope\n" @@ -359,9 +359,13 @@ func TestResolveEveryInstallManifest(t *testing.T) { for _, p := range paths { t.Run(filepath.Base(p), func(t *testing.T) { - out, err := resolver.ResolvePath(context.Background(), p) + manifest, err := kube.ReadPath(p) if err != nil { - t.Fatalf("ResolvePath() = %v", err) + t.Fatalf("ReadPath() = %v", err) + } + out, err := resolver.ResolveBytes(context.Background(), manifest) + if err != nil { + t.Fatalf("ResolveBytes() = %v", err) } if refs := koRefPattern.FindAllString(string(out), -1); len(refs) > 0 { t.Errorf("unresolved references survived: %v", refs) diff --git a/cmd/ate-setup/internal/ko/ko.go b/cmd/ate-setup/internal/ko/ko.go index 44d8d82852..9c64428929 100644 --- a/cmd/ate-setup/internal/ko/ko.go +++ b/cmd/ate-setup/internal/ko/ko.go @@ -117,7 +117,7 @@ func (r *Runner) Resolve(ctx context.Context, path string, stdinManifest []byte) // Build builds and publishes one Go binary's image and returns its pushed // reference (the last line ko prints). Resolve covers everything a manifest -// references; Build is for the images with no manifest names, such as the ateom +// references; Build is for the images with no manifest names, such as the worker // worker images a WorkerPool points at through workerImage. func (r *Runner) Build(ctx context.Context, importPath string) (string, error) { defer log.Elapsed(time.Now(), "ko build "+importPath) @@ -157,11 +157,6 @@ func (r *Runner) BuildTagged(ctx context.Context, tag string, importPaths ...str return refs, nil } -// ResolvePath resolves a manifest file or directory. -func (r *Runner) ResolvePath(ctx context.Context, path string) ([]byte, error) { - return r.Resolve(ctx, path, nil) -} - // ResolveBytes resolves an in-memory manifest, such as kustomize output. func (r *Runner) ResolveBytes(ctx context.Context, manifest []byte) ([]byte, error) { return r.Resolve(ctx, "-", manifest) diff --git a/cmd/ate-setup/internal/ko/ko_test.go b/cmd/ate-setup/internal/ko/ko_test.go index a1bbfa9e6a..911eed5bb3 100644 --- a/cmd/ate-setup/internal/ko/ko_test.go +++ b/cmd/ate-setup/internal/ko/ko_test.go @@ -38,7 +38,7 @@ func TestArgsAlwaysRequestBaseImportPaths(t *testing.T) { }{ {"resolve", r.args("resolve", "-f", "manifests/ate-install")}, {"resolve from stdin", r.args("resolve", "-f", "-")}, - {"build", r.args("build", "github.com/agent-substrate/substrate/cmd/ateom-gvisor")}, + {"build", r.args("build", "github.com/agent-substrate/substrate/cmd/ateworker-gvisor")}, } { t.Run(tc.name, func(t *testing.T) { if !slices.Contains(tc.args, "--base-import-paths") { diff --git a/cmd/ate-setup/internal/steps/csi.go b/cmd/ate-setup/internal/steps/csi.go index 129bdd4b5b..7455a49e12 100644 --- a/cmd/ate-setup/internal/steps/csi.go +++ b/cmd/ate-setup/internal/steps/csi.go @@ -35,10 +35,10 @@ import ( "github.com/agent-substrate/substrate/internal/nodepath" ) -// ateomHostDir is the host directory atelet mounts sandbox images under. Both +// workerHostDir is the host directory atelet mounts sandbox images under. Both // CSI drivers need it bind-mounted with bidirectional propagation so volumes // they mount inside it are visible to atelet. -const ateomHostDir = nodepath.BasePath +const workerHostDir = nodepath.BasePath // csiHostpathControllerService fronts the hostpath driver's socat sidecar, // which bridges the TCP endpoint atelet dials to the driver's unix socket. The @@ -161,9 +161,6 @@ func (e *Env) ensureCSIPrerequisites(ctx context.Context) error { if err := e.EnsureAteSystemNamespace(ctx); err != nil { return err } - if err := e.EnsurePodCertificateCAs(ctx); err != nil { - return err - } return e.DeployPodCertificateController(ctx) } @@ -183,7 +180,7 @@ func (e *Env) setupCSIHostpath(ctx context.Context) error { } // Remove any previous deployment first: the plugin StatefulSet holds - // mounts under ateomHostDir, and re-applying over a running one leaves + // mounts under workerHostDir, and re-applying over a running one leaves // stale mounts behind. log.Infof("Cleaning up existing CSI hostpath resources...") if err := e.Kube.Delete(ctx, objs); err != nil { @@ -204,7 +201,7 @@ func (e *Env) setupCSIHostpath(ctx context.Context) error { return err } log.Infof("Using Kind node: %s", node) - cleanupKindAteomDir(node) + cleanupKindWorkerDir(node) // Before the driver, not after: atelet talks to the controller over TCP, // the socat sidecar bridges that to the driver's unix socket, and @@ -229,7 +226,7 @@ func (e *Env) setupCSIHostpath(ctx context.Context) error { return fmt.Errorf("csi-hostpathplugin StatefulSet was not created: %w", err) } - // Pin the plugin to the node whose ateomHostDir it bind-mounts, and mount + // Pin the plugin to the node whose workerHostDir it bind-mounts, and mount // that directory with bidirectional propagation. log.Infof("Patching the CSI hostpath StatefulSet...") patch := fmt.Sprintf(` @@ -241,15 +238,15 @@ spec: containers: - name: hostpath volumeMounts: - - name: ateom-dir + - name: ate-base mountPath: %s mountPropagation: Bidirectional volumes: - - name: ateom-dir + - name: ate-base hostPath: path: %s type: DirectoryOrCreate -`, node, ateomHostDir, ateomHostDir) +`, node, workerHostDir, workerHostDir) patchJSON, err := yaml.YAMLToJSON([]byte(patch)) if err != nil { return fmt.Errorf("while converting csi-hostpathplugin patch to JSON: %w", err) @@ -289,7 +286,7 @@ spec: return err } - // The image cache directories under ateomHostDir were just wiped, so + // The image cache directories under workerHostDir were just wiped, so // atelet has to recreate them. log.Infof("Restarting the atelet DaemonSet (if present)...") return e.Kube.RolloutRestart(ctx, e.Namespace(), "atelet", time.Now()) @@ -329,15 +326,15 @@ spec: containers: - name: nfs volumeMounts: - - name: ateom-dir + - name: ate-base mountPath: %s mountPropagation: Bidirectional volumes: - - name: ateom-dir + - name: ate-base hostPath: path: %s type: DirectoryOrCreate -`, ateomHostDir, ateomHostDir) +`, workerHostDir, workerHostDir) nodePatchJSON, err := yaml.YAMLToJSON([]byte(nodePatch)) if err != nil { return fmt.Errorf("while converting csi-nfs-node patch to JSON: %w", err) @@ -485,7 +482,7 @@ func (e *Env) kindNodeName(ctx context.Context) (string, error) { return nodes.Items[0].Name, nil } -// cleanupKindAteomDir clears the sandbox image directory inside the Kind node +// cleanupKindWorkerDir clears the sandbox image directory inside the Kind node // container. // // Best effort throughout, as in the shell script: a stale mount from a @@ -493,16 +490,16 @@ func (e *Env) kindNodeName(ctx context.Context) (string, error) { // not be running at all when only the CSI drivers are being reinstalled. The // directory itself is kept so that mounts held by running pods such as atelet // survive. -func cleanupKindAteomDir(node string) { +func cleanupKindWorkerDir(node string) { log.Infof("Cleaning up CSI directories on the Kind node...") if !dockerContainerRunning(node) { log.Warnf("Kind node %s is not running. Skipping directory cleanup.", node) return } - unmount := fmt.Sprintf(`for mnt in $(mount | grep %s | awk '{print $3}'); do umount -f "${mnt}" || true; done`, ateomHostDir) + unmount := fmt.Sprintf(`for mnt in $(mount | grep %s | awk '{print $3}'); do umount -f "${mnt}" || true; done`, workerHostDir) runQuiet("docker", "exec", node, "sh", "-c", unmount) - runQuiet("docker", "exec", node, "sh", "-c", "rm -rf "+ateomHostDir+"/*") + runQuiet("docker", "exec", node, "sh", "-c", "rm -rf "+workerHostDir+"/*") } func dockerContainerRunning(name string) bool { diff --git a/cmd/ate-setup/internal/steps/deploy.go b/cmd/ate-setup/internal/steps/deploy.go index 4cf56d1208..dd015eb2bb 100644 --- a/cmd/ate-setup/internal/steps/deploy.go +++ b/cmd/ate-setup/internal/steps/deploy.go @@ -186,8 +186,9 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { return e.applyOtelEndpointOverride(ctx) } -// DeployPodCertificateController applies the podcertificate controller and -// waits until it is rolling and has published its trust bundles. +// DeployPodCertificateController creates any missing signer CA pools, applies +// the podcertificate controller, and waits until it is rolling and has +// published its trust bundles. // // size10 clusters render the podcert-size10 overlay instead of the base file. // It appends --kube-api-qps=100 / --kube-api-burst=200 so the controller keeps @@ -200,6 +201,11 @@ func (e *Env) DeployAteSystem(ctx context.Context, opts DeployOptions) error { // reason: one apply means one rollout, and the rollout wait sees the pod that // will actually serve. func (e *Env) DeployPodCertificateController(ctx context.Context) error { + // The controller mounts every pool, so one that an older release did not + // create keeps the new pod in ContainerCreating. + if err := e.EnsurePodCertificateCAs(ctx); err != nil { + return err + } path := e.Cfg.Manifest("pod-certificate-controller.yaml") if e.Cfg.Size10() { path = e.Cfg.Manifest("podcert-size10") @@ -300,7 +306,7 @@ func (e *Env) DeployAteAPIServer(ctx context.Context) error { return err } } - manifest, err := e.renderResolve(ctx, e.Cfg.Manifest("ate-api-server.yaml")) + manifest, err := e.renderResolve(ctx, e.apiServerManifestPath()) if err != nil { return err } @@ -314,6 +320,14 @@ func (e *Env) DeployAteAPIServer(ctx context.Context) error { return e.Kube.RolloutStatus(ctx, kube.KindDeployment, e.Namespace(), "ate-api-server", e.Cfg.RolloutTimeout) } +// apiServerManifestPath is the ate-api-server manifest for the environment. +func (e *Env) apiServerManifestPath() string { + if e.Cfg.Kind { + return e.Cfg.Manifest("kind", "ate-api-server") + } + return e.Cfg.Manifest("ate-api-server.yaml") +} + // DeployAteController redeploys only ate-controller. func (e *Env) DeployAteController(ctx context.Context) error { log.Step("deploy_ate_controller") @@ -330,7 +344,7 @@ func (e *Env) DeployAteController(ctx context.Context) error { if err := e.applyOtelConfig(ctx); err != nil { return err } - manifest, err := e.renderDataPlane(e.Cfg.Manifest("ate-controller.yaml")) + manifest, err := e.render(e.Cfg.Manifest("ate-controller.yaml")) if err != nil { return err } @@ -459,12 +473,7 @@ func (e *Env) DeployAtelet(ctx context.Context) error { return err } - path := e.Cfg.Manifest("atelet.yaml") - if e.Cfg.Kind { - // The kind overlay patches the DaemonSet for the local node layout. - path = e.Cfg.Path(installDir + "/kind/atelet") - } - manifest, err := e.renderDataPlane(path) + manifest, err := e.render(e.ateletManifestPath()) if err != nil { return err } @@ -481,6 +490,14 @@ func (e *Env) DeployAtelet(ctx context.Context) error { return e.waitFakeWorkersync(ctx) } +// ateletManifestPath is the atelet manifest for the environment. +func (e *Env) ateletManifestPath() string { + if e.Cfg.Kind { + return e.Cfg.Manifest("kind", "atelet") + } + return e.Cfg.Manifest("atelet.yaml") +} + // DeployAtenet redeploys the atenet dataplane: router and egress. func (e *Env) DeployAtenet(ctx context.Context) error { log.Step("deploy_atenet") diff --git a/cmd/ate-setup/internal/steps/deploy_test.go b/cmd/ate-setup/internal/steps/deploy_test.go index 48fb56ab11..4edffc0012 100644 --- a/cmd/ate-setup/internal/steps/deploy_test.go +++ b/cmd/ate-setup/internal/steps/deploy_test.go @@ -15,9 +15,14 @@ package steps import ( + "errors" + "io/fs" "path/filepath" "testing" + "github.com/google/go-cmp/cmp" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" @@ -112,6 +117,28 @@ func TestSetPodcertWorkersPerSignerRejectsMissingDeployment(t *testing.T) { } } +// Upgrading from a release that predates a signer pool: deploying the +// controller alone has to create the missing pool before the new pod needs it. +// The manifest root is empty, so the deploy stops at rendering, after the pools. +func TestDeployPodCertificateControllerCreatesMissingCAs(t *testing.T) { + e := &Env{ + Cfg: &config.Config{Root: t.TempDir()}, + Kube: fakeKube(t, &corev1.Namespace{ObjectMeta: metav1.ObjectMeta{Name: NamespacePodCert}}), + } + if err := e.CreatePodCertificateControllerCAs(t.Context()); err != nil { + t.Fatal(err) + } + if err := e.Kube.Typed.CoreV1().Secrets(NamespacePodCert).Delete(t.Context(), SecretPostgresCA, metav1.DeleteOptions{}); err != nil { + t.Fatal(err) + } + if err := e.DeployPodCertificateController(t.Context()); !errors.Is(err, fs.ErrNotExist) { + t.Fatalf("DeployPodCertificateController = %v, want the render error for the missing manifest", err) + } + if _, err := e.Kube.Typed.CoreV1().Secrets(NamespacePodCert).Get(t.Context(), SecretPostgresCA, metav1.GetOptions{}); err != nil { + t.Errorf("%s/%s after DeployPodCertificateController: %v", NamespacePodCert, SecretPostgresCA, err) + } +} + // Runs over the real generated manifests: EnsureCRDs and DeployCRDs act on what // generatedCRDs returns, so it has to hold every CRD file and no RBAC. func TestGeneratedCRDsLeavesOutRBAC(t *testing.T) { @@ -157,3 +184,47 @@ func TestMissingCRDsLeavesPresentOnesAlone(t *testing.T) { t.Errorf("missingCRDs = %v, want only csidriverconfigs.ate.dev", names) } } + +// A single-component redeploy has to apply what deploy ate-system does. On +// kind the bundle points the object-store plugins at rustfs, and a redeploy +// from a base file would fall back to GCS and never start. +func TestComponentManifestsMatchSystemBundle(t *testing.T) { + for _, tc := range []struct { + router string + kind bool + }{ + {config.RouterEnvoy, false}, + {config.RouterEnvoy, true}, + {config.RouterAgentgateway, false}, + {config.RouterAgentgateway, true}, + } { + e := &Env{Cfg: &config.Config{Root: repoRoot(t), Router: tc.router, Kind: tc.kind}} + render := func(path string) []*unstructured.Unstructured { + manifest, err := e.render(path) + if err != nil { + t.Fatalf("render(%s): %v", path, err) + } + objs, err := kube.DecodeManifestBytes(manifest) + if err != nil { + t.Fatalf("decoding %s: %v", path, err) + } + return objs + } + bundle := render(e.Cfg.Path(SystemOverlay(e.Cfg))) + for _, component := range []struct { + kind, name, path string + }{ + {"Deployment", "ate-api-server", e.apiServerManifestPath()}, + {"DaemonSet", "atelet", e.ateletManifestPath()}, + } { + want := findObject(bundle, component.kind, component.name) + got := findObject(render(component.path), component.kind, component.name) + if want == nil || got == nil { + t.Fatalf("router=%s kind=%v: %s/%s missing from the system bundle or %s", tc.router, tc.kind, component.kind, component.name, component.path) + } + if diff := cmp.Diff(want.Object, got.Object); diff != "" { + t.Errorf("router=%s kind=%v: %s renders a different %s/%s than the system bundle (-bundle +standalone):\n%s", tc.router, tc.kind, component.path, component.kind, component.name, diff) + } + } + } +} diff --git a/cmd/ate-setup/internal/steps/env.go b/cmd/ate-setup/internal/steps/env.go index af7f4a1a40..dc51749a5c 100644 --- a/cmd/ate-setup/internal/steps/env.go +++ b/cmd/ate-setup/internal/steps/env.go @@ -67,7 +67,6 @@ const ( // into one whose references are pullable. *ko.Runner builds and publishes them // from source; *images.Prebuilt maps them onto an already-published release. type imageResolver interface { - ResolvePath(ctx context.Context, path string) ([]byte, error) ResolveBytes(ctx context.Context, manifest []byte) ([]byte, error) } @@ -197,19 +196,29 @@ func (e *Env) ResolveAndApply(ctx context.Context, path string) error { // ResolveManifest turns the ko:// references in a manifest path into pullable // image references. func (e *Env) ResolveManifest(ctx context.Context, path string) ([]byte, error) { - resolver, err := e.imageResolver() + manifest, err := kube.ReadPath(path) if err != nil { return nil, err } - return resolver.ResolvePath(ctx, path) + resolved, err := e.ResolveManifestBytes(ctx, manifest) + if err != nil { + return nil, fmt.Errorf("in %s: %w", path, err) + } + return resolved, nil } // ResolveManifestBytes resolves an in-memory manifest, such as kustomize output. +// Every manifest that deploys a substrate component passes through here, so +// this is also where each component gets its --preview argument. func (e *Env) ResolveManifestBytes(ctx context.Context, manifest []byte) ([]byte, error) { resolver, err := e.imageResolver() if err != nil { return nil, err } + manifest, err = injectPreviewArg(manifest, e.Cfg.PreviewGates) + if err != nil { + return nil, err + } return resolver.ResolveBytes(ctx, manifest) } diff --git a/cmd/ate-setup/internal/steps/overlay.go b/cmd/ate-setup/internal/steps/overlay.go index fdf671e100..4d38363c01 100644 --- a/cmd/ate-setup/internal/steps/overlay.go +++ b/cmd/ate-setup/internal/steps/overlay.go @@ -48,6 +48,29 @@ const cordonControlPlaneComponent = installDir + "/components/cordon-control-pla // over the atelet and ate-controller applies under --fake-data-plane. const fakeDataPlaneComponent = installDir + "/components/fake-data-plane" +// routerStaticMTLSComponent is the kustomize component that switches the +// ingress router to --ingress-auth-mode=static-mtls, layered over every +// control plane apply under --ingress-auth-mode=static-mtls. +const routerStaticMTLSComponent = installDir + "/components/router-static-mtls" + +// components lists the kustomize components every control plane apply is +// composed with. Each patch targets its workloads by name, and kustomize leaves +// a stream alone when nothing in it matches, so composing a manifest that +// carries none of them is harmless. +func (e *Env) components() []string { + var components []string + if e.Cfg.FakeDataPlane { + components = append(components, e.Cfg.Path(fakeDataPlaneComponent)) + } + if e.Cfg.IngressAuthMode == config.IngressAuthStaticMTLS { + components = append(components, e.Cfg.Path(routerStaticMTLSComponent)) + } + if e.Cfg.CordonControlPlane { + components = append(components, e.Cfg.Path(cordonControlPlaneComponent)) + } + return components +} + // SystemOverlay picks the kustomization for a full control plane install. // // The choice is a product of two switches: kind vs GKE, and the atenet router @@ -70,19 +93,11 @@ func SystemOverlay(cfg *config.Config) string { } } -// render emits the manifests at path, an absolute manifest file or -// kustomization directory, before image resolution, composed with the given -// component directories. Under --cordon-control-plane it adds the -// cordon-control-plane component, so the same node pinning reaches every -// control plane workload whichever apply path delivers it. The component's -// patch has a name-regex target and kustomize leaves a stream alone when -// nothing in it matches, so wrapping a manifest that carries none of those -// workloads is harmless. -func (e *Env) render(path string, components ...string) ([]byte, error) { - if e.Cfg.CordonControlPlane { - components = append(components, e.Cfg.Path(cordonControlPlaneComponent)) - } - if len(components) > 0 { +// kustomization directory, before image resolution. It composes path with the +// selected components (see components), so the same patches reach every +// control plane workload whichever apply path delivers it. +func (e *Env) render(path string) ([]byte, error) { + if components := e.components(); len(components) > 0 { return kustomize.Compose(path, components...) } info, err := os.Stat(path) @@ -101,23 +116,12 @@ func (e *Env) render(path string, components ...string) ([]byte, error) { // renderBytes is render for a manifest already held in memory. func (e *Env) renderBytes(manifest []byte) ([]byte, error) { - if e.Cfg.CordonControlPlane { - return kustomize.ComposeBytes(manifest, e.Cfg.Path(cordonControlPlaneComponent)) + if components := e.components(); len(components) > 0 { + return kustomize.ComposeBytes(manifest, components...) } return manifest, nil } -// renderDataPlane is render for the manifests that carry atelet or -// ate-controller. Under --fake-data-plane it composes them with the -// fake-data-plane component, whose patches leave a manifest without their -// target alone. -func (e *Env) renderDataPlane(path string) ([]byte, error) { - if !e.Cfg.FakeDataPlane { - return e.render(path) - } - return e.render(path, e.Cfg.Path(fakeDataPlaneComponent)) -} - // renderResolve renders path and resolves its image references. func (e *Env) renderResolve(ctx context.Context, path string) ([]byte, error) { manifest, err := e.render(path) @@ -139,7 +143,7 @@ func (e *Env) renderResolveApply(ctx context.Context, path string) error { // renderSystemManifests produces the full control plane manifest with all // image references resolved. func (e *Env) renderSystemManifests(ctx context.Context) ([]byte, error) { - manifest, err := e.renderDataPlane(e.Cfg.Path(SystemOverlay(e.Cfg))) + manifest, err := e.render(e.Cfg.Path(SystemOverlay(e.Cfg))) if err != nil { return nil, err } @@ -511,7 +515,7 @@ const otelEndpointKey = "OTEL_EXPORTER_OTLP_ENDPOINT" // otelOverrideDeployments are the control plane Deployments that read // ate-otel-config. ate-controller additionally copies the values onto the -// ateom worker pods it creates, so one patch reaches the whole system. +// worker pods it creates, so one patch reaches the whole system. var otelOverrideDeployments = []string{"ate-api-server", "ate-controller", "atenet-router"} // applyOtelEndpointOverride points all control plane telemetry at a different diff --git a/cmd/ate-setup/internal/steps/overlay_test.go b/cmd/ate-setup/internal/steps/overlay_test.go index b40979a843..d3ec4ad261 100644 --- a/cmd/ate-setup/internal/steps/overlay_test.go +++ b/cmd/ate-setup/internal/steps/overlay_test.go @@ -440,6 +440,12 @@ func TestRenderCordonControlPlane(t *testing.T) { path: func(e *Env) string { return e.Cfg.Manifest("ate-api-server.yaml") }, want: []string{"ate-api-server"}, }, + { + name: "kind api server overlay", + cfg: config.Config{Kind: true}, + path: func(e *Env) string { return e.apiServerManifestPath() }, + want: []string{"ate-api-server"}, + }, { name: "podcert file", path: func(e *Env) string { return e.Cfg.Manifest("pod-certificate-controller.yaml") }, @@ -629,7 +635,7 @@ func TestRenderFakeDataPlane(t *testing.T) { { name: "kind atelet overlay", cfg: config.Config{Kind: true}, - path: func(e *Env) string { return e.Cfg.Path(installDir + "/kind/atelet") }, + path: func(e *Env) string { return e.ateletManifestPath() }, wantAtelet: true, wantStorage: "s3", }, { @@ -650,7 +656,7 @@ func TestRenderFakeDataPlane(t *testing.T) { cfg.FakeDataPlane = true e := &Env{Cfg: &cfg} - rendered, err := e.renderDataPlane(tc.path(e)) + rendered, err := e.render(tc.path(e)) if err != nil { t.Fatalf("renderDataPlane: %v", err) } @@ -755,20 +761,113 @@ func TestRenderFakeDataPlane(t *testing.T) { } } +// Under --ingress-auth-mode=static-mtls both paths that deliver the router +// -- the system bundle and the lone router redeploy -- have to carry the +// static-mtls flags, alone or alongside the cordon component. Whether the +// router accepts those flags is pinned on the router side, by +// TestRouterIngressAuthManifestsAreValid. +func TestRenderRouterStaticMTLS(t *testing.T) { + root := repoRoot(t) + for _, tc := range []struct { + name string + cfg config.Config + path func(e *Env) string + cordon bool + }{ + { + name: "base bundle", + cfg: config.Config{Router: config.RouterEnvoy}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + }, + { + name: "kind bundle", + cfg: config.Config{Router: config.RouterEnvoy, Kind: true}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + }, + { + name: "router file", + cfg: config.Config{Router: config.RouterEnvoy}, + path: func(e *Env) string { return e.Cfg.Manifest("atenet-router.yaml") }, + }, + { + name: "kind bundle with cordon", + cfg: config.Config{Router: config.RouterEnvoy, Kind: true}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + cordon: true, + }, + } { + t.Run(tc.name, func(t *testing.T) { + cfg := tc.cfg + cfg.Root = root + cfg.IngressAuthMode = config.IngressAuthStaticMTLS + cfg.CordonControlPlane = tc.cordon + e := &Env{Cfg: &cfg} + + rendered, err := e.render(tc.path(e)) + if err != nil { + t.Fatalf("render: %v", err) + } + args := routerArgs(t, rendered) + if !slices.Contains(args, "--ingress-auth-mode=static-mtls") { + t.Errorf("atenet-router args %v do not select static-mtls", args) + } + if tc.cordon { + if pinned, _ := pinnedWorkloads(t, rendered); pinned["atenet-router"] != "ate-control-plane" { + t.Errorf("composing both components dropped the cordon pinning (pinned: %v)", pinned) + } + } + t.Run("deprecated-insecure leaves the router alone", func(t *testing.T) { + e := &Env{Cfg: &config.Config{Root: root, Router: config.RouterEnvoy, IngressAuthMode: config.IngressAuthDeprecatedInsecure}} + rendered, err := e.render(e.Cfg.Manifest("atenet-router.yaml")) + if err != nil { + t.Fatalf("render: %v", err) + } + for _, arg := range routerArgs(t, rendered) { + if strings.HasPrefix(arg, "--ingress-auth-mode") { + t.Errorf("deprecated-insecure render sets %s", arg) + } + } + }) + }) + } +} + +// routerArgs returns the atenet-router container's args from a rendered +// manifest. +func routerArgs(t *testing.T, manifest []byte) []string { + t.Helper() + for _, doc := range strings.Split(string(manifest), "\n---\n") { + var deployment appsv1.Deployment + if err := yaml.Unmarshal([]byte(doc), &deployment); err != nil { + t.Fatalf("rendered document is not valid YAML: %v", err) + } + if deployment.Kind != "Deployment" || deployment.Name != "atenet-router" { + continue + } + for _, c := range deployment.Spec.Template.Spec.Containers { + if c.Name == "atenet-router" { + return c.Args + } + } + } + t.Fatal("rendered manifest has no atenet-router container") + return nil +} + // Without --fake-data-plane the data plane renders as it always has. func TestRenderDataPlaneWithoutFakeIsRender(t *testing.T) { e := &Env{Cfg: &config.Config{Root: repoRoot(t), Router: config.RouterEnvoy, Kind: true}} path := e.Cfg.Path(SystemOverlay(e.Cfg)) - got, err := e.renderDataPlane(path) + got, err := e.render(path) if err != nil { - t.Fatalf("renderDataPlane: %v", err) + t.Fatalf("render: %v", err) } want, err := e.render(path) if err != nil { t.Fatalf("render: %v", err) } if string(got) != string(want) { - t.Error("renderDataPlane without --fake-data-plane differs from render") + t.Error("render without --fake-data-plane differs from render") } if _, deployments := dataPlaneWorkloads(t, got); deployments[fakeWorkersyncDeployment].Name != "" { t.Errorf("render without --fake-data-plane has a %s Deployment", fakeWorkersyncDeployment) @@ -1214,7 +1313,7 @@ func TestRequireNoWorkerPods(t *testing.T) { workerPod := func(name string, terminating bool) *corev1.Pod { p := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{ Namespace: "benchmark-workloads", Name: name, - Labels: map[string]string{workerPoolLabel: "benchmark-ateom"}, + Labels: map[string]string{workerPoolLabel: "benchmark-worker"}, }, Status: corev1.PodStatus{Phase: corev1.PodRunning}} if terminating { now := metav1.Now() @@ -1249,7 +1348,7 @@ func TestRequireNoWorkerPods(t *testing.T) { if (err != nil) != tc.wantErr { t.Fatalf("requireNoWorkerPods() = %v, want error %v", err, tc.wantErr) } - if err != nil && strings.Count(err.Error(), "benchmark-workloads/benchmark-ateom") != 1 { + if err != nil && strings.Count(err.Error(), "benchmark-workloads/benchmark-worker") != 1 { t.Errorf("error %q does not name the WorkerPool exactly once", err) } }) diff --git a/cmd/ate-setup/internal/steps/postgres.go b/cmd/ate-setup/internal/steps/postgres.go index 5067d6a4c8..9287a75db2 100644 --- a/cmd/ate-setup/internal/steps/postgres.go +++ b/cmd/ate-setup/internal/steps/postgres.go @@ -344,9 +344,6 @@ func (e *Env) DeployPostgres(ctx context.Context) error { if err := e.EnsureAteSystemNamespace(ctx); err != nil { return err } - if err := e.EnsurePodCertificateCAs(ctx); err != nil { - return err - } // The StatefulSet's projected serving certificate is issued by this // controller. Applying it here makes `deploy postgres` usable on a fresh diff --git a/cmd/ate-setup/internal/steps/prereqs.go b/cmd/ate-setup/internal/steps/prereqs.go index 723ccae834..cf2dd0b4e1 100644 --- a/cmd/ate-setup/internal/steps/prereqs.go +++ b/cmd/ate-setup/internal/steps/prereqs.go @@ -45,9 +45,6 @@ func (e *Env) EnsureAPIServerPrerequisites(ctx context.Context) error { if err := e.ensureSecret(ctx, e.Namespace(), SecretActorIDCACerts, e.CreateActorIDCACertsSecret); err != nil { return err } - if err := e.EnsurePodCertificateCAs(ctx); err != nil { - return err - } // Always reconcile the PostgreSQL connection settings, so that a changed // ATE_API_POSTGRES_READ_WRITE_CONNECTION_STRING reaches an existing install. if err := e.CreateAPIServerEnvVars(ctx); err != nil { diff --git a/cmd/ate-setup/internal/steps/preview.go b/cmd/ate-setup/internal/steps/preview.go new file mode 100644 index 0000000000..0520ec8a5a --- /dev/null +++ b/cmd/ate-setup/internal/steps/preview.go @@ -0,0 +1,110 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package steps + +import ( + "bytes" + "fmt" + "strings" + + "sigs.k8s.io/kustomize/kyaml/kio" + "sigs.k8s.io/kustomize/kyaml/yaml" +) + +// substrateImagePrefix marks a container as running a substrate binary. Every +// such binary accepts --preview. +const substrateImagePrefix = "ko://github.com/agent-substrate/substrate/cmd/" + +const previewFlag = "--preview" + +// containerListPaths are where a workload keeps its containers. +var containerListPaths = [][]string{ + {"spec", "containers"}, + {"spec", "initContainers"}, + {"spec", "template", "spec", "containers"}, + {"spec", "template", "spec", "initContainers"}, + {"spec", "jobTemplate", "spec", "template", "spec", "containers"}, + {"spec", "jobTemplate", "spec", "template", "spec", "initContainers"}, +} + +// previewArg is the --preview argument every substrate component gets. +func previewArg(enabled []string) string { + if len(enabled) > 0 { + return previewFlag + "=" + strings.Join(enabled, ",") + } + return previewFlag + "=" +} + +// injectPreviewArg sets --preview on every substrate container in manifest, +// replacing any --preview it already carries. It must run before image +// resolution, which rewrites the ko:// references it keys on. +func injectPreviewArg(manifest []byte, enabled []string) ([]byte, error) { + if !bytes.Contains(manifest, []byte(substrateImagePrefix)) { + return manifest, nil + } + nodes, err := (&kio.ByteReader{ + Reader: bytes.NewReader(manifest), + OmitReaderAnnotations: true, + }).Read() + if err != nil { + return nil, fmt.Errorf("while parsing manifest: %w", err) + } + arg := previewArg(enabled) + for _, node := range nodes { + for _, path := range containerListPaths { + containers, err := node.Pipe(yaml.Lookup(path...)) + if err != nil { + return nil, err + } + if containers == nil { + continue + } + elements, err := containers.Elements() + if err != nil { + return nil, err + } + for _, container := range elements { + if err := setPreviewArg(container, arg); err != nil { + return nil, err + } + } + } + } + var out bytes.Buffer + if err := (kio.ByteWriter{Writer: &out}).Write(nodes); err != nil { + return nil, fmt.Errorf("while writing manifest: %w", err) + } + return out.Bytes(), nil +} + +func setPreviewArg(container *yaml.RNode, arg string) error { + image, err := container.GetString("image") + if err != nil || !strings.HasPrefix(image, substrateImagePrefix) { + return nil + } + args, err := container.Pipe(yaml.LookupCreate(yaml.SequenceNode, "args")) + if err != nil { + return err + } + kept := args.YNode().Content[:0] + for _, item := range args.YNode().Content { + if strings.HasPrefix(item.Value, previewFlag+"=") { + continue + } + kept = append(kept, item) + } + args.YNode().Content = append(kept, yaml.NewStringRNode(arg).YNode()) + return nil +} diff --git a/cmd/ate-setup/internal/steps/preview_test.go b/cmd/ate-setup/internal/steps/preview_test.go new file mode 100644 index 0000000000..18de8c6ef8 --- /dev/null +++ b/cmd/ate-setup/internal/steps/preview_test.go @@ -0,0 +1,352 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package steps + +import ( + "bytes" + "context" + "errors" + "io" + "slices" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" + yamlv3 "gopkg.in/yaml.v3" + "sigs.k8s.io/kustomize/kyaml/kio" + "sigs.k8s.io/kustomize/kyaml/yaml" + + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/config" + "github.com/agent-substrate/substrate/cmd/ate-setup/internal/kube" +) + +// containerArgs maps "//" to the container's args. +func containerArgs(t *testing.T, manifest []byte) map[string][]string { + t.Helper() + nodes, err := (&kio.ByteReader{Reader: bytes.NewReader(manifest), OmitReaderAnnotations: true}).Read() + if err != nil { + t.Fatalf("parsing manifest: %v", err) + } + out := map[string][]string{} + for _, node := range nodes { + for _, path := range containerListPaths { + containers, err := node.Pipe(yaml.Lookup(path...)) + if err != nil || containers == nil { + continue + } + elements, err := containers.Elements() + if err != nil { + t.Fatalf("listing containers: %v", err) + } + for _, c := range elements { + name, _ := c.GetString("name") + image, _ := c.GetString("image") + key := node.GetKind() + "/" + node.GetName() + "/" + name + var args []string + if list, _ := c.Pipe(yaml.Lookup("args")); list != nil { + for _, item := range list.YNode().Content { + args = append(args, item.Value) + } + } + // Tag the substrate containers so callers can tell them apart + // without repeating the prefix check. + if strings.HasPrefix(image, substrateImagePrefix) { + key = "substrate:" + key + } + out[key] = args + } + } + } + return out +} + +const previewTestManifest = `# leading comment +apiVersion: v1 +kind: ConfigMap +metadata: + name: cfg +data: + image: ko://github.com/agent-substrate/substrate/cmd/ateapi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: api +spec: + template: + spec: + initContainers: + - name: sidecar + image: ko://github.com/agent-substrate/substrate/cmd/atenet + restartPolicy: Always + args: + - "sdsmint" + containers: + - name: api + image: ko://github.com/agent-substrate/substrate/cmd/ateapi + args: + - --port=8080 # keep me + - --preview=Stale + - name: bare + image: ko://github.com/agent-substrate/substrate/cmd/atelet + - name: envoy + image: ${ENVOY_DATAPLANE_IMAGE} + args: + - -c +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod +spec: + containers: + - name: ctl + image: ko://github.com/agent-substrate/substrate/cmd/atecontroller +` + +func TestInjectPreviewArg(t *testing.T) { + for _, tc := range []struct { + enabled []string + arg string + }{ + {enabled: []string{"ALL"}, arg: "--preview=ALL"}, + {enabled: nil, arg: "--preview="}, + } { + t.Run(tc.arg, func(t *testing.T) { + out, err := injectPreviewArg([]byte(previewTestManifest), tc.enabled) + if err != nil { + t.Fatalf("injectPreviewArg: %v", err) + } + got := containerArgs(t, out) + want := map[string][]string{ + "substrate:Deployment/api/sidecar": {"sdsmint", tc.arg}, + "substrate:Deployment/api/api": {"--port=8080", tc.arg}, + "substrate:Deployment/api/bare": {tc.arg}, + "Deployment/api/envoy": {"-c"}, + "substrate:Pod/pod/ctl": {tc.arg}, + } + for key, wantArgs := range want { + if !slices.Equal(got[key], wantArgs) { + t.Errorf("%s args = %q, want %q", key, got[key], wantArgs) + } + } + if len(got) != len(want) { + t.Errorf("got containers %v, want %d", got, len(want)) + } + // Everything else survives: comments, placeholders, and the + // ConfigMap that merely mentions a substrate image. + for _, s := range []string{"# keep me", "${ENVOY_DATAPLANE_IMAGE}", "kind: ConfigMap", "image: ko://github.com/agent-substrate/substrate/cmd/ateapi\n---"} { + if !strings.Contains(string(out), s) { + t.Errorf("output lost %q:\n%s", s, out) + } + } + }) + } +} + +func TestInjectPreviewArgLeavesOtherManifestsAlone(t *testing.T) { + in := []byte("apiVersion: v1\nkind: Pod\nmetadata: {name: x}\nspec:\n containers:\n - {name: c, image: busybox}\n") + for _, enabled := range [][]string{{"ALL"}, {}} { + out, err := injectPreviewArg(in, enabled) + if err != nil { + t.Fatalf("injectPreviewArg: %v", err) + } + if !bytes.Equal(out, in) { + t.Errorf("enabled=%v: manifest changed:\n%s", enabled, out) + } + } +} + +type recordingResolver struct{ got []byte } + +func (r *recordingResolver) ResolveBytes(_ context.Context, manifest []byte) ([]byte, error) { + r.got = manifest + return manifest, nil +} + +// Every component manifest ate-setup ships, rendered the way it applies it, +// gets --preview on each substrate container, and only once. +func TestResolveManifestInjectsPreview(t *testing.T) { + root := repoRoot(t) + sources := []struct { + name string + cfg config.Config + path func(e *Env) string + // direct resolves path without rendering it, as DeployAtelet does. + direct bool + // want names substrate containers the rendered manifest must carry. + want []string + }{ + { + name: "base bundle", + cfg: config.Config{Router: config.RouterEnvoy}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + want: []string{"ate-api-server", "ate-controller", "atenet-router", "atelet"}, + }, + { + name: "kind bundle", + cfg: config.Config{Router: config.RouterEnvoy, Kind: true}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + want: []string{"ate-api-server", "ate-controller", "atenet-router", "atelet"}, + }, + { + name: "agentgateway bundle", + cfg: config.Config{Router: config.RouterAgentgateway}, + path: func(e *Env) string { return e.Cfg.Path(SystemOverlay(e.Cfg)) }, + want: []string{"ate-api-server", "ate-controller"}, + }, + { + name: "atelet file", + path: func(e *Env) string { return e.Cfg.Manifest("atelet.yaml") }, + direct: true, + want: []string{"atelet"}, + }, + { + name: "podcert file", + path: func(e *Env) string { return e.Cfg.Manifest("pod-certificate-controller.yaml") }, + want: []string{"podcertificate-controller"}, + }, + { + name: "podcert size10 overlay", + cfg: config.Config{ClusterSize: config.ClusterSizeSize10}, + path: func(e *Env) string { return e.Cfg.Manifest("podcert-size10") }, + want: []string{"podcertificate-controller"}, + }, + { + name: "router file", + path: func(e *Env) string { return e.Cfg.Manifest("atenet-router.yaml") }, + want: []string{"atenet-router"}, + }, + { + name: "egress file", + path: func(e *Env) string { return e.atenetEgressManifestPath() }, + want: []string{"atenet-egress"}, + }, + { + name: "credential provider file", + path: func(e *Env) string { + return e.Cfg.Path("manifests", "egress-credential-injection", "k8s-credential-provider.yaml") + }, + want: []string{"k8s-credential-provider"}, + }, + } + for _, src := range sources { + for _, enabled := range [][]string{{"ALL"}, {}} { + arg := previewArg(enabled) + t.Run(src.name+"/"+arg, func(t *testing.T) { + cfg := src.cfg + cfg.Root = root + cfg.PreviewGates = enabled + resolver := &recordingResolver{} + e := &Env{Cfg: &cfg, resolver: resolver} + + path := src.path(e) + var err error + if src.direct { + _, err = e.ResolveManifest(context.Background(), path) + } else { + _, err = e.renderResolve(context.Background(), path) + } + if err != nil { + t.Fatalf("resolving %s: %v", path, err) + } + + var original []byte + if src.direct { + original, err = kube.ReadPath(path) + } else { + original, err = e.render(path) + } + if err != nil { + t.Fatalf("rendering %s: %v", path, err) + } + if diff := cmp.Diff(decodeAll(t, original), withoutPreview(decodeAll(t, resolver.got))); diff != "" { + t.Errorf("injection changed more than --preview (-before +after):\n%s", diff) + } + + found := map[string]bool{} + for key, args := range containerArgs(t, resolver.got) { + previews := 0 + for _, a := range args { + if strings.HasPrefix(a, previewFlag) { + previews++ + } + } + if !strings.HasPrefix(key, "substrate:") { + if previews != 0 { + t.Errorf("%s is not a substrate container but got %q", key, args) + } + continue + } + if previews != 1 || !slices.Contains(args, arg) { + t.Errorf("%s args = %q, want exactly one %s", key, args, arg) + } + found[strings.Split(key, "/")[1]] = true + } + for _, name := range src.want { + if !found[name] { + t.Errorf("no substrate container in workload %s (found %v)", name, found) + } + } + }) + } + } +} + +// decodeAll decodes every non-empty document in a YAML stream. +func decodeAll(t *testing.T, stream []byte) []any { + t.Helper() + var docs []any + dec := yamlv3.NewDecoder(bytes.NewReader(stream)) + for { + var doc any + err := dec.Decode(&doc) + if errors.Is(err, io.EOF) { + return docs + } + if err != nil { + t.Fatalf("decoding manifest: %v", err) + } + if doc != nil { + docs = append(docs, doc) + } + } +} + +// withoutPreview strips every --preview argument, and any args list left empty +// by doing so, from decoded YAML. +func withoutPreview(v any) any { + switch v := v.(type) { + case []any: + out := make([]any, 0, len(v)) + for _, item := range v { + if s, ok := item.(string); ok && strings.HasPrefix(s, previewFlag+"=") { + continue + } + out = append(out, withoutPreview(item)) + } + return out + case map[string]any: + for k, item := range v { + v[k] = withoutPreview(item) + } + if args, ok := v["args"].([]any); ok && len(args) == 0 { + delete(v, "args") + } + return v + default: + return v + } +} diff --git a/cmd/ate-setup/internal/steps/publish.go b/cmd/ate-setup/internal/steps/publish.go index d25ed6c493..f3004188c9 100644 --- a/cmd/ate-setup/internal/steps/publish.go +++ b/cmd/ate-setup/internal/steps/publish.go @@ -23,12 +23,12 @@ import ( "github.com/agent-substrate/substrate/cmd/ate-setup/internal/log" ) -// workerImages are the ateom images a WorkerPool points at through +// workerImages are the worker images a WorkerPool points at through // workerImage, one per sandbox class. No manifest references them, so the // install never publishes them as a side effect. -var workerImages = []string{"ateom-gvisor", "ateom-microvm"} +var workerImages = []string{"ateworker-gvisor", "ateworker-microvm"} -// PublishWorkerImages builds and pushes the ateom images for this build and +// PublishWorkerImages builds and pushes the worker images for this build and // writes their pushed references to w, one ": " line per image, // after every build has finished so the refs sit together below ko's build // output. A WorkerPool moves to this build by pointing its workerImage at diff --git a/cmd/ate-setup/internal/steps/substrate.go b/cmd/ate-setup/internal/steps/substrate.go index 76bd0b4f55..a9d5999f50 100644 --- a/cmd/ate-setup/internal/steps/substrate.go +++ b/cmd/ate-setup/internal/steps/substrate.go @@ -19,6 +19,7 @@ import ( "fmt" "time" + "google.golang.org/grpc" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" "google.golang.org/protobuf/encoding/protojson" @@ -26,13 +27,19 @@ import ( "github.com/agent-substrate/substrate/cmd/ate-setup/internal/log" "github.com/agent-substrate/substrate/internal/ateclient" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) -// AteClient connects to the ate-api-server, port-forwarding if needed. +// AteClient connects to the ate-api-server, port-forwarding if needed. It +// opts in to preview features if the install enables any. func (e *Env) AteClient(ctx context.Context) (*ateclient.Client, error) { - return ateclient.NewClient(ctx, e.Cfg.Kubeconfig, e.Cfg.Context, "", "", false) + var opts []grpc.DialOption + if len(e.Cfg.PreviewGates) > 0 { + opts = preview.DialOptionsWithOptIn() + } + return ateclient.NewClient(ctx, e.Cfg.Kubeconfig, e.Cfg.Context, "", "", false, opts...) } // EnsureAtespace creates the atespace if it does not already exist. The store diff --git a/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go b/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go index 48d414ed8a..26feb30709 100644 --- a/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go +++ b/cmd/ate-setup/internal/steps/systemnetworkpolicy_test.go @@ -113,7 +113,7 @@ func loadSystemInstall(t *testing.T, cfg *config.Config) systemInstall { objs = append(objs, decoded...) } system := filepath.Join(root, SystemOverlay(cfg)) - rendered, err := e.renderDataPlane(system) + rendered, err := e.render(system) add(system, rendered, err) rendered, err = e.render(e.postgresManifestPath()) add(e.postgresManifestPath(), rendered, err) diff --git a/cmd/ateapi/helm_test.go b/cmd/ateapi/helm_test.go index 0b9760520d..6232627409 100644 --- a/cmd/ateapi/helm_test.go +++ b/cmd/ateapi/helm_test.go @@ -63,6 +63,45 @@ func TestHelmAPIServerFlags(t *testing.T) { } } +func TestHelmPreviewGates(t *testing.T) { + for _, gates := range []string{"", "ExternalVolumes", "ALL"} { + t.Run("preview="+gates, func(t *testing.T) { + args := []string{"template", "test", "../../charts/substrate", "-n", "custom"} + if gates != "" { + args = append(args, "--set", "preview[0]="+gates) + } + out, err := exec.CommandContext(t.Context(), "helm", args...).CombinedOutput() + if err != nil { + t.Fatalf("render: %v\n%s", err, out) + } + found := map[string]bool{"ate-api-server": false, "ate-controller": false, "atelet": false} + for _, doc := range strings.Split(string(out), "\n---\n") { + var workload appsv1.Deployment + if err := yaml.Unmarshal([]byte(doc), &workload); err != nil { + t.Fatal(err) + } + if workload.Kind != "Deployment" && workload.Kind != "DaemonSet" { + continue + } + for _, container := range workload.Spec.Template.Spec.Containers { + if _, ok := found[container.Name]; !ok { + continue + } + found[container.Name] = true + if !slices.Contains(container.Args, "--preview="+gates) { + t.Errorf("%s args = %v, want --preview=%s", container.Name, container.Args, gates) + } + } + } + for name, present := range found { + if !present { + t.Errorf("missing %s container", name) + } + } + }) + } +} + func TestHelmPostgresCertificates(t *testing.T) { for _, tc := range []struct { release, namespace string @@ -273,7 +312,7 @@ func TestHelmObjectStorePlugins(t *testing.T) { t.Error("API server still receives ATE_STORAGE_BACKEND") } } - if mode == "node" && (!slices.Contains(plugin.Args, "--root=/var/lib/ate") || !slices.ContainsFunc(plugin.VolumeMounts, func(m corev1.VolumeMount) bool { return m.Name == "run-ateom" && m.MountPath == "/var/lib/ate" })) { + if mode == "node" && (!slices.Contains(plugin.Args, "--root=/var/lib/ate") || !slices.ContainsFunc(plugin.VolumeMounts, func(m corev1.VolumeMount) bool { return m.Name == "ate-base" && m.MountPath == "/var/lib/ate" })) { t.Error("node plugin cannot access snapshot files") } found[mode] = true diff --git a/cmd/ateapi/internal/apivalidation/access_policy.go b/cmd/ateapi/internal/apivalidation/access_policy.go index 2871c9f45a..726a0e1d8a 100644 --- a/cmd/ateapi/internal/apivalidation/access_policy.go +++ b/cmd/ateapi/internal/apivalidation/access_policy.go @@ -35,20 +35,20 @@ var ( ) func ValidateCreateGlobalAccessPolicyRequest(ctx context.Context, req *ateapipb.CreateGlobalAccessPolicyRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) errs := Validate_CreateGlobalAccessPolicyRequest(ctx, op, nil, req, nil) return append(errs, validateBindings(bindingsPath, req.GetAccessPolicy(), validGlobalRoles)...) } func ValidateGetGlobalAccessPolicyRequest(ctx context.Context, req *ateapipb.GetGlobalAccessPolicyRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_GetGlobalAccessPolicyRequest(ctx, op, nil, req, nil) } func ValidateUpdateGlobalAccessPolicyRequest(ctx context.Context, req *ateapipb.UpdateGlobalAccessPolicyRequest) field.ErrorList { // Modeled as a create: this validates the request itself. The result is // validated against the stored value by ValidateGlobalAccessPolicyUpdate. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) errs := Validate_UpdateGlobalAccessPolicyRequest(ctx, op, nil, req, nil) return append(errs, validateBindings(bindingsPath, req.GetAccessPolicy(), validGlobalRoles)...) } @@ -56,26 +56,26 @@ func ValidateUpdateGlobalAccessPolicyRequest(ctx context.Context, req *ateapipb. // ValidateGlobalAccessPolicyUpdate validates an updated global policy against // the stored one. func ValidateGlobalAccessPolicyUpdate(ctx context.Context, fldPath *field.Path, newVal, oldVal *ateapipb.AccessPolicy) field.ErrorList { - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(ctx) errs := Validate_AccessPolicy(ctx, op, fldPath, newVal, oldVal) return append(errs, validateBindings(fldPath.Child("bindings"), newVal, validGlobalRoles)...) } func ValidateCreateAtespaceAccessPolicyRequest(ctx context.Context, req *ateapipb.CreateAtespaceAccessPolicyRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) errs := Validate_CreateAtespaceAccessPolicyRequest(ctx, op, nil, req, nil) return append(errs, validateBindings(bindingsPath, req.GetAccessPolicy(), validAtespaceRoles)...) } func ValidateGetAtespaceAccessPolicyRequest(ctx context.Context, req *ateapipb.GetAtespaceAccessPolicyRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_GetAtespaceAccessPolicyRequest(ctx, op, nil, req, nil) } func ValidateUpdateAtespaceAccessPolicyRequest(ctx context.Context, req *ateapipb.UpdateAtespaceAccessPolicyRequest) field.ErrorList { // Modeled as a create: this validates the request itself. The result is // validated against the stored value by ValidateAtespaceAccessPolicyUpdate. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) errs := Validate_UpdateAtespaceAccessPolicyRequest(ctx, op, nil, req, nil) return append(errs, validateBindings(bindingsPath, req.GetAccessPolicy(), validAtespaceRoles)...) } @@ -83,13 +83,13 @@ func ValidateUpdateAtespaceAccessPolicyRequest(ctx context.Context, req *ateapip // ValidateAtespaceAccessPolicyUpdate validates an updated atespace policy // against the stored one. func ValidateAtespaceAccessPolicyUpdate(ctx context.Context, fldPath *field.Path, newVal, oldVal *ateapipb.AccessPolicy) field.ErrorList { - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(ctx) errs := Validate_AccessPolicy(ctx, op, fldPath, newVal, oldVal) return append(errs, validateBindings(fldPath.Child("bindings"), newVal, validAtespaceRoles)...) } func ValidateDeleteAtespaceAccessPolicyRequest(ctx context.Context, req *ateapipb.DeleteAtespaceAccessPolicyRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_DeleteAtespaceAccessPolicyRequest(ctx, op, nil, req, nil) } diff --git a/cmd/ateapi/internal/apivalidation/actor.go b/cmd/ateapi/internal/apivalidation/actor.go index fa6a623786..a77942fcd3 100644 --- a/cmd/ateapi/internal/apivalidation/actor.go +++ b/cmd/ateapi/internal/apivalidation/actor.go @@ -26,19 +26,19 @@ import ( func ValidateCreateActorRequest(ctx context.Context, req *ateapipb.CreateActorRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_CreateActorRequest(ctx, op, nil, req, nil) } func ValidateGetActorRequest(ctx context.Context, req *ateapipb.GetActorRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_GetActorRequest(ctx, op, nil, req, nil) } func ValidateListActorsRequest(ctx context.Context, req *ateapipb.ListActorsRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_ListActorsRequest(ctx, op, nil, req, nil) } @@ -48,65 +48,65 @@ func ValidateUpdateActorRequest(ctx context.Context, req *ateapipb.UpdateActorRe // the existence of a "current" value, which we do not have yet. This is // validating the request itself. The result will be validated later, after // we have a current value to compare against. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_UpdateActorRequest(ctx, op, nil, req, nil) } func ValidateDeleteActorRequest(ctx context.Context, req *ateapipb.DeleteActorRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_DeleteActorRequest(ctx, op, nil, req, nil) } func ValidatePauseActorRequest(ctx context.Context, req *ateapipb.PauseActorRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_PauseActorRequest(ctx, op, nil, req, nil) } func ValidateResumeActorRequest(ctx context.Context, req *ateapipb.ResumeActorRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_ResumeActorRequest(ctx, op, nil, req, nil) } func ValidateSuspendActorRequest(ctx context.Context, req *ateapipb.SuspendActorRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_SuspendActorRequest(ctx, op, nil, req, nil) } func ValidateRevertActorRequest(ctx context.Context, req *ateapipb.RevertActorRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_RevertActorRequest(ctx, op, nil, req, nil) } func ValidateMintActorJWTRequest(ctx context.Context, req *ateapipb.MintActorJwtRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_MintActorJwtRequest(ctx, op, nil, req, nil) } func ValidateMintActorCertificateRequest(ctx context.Context, req *ateapipb.MintActorCertificateRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_MintActorCertificateRequest(ctx, op, nil, req, nil) } -func ValidateMintAteomActorCertificateRequest(ctx context.Context, req *ateapipb.MintAteomActorCertificateRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} - return Validate_MintAteomActorCertificateRequest(ctx, op, nil, req, nil) +func ValidateMintWorkerActorCertificateRequest(ctx context.Context, req *ateapipb.MintWorkerActorCertificateRequest) field.ErrorList { + op := MakeCreateOp(ctx) + return Validate_MintWorkerActorCertificateRequest(ctx, op, nil, req, nil) } -func ValidateMintAteomActorJwtRequest(ctx context.Context, req *ateapipb.MintAteomActorJwtRequest) field.ErrorList { +func ValidateMintWorkerActorJwtRequest(ctx context.Context, req *ateapipb.MintWorkerActorJwtRequest) field.ErrorList { op := operation.Operation{Type: operation.Create} - return Validate_MintAteomActorJwtRequest(ctx, op, nil, req, nil) + return Validate_MintWorkerActorJwtRequest(ctx, op, nil, req, nil) } func ValidateActorUpdate(ctx context.Context, fldPath *field.Path, newVal, oldVal *ateapipb.Actor, requireStatus bool) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(ctx) errs := Validate_Actor(ctx, op, fldPath, newVal, oldVal) if requireStatus { // Status is optional in the schema, but is actually required to be set diff --git a/cmd/ateapi/internal/apivalidation/actor_template.go b/cmd/ateapi/internal/apivalidation/actor_template.go index 15599571b8..f906c2e8b5 100644 --- a/cmd/ateapi/internal/apivalidation/actor_template.go +++ b/cmd/ateapi/internal/apivalidation/actor_template.go @@ -29,31 +29,31 @@ import ( func ValidateCreateActorTemplateRequest(ctx context.Context, req *ateapipb.CreateActorTemplateRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_CreateActorTemplateRequest(ctx, op, nil, req, nil) } func ValidateGetActorTemplateRequest(ctx context.Context, req *ateapipb.GetActorTemplateRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_GetActorTemplateRequest(ctx, op, nil, req, nil) } func ValidateListActorTemplatesRequest(ctx context.Context, req *ateapipb.ListActorTemplatesRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_ListActorTemplatesRequest(ctx, op, nil, req, nil) } func ValidateDeleteActorTemplateRequest(ctx context.Context, req *ateapipb.DeleteActorTemplateRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_DeleteActorTemplateRequest(ctx, op, nil, req, nil) } func ValidateActorTemplateUpdate(ctx context.Context, fldPath *field.Path, newVal, oldVal *ateapipb.ActorTemplate) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(ctx) return Validate_ActorTemplate(ctx, op, fldPath, newVal, oldVal) } @@ -91,6 +91,22 @@ func ValidateCustom_CreateActorTemplateRequest_ActorTemplate(_ context.Context, return errs } +// ValidateCustom_ActorTemplate_Resources requires both a cpu and a memory +// limit, which the scheduler books against worker capacity. +func ValidateCustom_ActorTemplate_Resources(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *ateapipb.Resources) field.ErrorList { + set := sets.New[string]() + for _, l := range value.GetLimits() { + set.Insert(l.GetName()) + } + var errs field.ErrorList + for _, name := range []string{resources.ResourceCPU, resources.ResourceMemory} { + if !set.Has(name) { + errs = append(errs, field.Required(fldPath.Child("limits"), "must set a "+name+" limit")) + } + } + return errs +} + func ValidateCustom_HttpGetAction_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList { return resources.ValidateHTTPGetPath(fldPath, *value) } diff --git a/cmd/ateapi/internal/apivalidation/actor_template_test.go b/cmd/ateapi/internal/apivalidation/actor_template_test.go index 4cf66a6db7..ebbba44356 100644 --- a/cmd/ateapi/internal/apivalidation/actor_template_test.go +++ b/cmd/ateapi/internal/apivalidation/actor_template_test.go @@ -20,8 +20,9 @@ import ( "strings" "testing" + "github.com/agent-substrate/substrate/internal/preview" + "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" - "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/util/validation/field" ) @@ -178,6 +179,33 @@ func TestValidateCreateActorTemplateRequest(t *testing.T) { tmpl.SandboxConfig.ConfigName = "" })}, field.ErrorList{field.Required(field.NewPath("actor_template", "sandbox_config", "config_name"), "")}, + }, { + "missing resources", + &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { + tmpl.Resources = nil + })}, + field.ErrorList{field.Required(field.NewPath("actor_template", "resources"), "")}, + }, { + "missing cpu limit", + &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { + tmpl.Resources = resources.CPUMemory(0, 512<<20) + })}, + field.ErrorList{field.Required(field.NewPath("actor_template", "resources", "limits"), "must set a cpu limit")}, + }, { + "missing memory limit", + &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { + tmpl.Resources = resources.CPUMemory(1000, 0) + })}, + field.ErrorList{field.Required(field.NewPath("actor_template", "resources", "limits"), "must set a memory limit")}, + }, { + "empty limits", + &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { + tmpl.Resources = &ateapipb.Resources{} + })}, + field.ErrorList{ + field.Required(field.NewPath("actor_template", "resources", "limits"), "must set a cpu limit"), + field.Required(field.NewPath("actor_template", "resources", "limits"), "must set a memory limit"), + }, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -186,6 +214,22 @@ func TestValidateCreateActorTemplateRequest(t *testing.T) { } } +func TestValidateCreateActorTemplateRequest_PreviewRestricted(t *testing.T) { + setPreviewForTest(t, "ALL") + req := &ateapipb.CreateActorTemplateRequest{ActorTemplate: validActorTemplate(func(tmpl *ateapipb.ActorTemplate) { + tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "fast-ssd"}}} + })} + + t.Run("allowed", func(t *testing.T) { + assertValidateErr(t, ValidateCreateActorTemplateRequest(context.Background(), req), nil) + }) + t.Run("restricted", func(t *testing.T) { + ctx := preview.ContextWithOptOut(context.Background()) + want := field.ErrorList{field.Forbidden(field.NewPath("actor_template", "volumes").Index(0).Child("external_volume_template"), "")} + assertValidateErr(t, ValidateCreateActorTemplateRequest(ctx, req), want) + }) +} + func TestValidateGetActorTemplateRequest(t *testing.T) { tests := []struct { name string @@ -284,7 +328,11 @@ func TestValidateActorTemplate(t *testing.T) { tests := []struct { name string mutate func(*ateapipb.ActorTemplate) // nil leaves the template valid - want field.ErrorList + // want is the result with no preview gates enabled. + want field.ErrorList + // wantWithPreview maps a --preview value to the result with those + // gates enabled. + wantWithPreview map[string]field.ErrorList }{{ name: "valid", }, { @@ -786,41 +834,50 @@ func TestValidateActorTemplate(t *testing.T) { mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "fast-ssd"}}} }, + want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")}, + wantWithPreview: map[string]field.ErrorList{"ALL": nil}, }, { name: "external volume template missing capacity", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast-ssd"}}} }, - want: field.ErrorList{field.Required(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), "")}, + want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")}, + wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), "")}}, }, { name: "external volume template malformed capacity", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "ten gigs", StorageClassName: "fast-ssd"}}} }, - want: field.ErrorList{field.Invalid(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), nil, "")}, + want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")}, + wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), nil, "")}}, }, { name: "external volume template capacity at the length bound", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: strings.Repeat("1", 30) + "Gi", StorageClassName: "fast-ssd"}}} }, + want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")}, + wantWithPreview: map[string]field.ErrorList{"ALL": nil}, }, { name: "external volume template capacity too long", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: strings.Repeat("1", 31) + "Gi", StorageClassName: "fast-ssd"}}} }, - want: field.ErrorList{field.TooLong(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), nil, 32).WithOrigin("maxLength")}, + want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")}, + wantWithPreview: map[string]field.ErrorList{"ALL": {field.TooLong(field.NewPath("volumes").Index(0).Child("external_volume_template", "capacity"), nil, 32).WithOrigin("maxLength")}}, }, { name: "external volume template missing storage_class_name", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi"}}} }, - want: field.ErrorList{field.Required(field.NewPath("volumes").Index(0).Child("external_volume_template", "storage_class_name"), "")}, + want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")}, + wantWithPreview: map[string]field.ErrorList{"ALL": {field.Required(field.NewPath("volumes").Index(0).Child("external_volume_template", "storage_class_name"), "")}}, }, { name: "external volume template invalid storage_class_name", mutate: func(tmpl *ateapipb.ActorTemplate) { tmpl.Volumes = []*ateapipb.Volume{{Name: "data", ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{Capacity: "10Gi", StorageClassName: "Fast SSD"}}} }, - want: field.ErrorList{field.Invalid(field.NewPath("volumes").Index(0).Child("external_volume_template", "storage_class_name"), nil, "").WithOrigin("format=k8s-long-name")}, + want: field.ErrorList{field.Forbidden(field.NewPath("volumes").Index(0).Child("external_volume_template"), "")}, + wantWithPreview: map[string]field.ErrorList{"ALL": {field.Invalid(field.NewPath("volumes").Index(0).Child("external_volume_template", "storage_class_name"), nil, "").WithOrigin("format=k8s-long-name")}}, }, { name: "valid resources", mutate: func(tmpl *ateapipb.ActorTemplate) { @@ -875,9 +932,12 @@ func TestValidateActorTemplate(t *testing.T) { }, { name: "template-level resources validated too", mutate: func(tmpl *ateapipb.ActorTemplate) { - tmpl.Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "gpu", Quantity: "1"}}} + tmpl.Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "gpu", Quantity: "1"}}} + }, + want: field.ErrorList{ + field.NotSupported[string](field.NewPath("resources", "limits").Index(1).Child("name"), nil, nil), + field.Required(field.NewPath("resources", "limits"), "must set a memory limit"), }, - want: field.ErrorList{field.NotSupported[string](field.NewPath("resources", "limits").Index(0).Child("name"), nil, nil)}, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { @@ -885,8 +945,10 @@ func TestValidateActorTemplate(t *testing.T) { if tt.mutate != nil { tt.mutate(tmpl) } - op := operation.Operation{Type: operation.Create} - assertValidateErr(t, Validate_ActorTemplate(context.Background(), op, nil, tmpl, nil), tt.want) + assertValidate(t, func() field.ErrorList { + op := MakeCreateOp(context.Background()) + return Validate_ActorTemplate(context.Background(), op, nil, tmpl, nil) + }, tt.want, tt.wantWithPreview) }) } } @@ -904,6 +966,7 @@ func validActorTemplate(mutations ...func(*ateapipb.ActorTemplate)) *ateapipb.Ac PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, + Resources: resources.CPUMemory(1000, 512<<20), } for _, m := range mutations { m(template) diff --git a/cmd/ateapi/internal/apivalidation/actor_test.go b/cmd/ateapi/internal/apivalidation/actor_test.go index ac3f0c6163..dea6ab5882 100644 --- a/cmd/ateapi/internal/apivalidation/actor_test.go +++ b/cmd/ateapi/internal/apivalidation/actor_test.go @@ -182,198 +182,192 @@ func TestValidateActorUpdate(t *testing.T) { name string oldVal *ateapipb.Actor newVal *ateapipb.Actor - want field.ErrorList + // want is the result with no preview gates enabled. + want field.ErrorList + // wantWithPreview maps a --preview value to the result with those + // gates enabled. + wantWithPreview map[string]field.ErrorList }{{ - "valid", - validInput(), - validOutput(), - nil, - }, { - "missing actor.metadata", - validInput(), - validOutput(func(a *ateapipb.Actor) { a.Metadata = nil }), - field.ErrorList{field.Required(field.NewPath("metadata"), "")}, - }, { - "missing actor.metadata.atespace", - validInput(), - validOutput(withMetadata(func(m *ateapipb.ResourceMetadata) { m.Atespace = "" })), - field.ErrorList{ + name: "valid", + oldVal: validInput(), + newVal: validOutput(), + }, { + name: "missing actor.metadata", + oldVal: validInput(), + newVal: validOutput(func(a *ateapipb.Actor) { a.Metadata = nil }), + want: field.ErrorList{field.Required(field.NewPath("metadata"), "")}, + }, { + name: "missing actor.metadata.atespace", + oldVal: validInput(), + newVal: validOutput(withMetadata(func(m *ateapipb.ResourceMetadata) { m.Atespace = "" })), + want: field.ErrorList{ field.Required(field.NewPath("metadata", "atespace"), ""), field.Invalid(field.NewPath("metadata", "atespace"), nil, "").WithOrigin("immutable"), }, }, { - "invalid actor.metadata.atespace", - validInput(), - validOutput(withMetadata(func(m *ateapipb.ResourceMetadata) { m.Atespace = "invalid value" })), - field.ErrorList{field.Invalid(field.NewPath("metadata", "atespace"), nil, "").WithOrigin("immutable")}, + name: "invalid actor.metadata.atespace", + oldVal: validInput(), + newVal: validOutput(withMetadata(func(m *ateapipb.ResourceMetadata) { m.Atespace = "invalid value" })), + want: field.ErrorList{field.Invalid(field.NewPath("metadata", "atespace"), nil, "").WithOrigin("immutable")}, }, { - "missing actor.metadata.name", - validInput(), - validOutput(withMetadata(func(m *ateapipb.ResourceMetadata) { m.Name = "" })), - field.ErrorList{ + name: "missing actor.metadata.name", + oldVal: validInput(), + newVal: validOutput(withMetadata(func(m *ateapipb.ResourceMetadata) { m.Name = "" })), + want: field.ErrorList{ field.Required(field.NewPath("metadata", "name"), ""), field.Invalid(field.NewPath("metadata", "name"), nil, "").WithOrigin("immutable"), }, }, { - "invalid actor.metadata.name", - validInput(), - validOutput(withMetadata(func(m *ateapipb.ResourceMetadata) { m.Name = "invalid value" })), - field.ErrorList{field.Invalid(field.NewPath("metadata", "name"), nil, "").WithOrigin("immutable")}, - }, { - "change actor.actor_template is allowed", - validInput(withActorTemplate("as1", "nm1")), - validOutput(withActorTemplate("as2", "nm2")), - nil, - }, { - "clear actor.actor_template", - validInput(withActorTemplate("as", "nm")), - validOutput(func(a *ateapipb.Actor) { a.ActorTemplate = nil }), - field.ErrorList{field.Required(field.NewPath("actor_template"), "")}, - }, { - "add actor.source_tag", - validInput(), - validOutput(withSourceTag("as", "nm")), - field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, - }, { - "clear actor.source_tag", - validInput(withSourceTag("as", "nm")), - validOutput(func(a *ateapipb.Actor) { a.SourceTag = nil }), - field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, - }, { - "change actor.source_tag", - validInput(withSourceTag("as1", "nm1")), - validOutput(withSourceTag("as2", "nm2")), - field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, - }, { - "set valid worker_selector", - validInput(), - validOutput(withWorkerSelector(map[string]string{"tier": "1"})), - nil, - }, { - "clear worker_selector", - validInput(withWorkerSelector(map[string]string{"tier": "1"})), - validOutput(), - nil, - }, { - "modify worker_selector", - validInput(withWorkerSelector(map[string]string{"tier": "1"})), - validOutput(withWorkerSelector(map[string]string{"tier": "2"})), - nil, - }, { - "invalid worker_selector with nil match_labels", - validInput(), - validOutput(func(a *ateapipb.Actor) { a.WorkerSelector = &ateapipb.Selector{} }), - field.ErrorList{field.Invalid(field.NewPath("worker_selector"), nil, "one of").WithOrigin("union")}, - }, { - "invalid worker_selector label key", - validInput(), - validOutput(withWorkerSelector(map[string]string{"bad key": "2"})), - field.ErrorList{field.Invalid(field.NewPath("worker_selector", "match_labels"), nil, "").WithOrigin("format=k8s-label-key")}, - }, { - "invalid worker_selector label value", - validInput(), - validOutput(withWorkerSelector(map[string]string{"tier": "bad value"})), - field.ErrorList{field.Invalid(field.NewPath("worker_selector", "match_labels").Key("tier"), nil, "").WithOrigin("format=k8s-label-value")}, - }, { - "too many worker_selector.match_labels", - validInput(), - validOutput(withWorkerSelector(selectorLabelsOfSize(11))), - field.ErrorList{field.TooMany(field.NewPath("worker_selector", "match_labels"), 11, 10).WithOrigin("maxProperties")}, - }, { - "add actor.source_tag", - validInput(), - validOutput(withSourceTag("as", "nm")), - field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, - }, { - "clear actor.source_tag", - validInput(withSourceTag("as", "nm")), - validOutput(func(a *ateapipb.Actor) { a.SourceTag = nil }), - field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, - }, { - "change actor.source_tag", - validInput(withSourceTag("as1", "nm1")), - validOutput(withSourceTag("as2", "nm2")), - field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, - }, { - "unspecified actor.status", - validInput(withStatus()), - validOutput(func(a *ateapipb.Actor) { a.Status = nil }), - field.ErrorList{field.Required(field.NewPath("status"), "")}, - }, { - "unspecified actor.status.state", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = 0 })), - field.ErrorList{field.Required(field.NewPath("status", "state"), "")}, - }, { - "change actor.status.state", - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = ateapipb.ActorState_ACTOR_STATE_RUNNING })), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = ateapipb.ActorState_ACTOR_STATE_CRASHED })), - nil, - }, { - "negative actor.status.state", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = -1 })), - field.ErrorList{field.Invalid(field.NewPath("status", "state"), nil, "").WithOrigin("minimum")}, - }, { - "just out of bounds actor.status.state", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = 10 })), - field.ErrorList{field.Invalid(field.NewPath("status", "state"), nil, "").WithOrigin("maximum")}, - }, { - "invalid actor.status.state", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = 1234567890 })), - field.ErrorList{field.Invalid(field.NewPath("status", "state"), nil, "").WithOrigin("maximum")}, - }, { - "set valid actor.status.worker_assignment, IPv4", - validInput(withStatus()), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"1.2.3.4"} }))), - nil, - }, { - "set valid actor.status.worker_assignment, IPv6", - validInput(withStatus()), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"1234::5678"} }))), - nil, - }, { - "set valid actor.status.worker_assignment, dual-stack", - validInput(withStatus()), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { + name: "invalid actor.metadata.name", + oldVal: validInput(), + newVal: validOutput(withMetadata(func(m *ateapipb.ResourceMetadata) { m.Name = "invalid value" })), + want: field.ErrorList{field.Invalid(field.NewPath("metadata", "name"), nil, "").WithOrigin("immutable")}, + }, { + name: "change actor.actor_template is allowed", + oldVal: validInput(withActorTemplate("as1", "nm1")), + newVal: validOutput(withActorTemplate("as2", "nm2")), + }, { + name: "clear actor.actor_template", + oldVal: validInput(withActorTemplate("as", "nm")), + newVal: validOutput(func(a *ateapipb.Actor) { a.ActorTemplate = nil }), + want: field.ErrorList{field.Required(field.NewPath("actor_template"), "")}, + }, { + name: "add actor.source_tag", + oldVal: validInput(), + newVal: validOutput(withSourceTag("as", "nm")), + want: field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, + }, { + name: "clear actor.source_tag", + oldVal: validInput(withSourceTag("as", "nm")), + newVal: validOutput(func(a *ateapipb.Actor) { a.SourceTag = nil }), + want: field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, + }, { + name: "change actor.source_tag", + oldVal: validInput(withSourceTag("as1", "nm1")), + newVal: validOutput(withSourceTag("as2", "nm2")), + want: field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, + }, { + name: "set valid worker_selector", + oldVal: validInput(), + newVal: validOutput(withWorkerSelector(map[string]string{"tier": "1"})), + }, { + name: "clear worker_selector", + oldVal: validInput(withWorkerSelector(map[string]string{"tier": "1"})), + newVal: validOutput(), + }, { + name: "modify worker_selector", + oldVal: validInput(withWorkerSelector(map[string]string{"tier": "1"})), + newVal: validOutput(withWorkerSelector(map[string]string{"tier": "2"})), + }, { + name: "invalid worker_selector with nil match_labels", + oldVal: validInput(), + newVal: validOutput(func(a *ateapipb.Actor) { a.WorkerSelector = &ateapipb.Selector{} }), + want: field.ErrorList{field.Invalid(field.NewPath("worker_selector"), nil, "one of").WithOrigin("union")}, + }, { + name: "invalid worker_selector label key", + oldVal: validInput(), + newVal: validOutput(withWorkerSelector(map[string]string{"bad key": "2"})), + want: field.ErrorList{field.Invalid(field.NewPath("worker_selector", "match_labels"), nil, "").WithOrigin("format=k8s-label-key")}, + }, { + name: "invalid worker_selector label value", + oldVal: validInput(), + newVal: validOutput(withWorkerSelector(map[string]string{"tier": "bad value"})), + want: field.ErrorList{field.Invalid(field.NewPath("worker_selector", "match_labels").Key("tier"), nil, "").WithOrigin("format=k8s-label-value")}, + }, { + name: "too many worker_selector.match_labels", + oldVal: validInput(), + newVal: validOutput(withWorkerSelector(selectorLabelsOfSize(11))), + want: field.ErrorList{field.TooMany(field.NewPath("worker_selector", "match_labels"), 11, 10).WithOrigin("maxProperties")}, + }, { + name: "add actor.source_tag", + oldVal: validInput(), + newVal: validOutput(withSourceTag("as", "nm")), + want: field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, + }, { + name: "clear actor.source_tag", + oldVal: validInput(withSourceTag("as", "nm")), + newVal: validOutput(func(a *ateapipb.Actor) { a.SourceTag = nil }), + want: field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, + }, { + name: "change actor.source_tag", + oldVal: validInput(withSourceTag("as1", "nm1")), + newVal: validOutput(withSourceTag("as2", "nm2")), + want: field.ErrorList{field.Invalid(field.NewPath("source_tag"), nil, "").WithOrigin("immutable")}, + }, { + name: "unspecified actor.status", + oldVal: validInput(withStatus()), + newVal: validOutput(func(a *ateapipb.Actor) { a.Status = nil }), + want: field.ErrorList{field.Required(field.NewPath("status"), "")}, + }, { + name: "unspecified actor.status.state", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = 0 })), + want: field.ErrorList{field.Required(field.NewPath("status", "state"), "")}, + }, { + name: "change actor.status.state", + oldVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = ateapipb.ActorState_ACTOR_STATE_RUNNING })), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = ateapipb.ActorState_ACTOR_STATE_CRASHED })), + }, { + name: "negative actor.status.state", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = -1 })), + want: field.ErrorList{field.Invalid(field.NewPath("status", "state"), nil, "").WithOrigin("minimum")}, + }, { + name: "just out of bounds actor.status.state", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = 10 })), + want: field.ErrorList{field.Invalid(field.NewPath("status", "state"), nil, "").WithOrigin("maximum")}, + }, { + name: "invalid actor.status.state", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.State = 1234567890 })), + want: field.ErrorList{field.Invalid(field.NewPath("status", "state"), nil, "").WithOrigin("maximum")}, + }, { + name: "set valid actor.status.worker_assignment, IPv4", + oldVal: validInput(withStatus()), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"1.2.3.4"} }))), + }, { + name: "set valid actor.status.worker_assignment, IPv6", + oldVal: validInput(withStatus()), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"1234::5678"} }))), + }, { + name: "set valid actor.status.worker_assignment, dual-stack", + oldVal: validInput(withStatus()), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"1234::5678", "1.2.3.4"} }))), - nil, }, { - "invalid actor.status.worker_assignment.worker_pod_ips: two IPv6", - validInput(withStatus()), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { + name: "invalid actor.status.worker_assignment.worker_pod_ips: two IPv6", + oldVal: validInput(withStatus()), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"1234::5678", "1234::9"} }))), - field.ErrorList{ + want: field.ErrorList{ field.Invalid(field.NewPath("status", "worker_assignment", "worker_pod_ips").Index(1), nil, ""), }, }, { - "invalid actor.status.worker_assignment.worker_pod_ips: too many", - validInput(withStatus()), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { + name: "invalid actor.status.worker_assignment.worker_pod_ips: too many", + oldVal: validInput(withStatus()), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"1.2.3.4", "1234::5678", "1.2.3.5"} }))), - field.ErrorList{ + want: field.ErrorList{ field.TooMany(field.NewPath("status", "worker_assignment", "worker_pod_ips"), 3, 2).WithOrigin("maxItems"), }, }, { - "clear actor.status.worker_assignment", - validInput(withStatus(withWorkerAssignment())), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.WorkerAssignment = nil })), - nil, + name: "clear actor.status.worker_assignment", + oldVal: validInput(withStatus(withWorkerAssignment())), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.WorkerAssignment = nil })), }, { - "modify actor.status.worker_assignment", - validInput(withStatus(withWorkerAssignment())), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPod = "pod2" }))), - field.ErrorList{field.Invalid(field.NewPath("status", "worker_assignment"), nil, "").WithOrigin("update")}, + name: "modify actor.status.worker_assignment", + oldVal: validInput(withStatus(withWorkerAssignment())), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPod = "pod2" }))), + want: field.ErrorList{field.Invalid(field.NewPath("status", "worker_assignment"), nil, "").WithOrigin("update")}, }, { - "empty actor.status.worker_assignment", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.WorkerAssignment = &ateapipb.WorkerAssignment{} })), - field.ErrorList{ + name: "empty actor.status.worker_assignment", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.WorkerAssignment = &ateapipb.WorkerAssignment{} })), + want: field.ErrorList{ field.Required(field.NewPath("status", "worker_assignment", "worker"), ""), field.Required(field.NewPath("status", "worker_assignment", "worker_namespace"), ""), field.Required(field.NewPath("status", "worker_assignment", "worker_pool"), ""), @@ -383,9 +377,9 @@ func TestValidateActorUpdate(t *testing.T) { field.Required(field.NewPath("status", "worker_assignment", "node_name"), ""), }, }, { - "invalid actor.status.worker_assignment", - validInput(), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { + name: "invalid actor.status.worker_assignment", + oldVal: validInput(), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.Worker = &ateapipb.ObjectRef{Atespace: "not-allowed", Name: "bad value"} wa.WorkerNamespace = "invalid namespace" wa.WorkerPool = "invalid pool" @@ -394,7 +388,7 @@ func TestValidateActorUpdate(t *testing.T) { wa.WorkerPodIps = []string{"invalid IP"} wa.NodeName = "invalid node" }))), - field.ErrorList{ + want: field.ErrorList{ field.Forbidden(field.NewPath("status", "worker_assignment", "worker", "atespace"), ""), field.Invalid(field.NewPath("status", "worker_assignment", "worker", "name"), nil, "").WithOrigin("format=k8s-short-name"), field.Invalid(field.NewPath("status", "worker_assignment", "worker_namespace"), nil, "").WithOrigin("format=k8s-short-name"), @@ -406,242 +400,229 @@ func TestValidateActorUpdate(t *testing.T) { }, }, { // because we have manual IP format validation, let's be sure - "invalid actor.status.worker_assignment.worker_pod_ips: leading 0s", - validInput(), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"001.002.003.004"} }))), - field.ErrorList{ + name: "invalid actor.status.worker_assignment.worker_pod_ips: leading 0s", + oldVal: validInput(), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"001.002.003.004"} }))), + want: field.ErrorList{ field.Invalid(field.NewPath("status", "worker_assignment", "worker_pod_ips").Index(0), nil, "").WithOrigin("format=ip-strict"), }, }, { // because we have manual IP format validation, let's be sure - "invalid actor.status.worker_assignment.worker_pod_ips: non-canonical", - validInput(), - validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"0012::0034"} }))), - field.ErrorList{ + name: "invalid actor.status.worker_assignment.worker_pod_ips: non-canonical", + oldVal: validInput(), + newVal: validOutput(withStatus(withWorkerAssignment(func(wa *ateapipb.WorkerAssignment) { wa.WorkerPodIps = []string{"0012::0034"} }))), + want: field.ErrorList{ field.Invalid(field.NewPath("status", "worker_assignment", "worker_pod_ips").Index(0), nil, "").WithOrigin("format=ip-strict"), }, }, { - "valid actor.status.assigned_node", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "valid actor.status.assigned_node", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.AssignedNode = "node-1" })), - nil, }, { - "invalid actor.status.assigned_node", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "invalid actor.status.assigned_node", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.AssignedNode = "NOT A NODE" })), - field.ErrorList{field.Invalid(field.NewPath("status", "assigned_node"), nil, "").WithOrigin("format=k8s-long-name")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "assigned_node"), nil, "").WithOrigin("format=k8s-long-name")}, }, { - "valid actor.status.last_assigned_generation", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "valid actor.status.last_assigned_generation", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 1 })), - nil, }, { - "increasing actor.status.last_assigned_generation", - validInput(withStatus(func(s *ateapipb.ActorStatus) { + name: "increasing actor.status.last_assigned_generation", + oldVal: validInput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 2 })), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 3 })), - nil, }, { - "unchanged actor.status.last_assigned_generation", - validInput(withStatus(func(s *ateapipb.ActorStatus) { + name: "unchanged actor.status.last_assigned_generation", + oldVal: validInput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 3 })), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 3 })), - nil, }, { - "decreasing actor.status.last_assigned_generation", - validInput(withStatus(func(s *ateapipb.ActorStatus) { + name: "decreasing actor.status.last_assigned_generation", + oldVal: validInput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 3 })), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 1 })), - field.ErrorList{field.Invalid(field.NewPath("status", "last_assigned_generation"), nil, "").WithOrigin("monotonic")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "last_assigned_generation"), nil, "").WithOrigin("monotonic")}, }, { - "unsetting actor.status.last_assigned_generation", - validInput(withStatus(func(s *ateapipb.ActorStatus) { + name: "unsetting actor.status.last_assigned_generation", + oldVal: validInput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 3 })), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 0 })), - field.ErrorList{field.Invalid(field.NewPath("status", "last_assigned_generation"), nil, "").WithOrigin("update")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "last_assigned_generation"), nil, "").WithOrigin("update")}, }, { - "negative actor.status.last_assigned_generation", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "negative actor.status.last_assigned_generation", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = -1 })), - field.ErrorList{field.Invalid(field.NewPath("status", "last_assigned_generation"), nil, "").WithOrigin("minimum")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "last_assigned_generation"), nil, "").WithOrigin("minimum")}, }, { - "valid actor.status.snapshots with durable object snapshot", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "valid actor.status.snapshots with durable object snapshot", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot()} })), - nil, }, { - "valid actor.status.snapshots actor_template_uid", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "valid actor.status.snapshots actor_template_uid", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.ActorTemplateUid = "01234567-89ab-cdef-0123-456789abcdef" })} })), - nil, }, { // Each suspend restamps the UID of the template the snapshot was captured under. - "changing actor.status.snapshots.actor_template_uid is allowed", - validInput(withStatus(func(s *ateapipb.ActorStatus) { + name: "changing actor.status.snapshots.actor_template_uid is allowed", + oldVal: validInput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.ActorTemplateUid = "01234567-89ab-cdef-0123-456789abcdef" })} })), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.ActorTemplateUid = "fedcba98-7654-3210-fedc-ba9876543210" })} })), - nil, }, { - "invalid actor.status.snapshots.actor_template_uid", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "invalid actor.status.snapshots.actor_template_uid", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.ActorTemplateUid = "not-a-uuid" })} })), - field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("actor_template_uid"), nil, "").WithOrigin("format=k8s-uuid")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("actor_template_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { - "valid actor.status.snapshots uuid and locality", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "valid actor.status.snapshots uuid and locality", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.Uuid = "snap-1" snap.Locality = "node-1" })} })), - nil, }, { - "missing actor.status.snapshots.uuid", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "missing actor.status.snapshots.uuid", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.Uuid = "" })} })), - field.ErrorList{field.Required(field.NewPath("status", "snapshots").Index(0).Child("uuid"), "")}, + want: field.ErrorList{field.Required(field.NewPath("status", "snapshots").Index(0).Child("uuid"), "")}, }, { - "invalid actor.status.snapshots.uuid", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "invalid actor.status.snapshots.uuid", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.Uuid = "SNAP 1" })} })), - field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("uuid"), nil, "").WithOrigin("format=k8s-short-name")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("uuid"), nil, "").WithOrigin("format=k8s-short-name")}, }, { - "invalid actor.status.snapshots.locality", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "invalid actor.status.snapshots.locality", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.Locality = "NODE 1" })} })), - field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("locality"), nil, "").WithOrigin("format=k8s-long-name")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("locality"), nil, "").WithOrigin("format=k8s-long-name")}, }, { - "duplicate actor.status.snapshots generation", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "duplicate actor.status.snapshots generation", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{ validSnapshot(func(snap *ateapipb.Snapshot) { snap.Generation = 1 }), validSnapshot(func(snap *ateapipb.Snapshot) { snap.Generation = 1 }), } })), - field.ErrorList{field.Duplicate(field.NewPath("status", "snapshots").Index(1), nil)}, + want: field.ErrorList{field.Duplicate(field.NewPath("status", "snapshots").Index(1), nil)}, }, { - "valid actor.status.snapshots.fidelity", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "valid actor.status.snapshots.fidelity", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.Fidelity = ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES })} })), - nil, }, { - "missing actor.status.snapshots.fidelity", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "missing actor.status.snapshots.fidelity", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.Fidelity = ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED })} })), - field.ErrorList{field.Required(field.NewPath("status", "snapshots").Index(0).Child("fidelity"), "")}, + want: field.ErrorList{field.Required(field.NewPath("status", "snapshots").Index(0).Child("fidelity"), "")}, }, { - "negative actor.status.snapshots.fidelity", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "negative actor.status.snapshots.fidelity", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.Fidelity = ateapipb.SnapshotFidelity(-1) })} })), - field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("fidelity"), nil, "").WithOrigin("minimum")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("fidelity"), nil, "").WithOrigin("minimum")}, }, { - "invalid actor.status.snapshots.fidelity", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "invalid actor.status.snapshots.fidelity", + oldVal: validInput(), + newVal: validOutput(withStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{validSnapshot(func(snap *ateapipb.Snapshot) { snap.Fidelity = ateapipb.SnapshotFidelity(4) })} })), - field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("fidelity"), nil, "").WithOrigin("maximum")}, + want: field.ErrorList{field.Invalid(field.NewPath("status", "snapshots").Index(0).Child("fidelity"), nil, "").WithOrigin("maximum")}, }, { - "too many external_volumes", - validInput(), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { - vols := make([]*ateapipb.ExternalVolume, 33) - for i := range vols { - vols[i] = &ateapipb.ExternalVolume{Name: fmt.Sprintf("vol-%d", i), VolumeType: "substrate.io/mock"} - } - s.ExternalVolumes = vols - })), - field.ErrorList{field.TooMany(field.NewPath("status", "external_volumes"), 33, 32).WithOrigin("maxItems")}, + // Status is written by the server, so it is not preview-gated: the + // volume cases below expect the same result with and without preview. + name: "no volumes", + oldVal: validActor(withActorStatus()), + newVal: validActor(withActorStatus()), + want: nil, + wantWithPreview: map[string]field.ErrorList{"ALL": nil}, }, { // Set-once fields permit the nil->set transition, so a volume added // in an update validates like one added at creation. - "adding a volume on update is allowed", - validInput(withStatus()), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + name: "adding a volume", + oldVal: validActor(withActorStatus()), + newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) { s.ExternalVolumes = []*ateapipb.ExternalVolume{{Name: "vol-a", VolumeType: "substrate.io/mock"}} })), - nil, + want: nil, + wantWithPreview: map[string]field.ErrorList{"ALL": nil}, }, { - "duplicate external_volumes name", - validInput(withStatus()), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { - s.ExternalVolumes = []*ateapipb.ExternalVolume{ - {Name: "vol-a", VolumeType: "substrate.io/mock"}, - {Name: "vol-a", VolumeType: "substrate.io/mock"}, - } + name: "adding a volume to an actor with no status", + oldVal: validActor(), + newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) { + s.ExternalVolumes = []*ateapipb.ExternalVolume{{Name: "vol-a", VolumeType: "substrate.io/mock"}} })), - field.ErrorList{field.Duplicate(field.NewPath("status", "external_volumes").Index(1), nil)}, + want: nil, + wantWithPreview: map[string]field.ErrorList{"ALL": nil}, }, { - "provisioning transition on an existing volume is valid", - validInput(withStatus(func(s *ateapipb.ActorStatus) { + name: "provisioning transition on an existing volume", + oldVal: validInput(withStatus(func(s *ateapipb.ActorStatus) { s.ExternalVolumes = []*ateapipb.ExternalVolume{{Name: "vol-a", VolumeType: "substrate.io/mock"}} })), - validOutput(withStatus(func(s *ateapipb.ActorStatus) { + newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) { s.ExternalVolumes = []*ateapipb.ExternalVolume{{ Name: "vol-a", VolumeType: "substrate.io/mock", @@ -649,11 +630,41 @@ func TestValidateActorUpdate(t *testing.T) { VolumeContext: map[string]string{"attachment": "iqn.2026-08.io.ate:vol-a"}, }} })), - nil, + want: nil, + wantWithPreview: map[string]field.ErrorList{"ALL": nil}, + }, { + name: "duplicate external_volumes volume_name", + oldVal: validActor(withActorStatus()), + newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) { + s.ExternalVolumes = []*ateapipb.ExternalVolume{ + {Name: "vol-a", VolumeType: "substrate.io/mock"}, + {Name: "vol-a", VolumeType: "substrate.io/mock"}, + } + })), + want: field.ErrorList{field.Duplicate(field.NewPath("status", "external_volumes").Index(1), nil)}, + wantWithPreview: map[string]field.ErrorList{ + "ALL": {field.Duplicate(field.NewPath("status", "external_volumes").Index(1), nil)}, + }, + }, { + name: "too many external_volumes", + oldVal: validActor(withActorStatus()), + newVal: validActor(withActorStatus(func(s *ateapipb.ActorStatus) { + vols := make([]*ateapipb.ExternalVolume, 33) + for i := range vols { + vols[i] = &ateapipb.ExternalVolume{Name: fmt.Sprintf("vol-%d", i), VolumeType: "substrate.io/mock"} + } + s.ExternalVolumes = vols + })), + want: field.ErrorList{field.TooMany(field.NewPath("status", "external_volumes"), 33, 32).WithOrigin("maxItems")}, + wantWithPreview: map[string]field.ErrorList{ + "ALL": {field.TooMany(field.NewPath("status", "external_volumes"), 33, 32).WithOrigin("maxItems")}, + }, }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - assertValidateErr(t, ValidateActorUpdate(context.Background(), nil, tt.newVal, tt.oldVal, true), tt.want) + assertValidate(t, func() field.ErrorList { + return ValidateActorUpdate(context.Background(), nil, tt.newVal, tt.oldVal, true) + }, tt.want, tt.wantWithPreview) }) } } diff --git a/cmd/ateapi/internal/apivalidation/atespace.go b/cmd/ateapi/internal/apivalidation/atespace.go index baddcfa634..4bc4c3060b 100644 --- a/cmd/ateapi/internal/apivalidation/atespace.go +++ b/cmd/ateapi/internal/apivalidation/atespace.go @@ -18,30 +18,29 @@ import ( "context" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" - "k8s.io/apimachinery/pkg/api/operation" "k8s.io/apimachinery/pkg/util/validation/field" ) func ValidateCreateAtespaceRequest(ctx context.Context, req *ateapipb.CreateAtespaceRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_CreateAtespaceRequest(ctx, op, nil, req, nil) } func ValidateGetAtespaceRequest(ctx context.Context, req *ateapipb.GetAtespaceRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_GetAtespaceRequest(ctx, op, nil, req, nil) } func ValidateListAtespacesRequest(ctx context.Context, req *ateapipb.ListAtespacesRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_ListAtespacesRequest(ctx, op, nil, req, nil) } func ValidateDeleteAtespaceRequest(ctx context.Context, req *ateapipb.DeleteAtespaceRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_DeleteAtespaceRequest(ctx, op, nil, req, nil) } diff --git a/cmd/ateapi/internal/apivalidation/common.go b/cmd/ateapi/internal/apivalidation/common.go index d99c419428..73c8dcd959 100644 --- a/cmd/ateapi/internal/apivalidation/common.go +++ b/cmd/ateapi/internal/apivalidation/common.go @@ -19,6 +19,7 @@ import ( "fmt" "net/netip" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "k8s.io/apimachinery/pkg/api/operation" @@ -41,6 +42,29 @@ func validateSingletonName(fldPath *field.Path, name string) field.ErrorList { return field.ErrorList{field.Invalid(fldPath, name, fmt.Sprintf("must be %q", resources.SingletonName)).WithOrigin("custom=default")} } +// MakeCreateOp returns an operation.Operation for a create operation. If ctx +// has opted-out of preview features, the returned operation will have all +// preview gates disabled (see preview.Header). +func MakeCreateOp(ctx context.Context) operation.Operation { + return operation.Operation{Type: operation.Create, Options: optionsFor(ctx)} +} + +// MakeUpdateOp returns an operation.Operation for an update operation. If ctx +// has opted-out of preview features, the returned operation will have all +// preview gates disabled (see preview.Header). +func MakeUpdateOp(ctx context.Context) operation.Operation { + return operation.Operation{Type: operation.Update, Options: optionsFor(ctx)} +} + +// optionsFor returns the preview gates enabled for ctx. +// The returned map MUST NOT be modified. +func optionsFor(ctx context.Context) map[string]bool { + if preview.IsOptedOut(ctx) { + return preview.DisabledMap() + } + return preview.AsMap() +} + // ValidateCustom_ResourceMetadata checks the server-stamped timestamps: each, // when set, must be a valid google.protobuf.Timestamp, and update_time must // not precede create_time. Both fields are scrubbed from input, so a diff --git a/cmd/ateapi/internal/apivalidation/common_test.go b/cmd/ateapi/internal/apivalidation/common_test.go index 5df329e104..150340911a 100644 --- a/cmd/ateapi/internal/apivalidation/common_test.go +++ b/cmd/ateapi/internal/apivalidation/common_test.go @@ -16,8 +16,12 @@ package apivalidation import ( "fmt" + "maps" + "slices" + "strings" "testing" + "github.com/agent-substrate/substrate/internal/preview" "k8s.io/apimachinery/pkg/util/validation/field" ) @@ -32,6 +36,34 @@ const ( someActorUID = "6b1f9d0c-4a2e-4d38-9c77-5e0a1b2c3d4e" ) +// assertValidate checks validate against want with no preview gates enabled. +// Then, for each entry in wantWithPreview, it enables the gates the key names +// and checks validate against that entry's errors. Only cases that set fields +// behind a preview gate need wantWithPreview. +func assertValidate(t *testing.T, validate func() field.ErrorList, want field.ErrorList, wantWithPreview map[string]field.ErrorList) { + t.Helper() + setPreviewForTest(t, "") + assertValidateErr(t, validate(), want) + for _, gates := range slices.Sorted(maps.Keys(wantWithPreview)) { + t.Run("preview="+gates, func(t *testing.T) { + t.Helper() + setPreviewForTest(t, gates) + assertValidateErr(t, validate(), wantWithPreview[gates]) + }) + } +} + +// setPreviewForTest enables the preview gates named by gates, a --preview +// value such as "ALL", for the duration of t. An empty value enables none. +func setPreviewForTest(t *testing.T, gates string) { + t.Helper() + var values []string + if gates != "" { + values = strings.Split(gates, ",") + } + preview.InitForTest(t, values...) +} + func selectorLabelsOfSize(n int) map[string]string { labels := make(map[string]string, n) for i := 0; i < n; i++ { diff --git a/cmd/ateapi/internal/apivalidation/egress_policy.go b/cmd/ateapi/internal/apivalidation/egress_policy.go index c22f0a7c9e..c890c45179 100644 --- a/cmd/ateapi/internal/apivalidation/egress_policy.go +++ b/cmd/ateapi/internal/apivalidation/egress_policy.go @@ -28,23 +28,28 @@ import ( ) func ValidateCreateActorEgressPolicyRequest(ctx context.Context, req *ateapipb.CreateActorEgressPolicyRequest) field.ErrorList { - return Validate_CreateActorEgressPolicyRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) + op := MakeCreateOp(ctx) + return Validate_CreateActorEgressPolicyRequest(ctx, op, nil, req, nil) } func ValidateGetActorEgressPolicyRequest(ctx context.Context, req *ateapipb.GetActorEgressPolicyRequest) field.ErrorList { - return Validate_GetActorEgressPolicyRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) + op := MakeCreateOp(ctx) + return Validate_GetActorEgressPolicyRequest(ctx, op, nil, req, nil) } func ValidateUpdateActorEgressPolicyRequest(ctx context.Context, req *ateapipb.UpdateActorEgressPolicyRequest) field.ErrorList { - return Validate_UpdateActorEgressPolicyRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) + op := MakeCreateOp(ctx) + return Validate_UpdateActorEgressPolicyRequest(ctx, op, nil, req, nil) } func ValidateDeleteActorEgressPolicyRequest(ctx context.Context, req *ateapipb.DeleteActorEgressPolicyRequest) field.ErrorList { - return Validate_DeleteActorEgressPolicyRequest(ctx, operation.Operation{Type: operation.Create}, nil, req, nil) + op := MakeCreateOp(ctx) + return Validate_DeleteActorEgressPolicyRequest(ctx, op, nil, req, nil) } func ValidateEgressPolicyUpdate(ctx context.Context, p *field.Path, newVal, oldVal *ateapipb.EgressPolicy) field.ErrorList { - return Validate_EgressPolicy(ctx, operation.Operation{Type: operation.Update}, p, newVal, oldVal) + op := MakeUpdateOp(ctx) + return Validate_EgressPolicy(ctx, op, p, newVal, oldVal) } func ValidateCustom_CreateActorEgressPolicyRequest(_ context.Context, _ operation.Operation, p *field.Path, req, _ *ateapipb.CreateActorEgressPolicyRequest) field.ErrorList { diff --git a/cmd/ateapi/internal/apivalidation/tag.go b/cmd/ateapi/internal/apivalidation/tag.go index d109408f2b..add2300c6f 100644 --- a/cmd/ateapi/internal/apivalidation/tag.go +++ b/cmd/ateapi/internal/apivalidation/tag.go @@ -24,17 +24,17 @@ import ( ) func ValidateCreateTagRequest(ctx context.Context, req *ateapipb.CreateTagRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_CreateTagRequest(ctx, op, nil, req, nil) } func ValidateGetTagRequest(ctx context.Context, req *ateapipb.GetTagRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_GetTagRequest(ctx, op, nil, req, nil) } func ValidateListTagsRequest(ctx context.Context, req *ateapipb.ListTagsRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_ListTagsRequest(ctx, op, nil, req, nil) } @@ -43,17 +43,17 @@ func ValidateUpdateTagRequest(ctx context.Context, req *ateapipb.UpdateTagReques // the existence of a "current" value, which we do not have yet. This is // validating the request itself. The result will be validated later, after // we have a current value to compare against. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_UpdateTagRequest(ctx, op, nil, req, nil) } func ValidateDeleteTagRequest(ctx context.Context, req *ateapipb.DeleteTagRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_DeleteTagRequest(ctx, op, nil, req, nil) } func ValidateTagUpdate(ctx context.Context, fldPath *field.Path, newVal, oldVal *ateapipb.Tag) field.ErrorList { - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(ctx) return Validate_Tag(ctx, op, fldPath, newVal, oldVal) } diff --git a/cmd/ateapi/internal/apivalidation/validation_test.go b/cmd/ateapi/internal/apivalidation/validation_test.go index 1803f9e1f2..120335bb21 100644 --- a/cmd/ateapi/internal/apivalidation/validation_test.go +++ b/cmd/ateapi/internal/apivalidation/validation_test.go @@ -116,7 +116,7 @@ func TestValidateResourceMetadataCreate(t *testing.T) { }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) matcher := field.ErrorMatcher{}.ByType().ByField().ByOrigin() matcher.Test(t, tt.want, Validate_ResourceMetadata(context.Background(), op, nil, tt.obj, nil)) }) @@ -232,7 +232,7 @@ func TestValidateResourceMetadataUpdate(t *testing.T) { }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(context.Background()) matcher := field.ErrorMatcher{}.ByType().ByField().ByOrigin() matcher.Test(t, tt.want, Validate_ResourceMetadata(context.Background(), op, nil, tt.newObj, tt.oldObj)) }) @@ -431,7 +431,7 @@ func TestValidateResourceMetadataNameAndAtespaceFormat(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { obj := proto.CloneOf(tt.obj) // avoid internal mutations - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) matcher := field.ErrorMatcher{}.ByType().ByField().ByOrigin() matcher.Test(t, tt.want, Validate_ResourceMetadata(context.Background(), op, nil, obj, nil)) }) @@ -645,7 +645,7 @@ func TestValidateObjectRef(t *testing.T) { }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) matcher := field.ErrorMatcher{}.ByType().ByField().ByOrigin() matcher.Test(t, tt.want, Validate_ObjectRef(context.Background(), op, nil, tt.ref, nil)) }) @@ -800,7 +800,7 @@ func TestValidateSystemInfoVolumeSource(t *testing.T) { }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) matcher := field.ErrorMatcher{}.ByType().ByField().ByOrigin() matcher.Test(t, tt.want, Validate_SystemInfoVolumeSource(context.Background(), op, nil, tt.obj, nil)) }) @@ -864,7 +864,7 @@ func TestValidateTrustBundleDataSource(t *testing.T) { }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) matcher := field.ErrorMatcher{}.ByType().ByField().ByOrigin() matcher.Test(t, tt.want, Validate_TrustBundleDataSource(context.Background(), op, nil, tt.obj, nil)) }) @@ -959,7 +959,7 @@ func TestValidateExternalVolume(t *testing.T) { }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) assertValidateErr(t, Validate_ExternalVolume(context.Background(), op, nil, tt.obj, nil), tt.want) }) } @@ -1008,7 +1008,7 @@ func TestValidateExternalVolume_Update(t *testing.T) { }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(context.Background()) assertValidateErr(t, Validate_ExternalVolume(context.Background(), op, nil, tt.newObj, tt.oldObj), tt.want) }) } @@ -1049,7 +1049,7 @@ func TestValidateDeleteOptions(t *testing.T) { }} for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) matcher := field.ErrorMatcher{}.ByType().ByField().ByOrigin() matcher.Test(t, tt.want, Validate_DeleteOptions(context.Background(), op, nil, tt.obj, nil)) }) @@ -1245,7 +1245,7 @@ func TestValidateSnapshot(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) assertValidateErr(t, Validate_Snapshot(context.Background(), op, nil, tt.obj, nil), tt.want) }) } @@ -1332,7 +1332,7 @@ func TestValidateSnapshotUpdate(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(context.Background()) assertValidateErr(t, Validate_Snapshot(context.Background(), op, nil, tt.newObj, tt.oldObj), tt.want) }) } @@ -1354,7 +1354,7 @@ func TestValidateNestedSnapshot(t *testing.T) { validate: func(ctx context.Context) field.ErrorList { // The live path: the server validates the Actor it is about to // write, as an update against the stored one. - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(context.Background()) oldVal := validActor(withActorStatus()) newVal := validActor(withActorStatus(func(s *ateapipb.ActorStatus) { s.Snapshots = []*ateapipb.Snapshot{badSnapshot()} @@ -1366,7 +1366,7 @@ func TestValidateNestedSnapshot(t *testing.T) { name: "tag.status.snapshot", path: field.NewPath("status", "snapshot"), validate: func(ctx context.Context) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) obj := validTag(func(tag *ateapipb.Tag) { tag.Status.Snapshot = badSnapshot() }) @@ -1494,7 +1494,7 @@ func TestValidateTag(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) assertValidateErr(t, Validate_Tag(context.Background(), op, nil, tt.obj, nil), tt.want) }) } @@ -1538,7 +1538,7 @@ func TestValidateTagUpdate(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(context.Background()) assertValidateErr(t, Validate_Tag(context.Background(), op, nil, tt.newObj, tt.oldObj), tt.want) }) } @@ -1704,7 +1704,7 @@ func TestValidateTagRequestPayloads(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) assertValidateErr(t, tt.validate(context.Background(), op), tt.want) }) } @@ -1768,7 +1768,7 @@ func TestValidateGoldenSnapshotStatus(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(context.Background()) assertValidateErr(t, Validate_GoldenSnapshotStatus(context.Background(), op, nil, tt.obj, nil), tt.want) }) } @@ -1820,7 +1820,7 @@ func TestValidateGoldenSnapshotStatusUpdate(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(context.Background()) assertValidateErr(t, Validate_GoldenSnapshotStatus(context.Background(), op, nil, tt.newObj, tt.oldObj), tt.want) }) } diff --git a/cmd/ateapi/internal/apivalidation/worker.go b/cmd/ateapi/internal/apivalidation/worker.go index cfb0ea9562..c56056a361 100644 --- a/cmd/ateapi/internal/apivalidation/worker.go +++ b/cmd/ateapi/internal/apivalidation/worker.go @@ -26,25 +26,25 @@ import ( ) func ValidateListWorkerActorAssignmentsRequest(ctx context.Context, req *ateapipb.ListWorkerActorAssignmentsRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_ListWorkerActorAssignmentsRequest(ctx, op, nil, req, nil) } func ValidateListWorkersRequest(ctx context.Context, req *ateapipb.ListWorkersRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_ListWorkersRequest(ctx, op, nil, req, nil) } func ValidateGetWorkerRequest(ctx context.Context, req *ateapipb.GetWorkerRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_GetWorkerRequest(ctx, op, nil, req, nil) } func ValidateCreateWorkerRequest(ctx context.Context, req *ateapipb.CreateWorkerRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_CreateWorkerRequest(ctx, op, nil, req, nil) } @@ -54,7 +54,7 @@ func ValidateUpdateWorkerRequest(ctx context.Context, req *ateapipb.UpdateWorker // the existence of a "current" value, which we do not have yet. This is // validating the request itself. The result will be validated later, after // we have a current value to compare against. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_UpdateWorkerRequest(ctx, op, nil, req, nil) } @@ -62,23 +62,23 @@ func ValidateDeleteWorkerRequest(ctx context.Context, req *ateapipb.DeleteWorker // Call the generated validation. The preconditions in options are each // optional: a zero value waives that guard, so only non-zero values are // checked for shape. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_DeleteWorkerRequest(ctx, op, nil, req, nil) } func ValidateDrainWorkerRequest(ctx context.Context, req *ateapipb.DrainWorkerRequest) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_DrainWorkerRequest(ctx, op, nil, req, nil) } func ValidateRegisterWorkerRequest(ctx context.Context, req *ateapipb.RegisterWorkerRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_RegisterWorkerRequest(ctx, op, nil, req, nil) } func ValidateRequestActorSuspendRequest(ctx context.Context, req *ateapipb.RequestActorSuspendRequest) field.ErrorList { - op := operation.Operation{Type: operation.Create} + op := MakeCreateOp(ctx) return Validate_RequestActorSuspendRequest(ctx, op, nil, req, nil) } @@ -104,7 +104,7 @@ func ValidateReportActorSnapshotStateRequest(ctx context.Context, req *ateapipb. // against. func ValidateWorkerUpdate(ctx context.Context, fldPath *field.Path, newVal, oldVal *ateapipb.Worker, requireStatus bool) field.ErrorList { // Call the generated validation. - op := operation.Operation{Type: operation.Update} + op := MakeUpdateOp(ctx) errs := Validate_Worker(ctx, op, fldPath, newVal, oldVal) if requireStatus { // Status is optional in the schema, but is actually required to be set diff --git a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go index c05764ba1c..d6e8ce53b7 100644 --- a/cmd/ateapi/internal/apivalidation/zz_generated.validation.go +++ b/cmd/ateapi/internal/apivalidation/zz_generated.validation.go @@ -1282,12 +1282,17 @@ func Validate_ActorTemplate( } // call field-attached validations earlyReturn := false - if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.RequiredPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) earlyReturn = true } if earlyReturn { return // do not proceed } + // custom validation + if e := ValidateCustom_ActorTemplate_Resources(ctx, op, fldPath, obj, oldObj); len(e) != 0 { + errs = append(errs, e...) + } // call the type's validation function errs = append(errs, Validate_Resources(ctx, op, fldPath, obj, oldObj)...) return @@ -5925,13 +5930,13 @@ func Validate_MintActorJwtRequest( return errs } -// Validate_MintAteomActorCertificateRequest validates an instance of MintAteomActorCertificateRequest according +// Validate_MintWorkerActorCertificateRequest validates an instance of MintWorkerActorCertificateRequest according // to declarative validation rules in the API schema. -func Validate_MintAteomActorCertificateRequest( +func Validate_MintWorkerActorCertificateRequest( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.MintAteomActorCertificateRequest) (errs field.ErrorList) { + obj, oldObj *ateapipb.MintWorkerActorCertificateRequest) (errs field.ErrorList) { - { // field ateapipb.MintAteomActorCertificateRequest.Actor + { // field ateapipb.MintWorkerActorCertificateRequest.Actor fn := func( fldPath *field.Path, obj, oldObj *ateapipb.ObjectRef, @@ -5971,13 +5976,13 @@ func Validate_MintAteomActorCertificateRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintAteomActorCertificateRequest) *ateapipb.ObjectRef { + func(oldObj *ateapipb.MintWorkerActorCertificateRequest) *ateapipb.ObjectRef { return oldObj.Actor }) errs = append(errs, fn(fldPath.Child("actor"), obj.Actor, oldVal, oldObj != nil)...) } - { // field ateapipb.MintAteomActorCertificateRequest.ActorUid + { // field ateapipb.MintWorkerActorCertificateRequest.ActorUid fn := func( fldPath *field.Path, obj, oldObj *string, @@ -6003,13 +6008,13 @@ func Validate_MintAteomActorCertificateRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintAteomActorCertificateRequest) *string { + func(oldObj *ateapipb.MintWorkerActorCertificateRequest) *string { return &oldObj.ActorUid }) errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) } - { // field ateapipb.MintAteomActorCertificateRequest.CertificateSigningRequest + { // field ateapipb.MintWorkerActorCertificateRequest.CertificateSigningRequest fn := func( fldPath *field.Path, obj, oldObj []byte, @@ -6035,7 +6040,7 @@ func Validate_MintAteomActorCertificateRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintAteomActorCertificateRequest) []byte { + func(oldObj *ateapipb.MintWorkerActorCertificateRequest) []byte { return oldObj.CertificateSigningRequest }) errs = append(errs, fn(fldPath.Child("certificate_signing_request"), obj.CertificateSigningRequest, oldVal, oldObj != nil)...) @@ -6044,13 +6049,13 @@ func Validate_MintAteomActorCertificateRequest( return errs } -// Validate_MintAteomActorJwtRequest validates an instance of MintAteomActorJwtRequest according +// Validate_MintWorkerActorJwtRequest validates an instance of MintWorkerActorJwtRequest according // to declarative validation rules in the API schema. -func Validate_MintAteomActorJwtRequest( +func Validate_MintWorkerActorJwtRequest( ctx context.Context, op operation.Operation, fldPath *field.Path, - obj, oldObj *ateapipb.MintAteomActorJwtRequest) (errs field.ErrorList) { + obj, oldObj *ateapipb.MintWorkerActorJwtRequest) (errs field.ErrorList) { - { // field ateapipb.MintAteomActorJwtRequest.Actor + { // field ateapipb.MintWorkerActorJwtRequest.Actor fn := func( fldPath *field.Path, obj, oldObj *ateapipb.ObjectRef, @@ -6090,13 +6095,13 @@ func Validate_MintAteomActorJwtRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintAteomActorJwtRequest) *ateapipb.ObjectRef { + func(oldObj *ateapipb.MintWorkerActorJwtRequest) *ateapipb.ObjectRef { return oldObj.Actor }) errs = append(errs, fn(fldPath.Child("actor"), obj.Actor, oldVal, oldObj != nil)...) } - { // field ateapipb.MintAteomActorJwtRequest.ActorUid + { // field ateapipb.MintWorkerActorJwtRequest.ActorUid fn := func( fldPath *field.Path, obj, oldObj *string, @@ -6122,13 +6127,13 @@ func Validate_MintAteomActorJwtRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintAteomActorJwtRequest) *string { + func(oldObj *ateapipb.MintWorkerActorJwtRequest) *string { return &oldObj.ActorUid }) errs = append(errs, fn(fldPath.Child("actor_uid"), &obj.ActorUid, oldVal, oldObj != nil)...) } - { // field ateapipb.MintAteomActorJwtRequest.Audiences + { // field ateapipb.MintWorkerActorJwtRequest.Audiences fn := func( fldPath *field.Path, obj, oldObj []string, @@ -6171,7 +6176,7 @@ func Validate_MintAteomActorJwtRequest( return } oldVal := safe.Field(oldObj, - func(oldObj *ateapipb.MintAteomActorJwtRequest) []string { + func(oldObj *ateapipb.MintWorkerActorJwtRequest) []string { return oldObj.Audiences }) errs = append(errs, fn(fldPath.Child("audiences"), obj.Audiences, oldVal, oldObj != nil)...) @@ -9367,6 +9372,13 @@ func Validate_Volume( } // call field-attached validations earlyReturn := false + if e := validate.IfOption(ctx, op, fldPath, obj, oldObj, "ExternalVolumes", false, validate.ForbiddenPointer).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) + earlyReturn = true + } + if e := validate.IfOption(ctx, op, fldPath, obj, oldObj, "ExternalVolumes", false, validate.OptionalPointer).MarkShortCircuit(); len(e) != 0 { + earlyReturn = true + } if e := validate.OptionalPointer(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { earlyReturn = true } diff --git a/cmd/ateapi/internal/authz/interceptor_test.go b/cmd/ateapi/internal/authz/interceptor_test.go index e8c5da651a..f7bf2c4fb6 100644 --- a/cmd/ateapi/internal/authz/interceptor_test.go +++ b/cmd/ateapi/internal/authz/interceptor_test.go @@ -283,8 +283,8 @@ func TestUnaryServerInterceptor_UnregisteredAndHandlerAuthorizedRPCs(t *testing. {"unregistered RPC passes with enforcement off", false, aliceCtx, unregistered, codes.OK}, {"RegisterWorker passes with enforcement on", true, context.Background(), ateapipb.WorkerService_RegisterWorker_FullMethodName, codes.OK}, {"RequestActorSuspend passes with enforcement on", true, context.Background(), ateapipb.WorkerService_RequestActorSuspend_FullMethodName, codes.OK}, - {"MintAteomActorCertificate passes with enforcement on", true, context.Background(), ateapipb.WorkerService_MintAteomActorCertificate_FullMethodName, codes.OK}, - {"MintAteomActorJwt passes with enforcement on", true, context.Background(), ateapipb.WorkerService_MintAteomActorJwt_FullMethodName, codes.OK}, + {"MintWorkerActorCertificate passes with enforcement on", true, context.Background(), ateapipb.WorkerService_MintWorkerActorCertificate_FullMethodName, codes.OK}, + {"MintWorkerActorJwt passes with enforcement on", true, context.Background(), ateapipb.WorkerService_MintWorkerActorJwt_FullMethodName, codes.OK}, {"RegisterWorker passes with enforcement off", false, context.Background(), ateapipb.WorkerService_RegisterWorker_FullMethodName, codes.OK}, } for _, tc := range tests { diff --git a/cmd/ateapi/internal/authz/model.fga b/cmd/ateapi/internal/authz/model.fga index b65bec8163..ccd8242108 100644 --- a/cmd/ateapi/internal/authz/model.fga +++ b/cmd/ateapi/internal/authz/model.fga @@ -25,7 +25,7 @@ type user # Physical host node running atelet type node -# Worker pod running ateom (global-scoped) +# Worker pod running ateworker (global-scoped) type worker relations define parent_global: [global] @@ -148,8 +148,8 @@ type actor define can_update_egress_policy: editor from parent_atespace define can_delete_egress_policy: editor from parent_atespace - # Node Restriction: Only the node hosting the scheduled worker can mint ateom actor credentials - define can_mint_ateom_actor_credential: host_node + # Node Restriction: Only the node hosting the scheduled worker can mint ateworker actor credentials + define can_mint_worker_actor_credential: host_node # Credentials asserting the actor's own identity. Only the egress gateway # mints them; an editor holding one could impersonate the actor. diff --git a/cmd/ateapi/internal/authz/model_test.fga.yaml b/cmd/ateapi/internal/authz/model_test.fga.yaml index 0fb8c2f48e..0464b336cd 100644 --- a/cmd/ateapi/internal/authz/model_test.fga.yaml +++ b/cmd/ateapi/internal/authz/model_test.fga.yaml @@ -332,16 +332,16 @@ tests: can_suspend: true can_revert: true - - name: 5. Node Restriction (can_mint_ateom_actor_credential) + - name: 5. Node Restriction (can_mint_worker_actor_credential) check: - user: node:node-1 # Active host node object: actor:runner-1 assertions: - can_mint_ateom_actor_credential: true + can_mint_worker_actor_credential: true - user: node:node-2 # Other node object: actor:runner-1 assertions: - can_mint_ateom_actor_credential: false + can_mint_worker_actor_credential: false - name: 6. Worker Operations (Inherited from Global) check: diff --git a/cmd/ateapi/internal/authz/registry.go b/cmd/ateapi/internal/authz/registry.go index db8de54a6b..65231d9e9c 100644 --- a/cmd/ateapi/internal/authz/registry.go +++ b/cmd/ateapi/internal/authz/registry.go @@ -341,8 +341,8 @@ var defaultRPCPermissions = map[string]rpcRule{ ateapipb.WorkerService_RegisterWorker_FullMethodName: inHandler("ateletauth: caller is atelet on the Worker's node"), ateapipb.WorkerService_RequestActorSuspend_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker hosting the Actor"), // TODO: also require that the Actor runs on the caller's node. - ateapipb.WorkerService_MintAteomActorCertificate_FullMethodName: inHandler("ateletauth: caller is atelet"), - ateapipb.WorkerService_MintAteomActorJwt_FullMethodName: inHandler("ateletauth: caller is atelet on the node hosting the Actor"), + ateapipb.WorkerService_MintWorkerActorCertificate_FullMethodName: inHandler("ateletauth: caller is atelet"), + ateapipb.WorkerService_MintWorkerActorJwt_FullMethodName: inHandler("ateletauth: caller is atelet on the node hosting the Actor"), // TODO(#2570): not served yet; the handler must authenticate atelet on the // node of the Worker hosting the Actor when it lands. ateapipb.WorkerService_ReportActorSnapshotState_FullMethodName: inHandler("ateletauth: caller is atelet on the node of the Worker hosting the Actor"), diff --git a/cmd/ateapi/internal/controlapi/actor.go b/cmd/ateapi/internal/controlapi/actor.go index 75237457bf..b0591d18e0 100644 --- a/cmd/ateapi/internal/controlapi/actor.go +++ b/cmd/ateapi/internal/controlapi/actor.go @@ -31,6 +31,7 @@ import ( "github.com/agent-substrate/substrate/internal/actoridjwt" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/substratex509" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -92,6 +93,8 @@ func (s *ServiceImpl) CreateActor(ctx context.Context, inActor *ateapipb.Actor) if tagRef == nil { tagRef = template.GetStatus().GetGoldenSnapshotStatus().GetGoldenTag() } else { + // NOTE: whether the ExternalVolumes gate is enabled or not, we do not + // support cloning from a tag that has external volumes. for _, volume := range template.GetVolumes() { if volume.GetExternalVolumeTemplate() != nil { // TODO: Permit cloning after CSI volume snapshots are supported. @@ -116,9 +119,17 @@ func (s *ServiceImpl) CreateActor(ctx context.Context, inActor *ateapipb.Actor) name := inActor.GetMetadata().GetName() // Volume creation is completed asynchronously after the actor is recorded. - initVols, err := initialActorVolumes(ctx, s.storageClassLister, template) - if err != nil { + var initVols []*ateapipb.ExternalVolume + if iv, err := initialExternalVolumes(ctx, s.storageClassLister, template); err != nil { return nil, err + } else if preview.IsEnabledFor(ctx, preview.GateExternalVolumes) { + initVols = iv + } else if len(iv) > 0 { + errs := field.ErrorList{ + field.Forbidden(field.NewPath("actor", "actor_template"), + "ActorTemplates with external volumes require the ExternalVolumes preview gate to be enabled on the server and the client to opt in to preview features (kubectl ate --preview, or the "+preview.Header+" header)"), + } + return nil, resources.ToAPIError(errs) } // Verify that the result is properly valid before storing it. diff --git a/cmd/ateapi/internal/controlapi/actor_template_test.go b/cmd/ateapi/internal/controlapi/actor_template_test.go index b1a765421c..b28a4d5747 100644 --- a/cmd/ateapi/internal/controlapi/actor_template_test.go +++ b/cmd/ateapi/internal/controlapi/actor_template_test.go @@ -48,6 +48,7 @@ func validActorTemplate(mutations ...func(*ateapipb.ActorTemplate)) *ateapipb.Ac PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, + Resources: resources.CPUMemory(1000, 512<<20), } for _, m := range mutations { m(template) @@ -166,7 +167,7 @@ func TestCreateActorTemplateIgnoresServerOwnedFields(t *testing.T) { tmpl.Metadata.Version = 42 tmpl.WorkerSelector = &ateapipb.Selector{MatchLabels: map[string]string{"pool": "default"}} tmpl.Containers = []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}} - tmpl.Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "1Gi"}}} + tmpl.Resources = &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}} // Server-owned status a client must not be able to set. tmpl.Status = &ateapipb.ActorTemplateStatus{ GoldenSnapshotStatus: &ateapipb.GoldenSnapshotStatus{ diff --git a/cmd/ateapi/internal/controlapi/actor_test.go b/cmd/ateapi/internal/controlapi/actor_test.go index ce5dbf3011..479cb03204 100644 --- a/cmd/ateapi/internal/controlapi/actor_test.go +++ b/cmd/ateapi/internal/controlapi/actor_test.go @@ -278,6 +278,7 @@ func TestUpdateActor_RepointTemplate(t *testing.T) { Volumes: tmpl.volumes, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, SandboxConfig: tmpl.sandboxConfig, + Resources: resources.CPUMemory(1000, 512<<20), }); err != nil { t.Fatalf("creating template %s: %v", name, err) } diff --git a/cmd/ateapi/internal/controlapi/crash.go b/cmd/ateapi/internal/controlapi/crash.go index ee3bfc0988..c40628c995 100644 --- a/cmd/ateapi/internal/controlapi/crash.go +++ b/cmd/ateapi/internal/controlapi/crash.go @@ -44,7 +44,7 @@ const ( crashMessageWorkerReassigned = "assigned worker no longer hosts the actor" crashMessageWorkerIneligible = "assigned worker no longer satisfies the actor's placement constraints" crashMessageWorkerPodGone = "worker pod went away while hosting the actor" - crashMessageAteomRestarted = "ateom restarted while hosting the actor" + crashMessageWorkerRestarted = "worker restarted while hosting the actor" ) // maxCrashMessageBytes matches the maxLength on ActorCrash.message. @@ -190,8 +190,8 @@ func newActorCrash(opName, message string) *ateapipb.ActorCrash { // is barred from metric labels, so this record is the only way to attribute a // crash to one agent. Call it beside recordActorCrash, under the same guard. // -// It names ate.actor.state for the same reason ateom's lifecycle records do: a -// crash is the one transition ateom never observes, so a consumer taking the +// It names ate.actor.state for the same reason worker's lifecycle records do: a +// crash is the one transition worker never observes, so a consumer taking the // last state an actor reached has to see this record to reach "crashed" at all. func logActorCrashed(ctx context.Context, actor *ateapipb.Actor, opName string) { attrs := ateattr.ActorLogAttrs(resources.ActorAttributionFromActor(actor)) diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go index 9b09e2bee2..2ff8dbdd28 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_template_test.go @@ -21,6 +21,7 @@ import ( "time" "github.com/agent-substrate/substrate/cmd/ateapi/internal/controlapi" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "google.golang.org/grpc/status" "k8s.io/apimachinery/pkg/util/wait" @@ -28,6 +29,7 @@ import ( "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/google/go-cmp/cmp" "google.golang.org/grpc/codes" + "google.golang.org/grpc/metadata" "google.golang.org/protobuf/testing/protocmp" ) @@ -47,7 +49,7 @@ func TestActorTemplateCRUD(t *testing.T) { SandboxClass: "gvisor", ConfigName: "gvisor-default", }, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "1Gi"}}}, + Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}}, // Server-owned status on the request is ignored. Status: &ateapipb.ActorTemplateStatus{ GoldenSnapshotStatus: &ateapipb.GoldenSnapshotStatus{ @@ -71,7 +73,7 @@ func TestActorTemplateCRUD(t *testing.T) { SandboxClass: "gvisor", ConfigName: "gvisor-default", }, - Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "memory", Quantity: "1Gi"}}}, + Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}}, Status: &ateapipb.ActorTemplateStatus{}, } if diff := cmp.Diff(want, created, protocmp.Transform(), ignoreUID, ignoreTimestamps); diff != "" { @@ -84,6 +86,7 @@ func TestActorTemplateCRUD(t *testing.T) { Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, + Resources: resources.CPUMemory(1000, 512<<20), }, }) assertGrpcError(t, err, codes.AlreadyExists, "ActorTemplate "+testAtespace+"/tmpl-a already exists") @@ -140,6 +143,7 @@ func TestActorTemplateCRUD(t *testing.T) { Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}, + Resources: resources.CPUMemory(1000, 512<<20), }, }) assertGrpcErrorRegex(t, err, codes.InvalidArgument, `sandbox_config\.config_name`) @@ -254,6 +258,7 @@ func TestActorLifecycleWithoutGoldenSnapshot(t *testing.T) { SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, Containers: []*ateapipb.Container{{Name: "main", Image: "main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", Command: []string{"/main"}}}, WorkerSelector: &ateapipb.Selector{MatchLabels: map[string]string{poolLabelKey: ns}}, + Resources: resources.CPUMemory(1000, 512<<20), }}) if err != nil { t.Fatal(err) @@ -388,3 +393,55 @@ func TestDeleteActorTemplate_Preconditions(t *testing.T) { _, err := tc.client.GetActorTemplate(ctx, &ateapipb.GetActorTemplateRequest{ActorTemplate: ref}) assertGrpcError(t, err, codes.NotFound, "ActorTemplate "+templateRef.String()+" not found") } + +// TestCreateActorTemplate_ExternalVolumesPreview checks that external volumes +// are accepted only when the server enables the gate and the client opts in +// to preview features. +func TestCreateActorTemplate_ExternalVolumesPreview(t *testing.T) { + ns := namespaceForTest("ns-template-ext-vol-preview") + tc := setupTest(t, ns) + defer tc.cleanup() + ensureDefaultGvisorSandboxConfig(t, tc) + + for i, tt := range []struct { + name string + gates []string + optIn bool + wantCode codes.Code + }{ + {name: "gate off", gates: nil, optIn: true, wantCode: codes.InvalidArgument}, + {name: "no opt-in", gates: []string{"ExternalVolumes"}, optIn: false, wantCode: codes.InvalidArgument}, + {name: "gate on and opt-in", gates: []string{"ExternalVolumes"}, optIn: true, wantCode: codes.OK}, + } { + t.Run(tt.name, func(t *testing.T) { + preview.InitForTest(t, tt.gates...) + ctx := context.Background() + if tt.optIn { + ctx = metadata.AppendToOutgoingContext(ctx, preview.Header, "true") + } + _, err := tc.client.CreateActorTemplate(ctx, &ateapipb.CreateActorTemplateRequest{ + ActorTemplate: &ateapipb.ActorTemplate{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: fmt.Sprintf("tmpl-ext-%d", i)}, + Containers: []*ateapipb.Container{{ + Name: "main", + Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + VolumeMounts: []*ateapipb.VolumeMount{{Name: "data", MountPath: "/data"}}, + }}, + Volumes: []*ateapipb.Volume{{ + Name: "data", + ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", Capacity: "1Gi"}, + }}, + Resources: &ateapipb.Resources{Limits: []*ateapipb.Limits{{Name: "cpu", Quantity: "1"}, {Name: "memory", Quantity: "1Gi"}}}, + SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, + SandboxConfig: &ateapipb.SandboxConfig{ + SandboxClass: "gvisor", + ConfigName: "gvisor-default", + }, + }, + }) + if status.Code(err) != tt.wantCode { + t.Fatalf("CreateActorTemplate error = %v, want %v", err, tt.wantCode) + } + }) + } +} diff --git a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go index 24ae268e3c..02f6acd32d 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/actor_test.go @@ -33,6 +33,7 @@ import ( "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/objectstore" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/volume" @@ -111,8 +112,10 @@ func TestCreateActor_Success(t *testing.T) { } func TestCreateActor_WithExternalVolumes(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-create-ext-vols") - tc := setupTest(t, ns) + tc := setupPreviewTest(t, ns) defer tc.cleanup() volumes := []*ateapipb.Volume{ @@ -168,6 +171,84 @@ func TestCreateActor_WithExternalVolumes(t *testing.T) { } } +// TestCreateActor_ExternalVolumesWithoutOptIn checks that a client which did +// not opt in to preview features cannot create an Actor whose template has +// external volumes, even though the server enables the gate. +func TestCreateActor_ExternalVolumesWithoutOptIn(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + + ns := namespaceForTest("ns-create-ext-vols-no-opt-in") + tc := setupPreviewTest(t, ns) + defer tc.cleanup() + + createTemplateWithVolumes(t, tc, ns, + []*ateapipb.Volume{{ + Name: "ext-vol-1", + ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "standard", Capacity: "10Gi"}, + }}, + []*ateapipb.VolumeMount{{Name: "ext-vol-1", MountPath: "/data"}}) + + _, err := tc.noOptInClient.CreateActor(context.Background(), &ateapipb.CreateActorRequest{ + Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: "vol-actor-1"}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, + }, + }) + if status.Code(err) != codes.InvalidArgument { + t.Fatalf("CreateActor error = %v, want InvalidArgument", err) + } + if !strings.Contains(err.Error(), preview.Header) { + t.Errorf("CreateActor error = %v, want it to mention %s", err, preview.Header) + } +} + +// TestActorLifecycle_ExternalVolumesWithoutOptIn checks that a client which +// did not opt in to preview features can still resume, suspend and delete an +// Actor with external volumes: those requests carry no preview fields, and the +// volume state they change is written by workflows on the system's behalf. +func TestActorLifecycle_ExternalVolumesWithoutOptIn(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + + ns := namespaceForTest("ns-lifecycle-ext-vols-no-opt-in") + tc := setupPreviewTest(t, ns) + defer tc.cleanup() + + createTemplateWithVolumes(t, tc, ns, + []*ateapipb.Volume{{ + Name: "data-vol", + ExternalVolumeTemplate: &ateapipb.ExternalVolumeTemplate{StorageClassName: "fast", Capacity: "20Gi"}, + }}, + []*ateapipb.VolumeMount{{Name: "data-vol", MountPath: "/mnt/data"}}) + createWorkerPod(t, tc, ns, "worker-1", "node1", "pool1") + + actorRef := &ateapipb.ObjectRef{Atespace: testAtespace, Name: "actor-vol-no-opt-in"} + if _, err := tc.client.CreateActor(context.Background(), &ateapipb.CreateActorRequest{ + Actor: &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: testAtespace, Name: actorRef.Name}, + ActorTemplate: &ateapipb.ObjectRef{Atespace: testAtespace, Name: "tmpl1"}, + }, + }); err != nil { + t.Fatalf("CreateActor failed: %v", err) + } + + _, err := tc.noOptInClient.ResumeActor(context.Background(), &ateapipb.ResumeActorRequest{Actor: actorRef}) + if err != nil { + t.Fatalf("ResumeActor failed: %v", err) + } + + if _, err := tc.noOptInClient.SuspendActor(context.Background(), &ateapipb.SuspendActorRequest{Actor: actorRef}); err != nil { + t.Fatalf("SuspendActor failed: %v", err) + } + + if _, err := tc.noOptInClient.DeleteActor(context.Background(), &ateapipb.DeleteActorRequest{Actor: actorRef}); err != nil { + t.Fatalf("DeleteActor failed: %v", err) + } + _, err = tc.client.GetActor(context.Background(), &ateapipb.GetActorRequest{Actor: actorRef}) + if status.Code(err) != codes.NotFound { + t.Errorf("GetActor after delete err = %v, want NotFound", err) + } +} + // TestCreateActor_TemplateNotFound tests that creating an actor with a non-existent template fails with FailedPrecondition. func TestCreateActor_TemplateNotFound(t *testing.T) { ns := namespaceForTest("ns-create-notfound") @@ -199,6 +280,7 @@ func TestCreateActor_SubstrateTemplateRef(t *testing.T) { Containers: []*ateapipb.Container{{Name: "main", Image: "example.com/app:v1@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: "gvisor-default"}, + Resources: resources.CPUMemory(1000, 512<<20), }, }); err != nil { t.Fatalf("CreateActorTemplate failed: %v", err) @@ -312,8 +394,10 @@ func TestCreateActor_RejectsDifferentTemplateForDataSnapshot(t *testing.T) { } func TestCreateActor_RejectsSnapshotWithExternalVolumes(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-snapshot-external-volume") - tc := setupTest(t, ns) + tc := setupPreviewTest(t, ns) defer tc.cleanup() ensureDefaultGvisorSandboxConfig(t, tc) template, err := tc.client.CreateActorTemplate(context.Background(), &ateapipb.CreateActorTemplateRequest{ @@ -333,6 +417,7 @@ func TestCreateActor_RejectsSnapshotWithExternalVolumes(t *testing.T) { Capacity: "1Gi", StorageClassName: "standard", }, }}, + Resources: resources.CPUMemory(1000, 512<<20), }, }) if err != nil { @@ -769,6 +854,7 @@ func TestUpdateActor_RepointTemplate(t *testing.T) { Volumes: tmpl.volumes, SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://my-bucket/snapshots"}, SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor", ConfigName: tmpl.configName}, + Resources: resources.CPUMemory(1000, 512<<20), }, }); err != nil { t.Fatalf("CreateActorTemplate %s failed: %v", name, err) @@ -1403,6 +1489,8 @@ func (f *failingVolumePlugin) DeleteVolume(ctx context.Context, volumeID string) } func TestDeleteActor_MultipleVolumeDeletionFailures(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-delete-multivol-fail") plugin := &failingVolumePlugin{} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -1478,6 +1566,8 @@ func (r *retryDeleteVolumePlugin) DeleteVolume(ctx context.Context, volumeID str // 5. Retries DeleteActor, which re-attempts volume deletion and cleans up the actor from the store. // 6. Confirms GetActor returns NotFound. func TestDeleteActor_VolumeDeletionFailure_RetrySuccess(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-delete-vol-retry") plugin := &retryDeleteVolumePlugin{failDelete: true} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -1620,8 +1710,10 @@ func TestValidation_Actor(t *testing.T) { } func TestActorLifecycle_WithExternalVolumes(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-lifecycle-ext-vols") - tc := setupTest(t, ns) + tc := setupPreviewTest(t, ns) defer tc.cleanup() volumes := []*ateapipb.Volume{ @@ -1763,6 +1855,8 @@ func (f *partialFailVolumePlugin) DeleteVolume(ctx context.Context, volumeID str // successfully created volumes are saved, the actor remains in ACTOR_STATE_SUSPENDED, // and that calling DeleteActor on the suspended actor cleans up all partially created volumes. func TestResumeActor_VolumeCreationFailure(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-resume-vol-fail") plugin := &partialFailVolumePlugin{} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -1905,6 +1999,8 @@ func (r *retrySuccessVolumePlugin) DeleteVolume(ctx context.Context, volumeID st // TestResumeActor_VolumeCreationRetrySuccess tests that when volume provisioning fails on the first ResumeActor call, // a subsequent call to ResumeActor retries provisioning only the pending volumes and succeeds. func TestResumeActor_VolumeCreationRetrySuccess(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-resume-vol-retry") plugin := &retrySuccessVolumePlugin{} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -2059,6 +2155,8 @@ func (a *attachFailVolumePlugin) DeleteVolume(ctx context.Context, volumeID stri // and provisioned volumes intact, and that a subsequent call to ResumeActor re-attempts volume attachment // and successfully transitions the actor to ACTOR_STATE_RUNNING. func TestResumeActor_VolumeAttachFailureAndRetry(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-resume-vol-attach-retry") plugin := &attachFailVolumePlugin{failUntil: 1} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -2161,6 +2259,8 @@ func TestResumeActor_VolumeAttachFailureAndRetry(t *testing.T) { // calling DeleteActor without any_state is rejected because the actor is in ACTOR_STATE_RESUMING, // but calling DeleteActor with any_state=true succeeds, cleaning up worker assignment and volumes. func TestResumeActor_VolumeAttachFailure_DeleteActor(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-resume-vol-attach-del") plugin := &attachFailVolumePlugin{failUntil: 100} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -2315,6 +2415,8 @@ func (m *multiVolAttachPlugin) DeleteVolume(ctx context.Context, volumeID string // 6. Verifies ResumeActor succeeds and actor transitions to ACTOR_STATE_RUNNING. // 7. Cleans up by suspending and deleting the actor. func TestResumeActor_MultiVolumePartialAttachFailure_Retry(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-resume-multivol-attach-retry") plugin := newMultiVolAttachPlugin("vol2", 1) tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -2485,6 +2587,8 @@ func (d *detachFailVolumePlugin) DeleteVolume(ctx context.Context, volumeID stri // 6. Verifies actor transitions to ACTOR_STATE_SUSPENDED and worker assignment is cleared. // 7. Cleans up by deleting the actor. func TestSuspendActor_VolumeDetachFailure_RetrySuccess(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-suspend-vol-detach-retry") plugin := &detachFailVolumePlugin{failUntil: 1} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -2595,6 +2699,8 @@ func TestSuspendActor_VolumeDetachFailure_RetrySuccess(t *testing.T) { // 5. Resets the plugin to allow detachment, then calls DeleteActor with AnyState=true. // 6. Verifies the actor is deleted and GetActor returns NotFound. func TestSuspendActor_VolumeDetachFailure_DeleteActorAnyState(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-suspend-vol-detach-del") plugin := &detachFailVolumePlugin{failUntil: 100} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -2694,6 +2800,8 @@ func TestSuspendActor_VolumeDetachFailure_DeleteActorAnyState(t *testing.T) { // 4. Calls ResumeActor from pause; verifies volume is re-attached to node1 and actor transitions to ACTOR_STATE_RUNNING. // 5. Cleans up by suspending and deleting the actor. func TestPauseActor_VolumeLifecycle_DetachAndResumeAttach(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-pause-vol-lifecycle") plugin := &attachFailVolumePlugin{} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -2820,6 +2928,8 @@ func TestPauseActor_VolumeLifecycle_DetachAndResumeAttach(t *testing.T) { // 6. Verifies actor state transitions to ACTOR_STATE_SUSPENDED with a local snapshot. // 7. Cleans up by deleting the actor with AnyState=true. func TestPauseActor_VolumeDetachFailure_RetrySuccess(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") + ns := namespaceForTest("ns-pause-vol-detach-retry") plugin := &detachFailVolumePlugin{failUntil: 1} tc := setupTestWithVolumePlugins(t, ns, map[string]volume.VolumePluginControlPlane{ @@ -3013,11 +3123,10 @@ func TestResumeActor(t *testing.T) { Status: &ateapipb.WorkerStatus{ State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, // Only the ceiling the worker reported. - Capacity: &ateapipb.WorkerResources{Actors: 1}, - // All a listing reports of the assignments. The actor declares - // no compute limits, so it registers as one actor and nothing - // else. - Allocated: &ateapipb.WorkerResources{Actors: 1}, + Capacity: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)}, + // All a listing reports of the assignments: one actor and the + // compute its template declares. + Allocated: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)}, }, } @@ -3110,6 +3219,7 @@ func createDataFidelityTemplate(t *testing.T, tc *testContext, ns string) *ateap WorkerSelector: &ateapipb.Selector{ MatchLabels: map[string]string{poolLabelKey: ns}, }, + Resources: resources.CPUMemory(1000, 512<<20), }, }) if err != nil { diff --git a/cmd/ateapi/internal/controlapi/functionaltest/common_test.go b/cmd/ateapi/internal/controlapi/functionaltest/common_test.go index 8a4cf14662..c0f912416b 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/common_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/common_test.go @@ -32,6 +32,7 @@ import ( "github.com/agent-substrate/substrate/internal/localjwtauthority" "github.com/agent-substrate/substrate/internal/objectstore/objectstoretest" "github.com/agent-substrate/substrate/internal/objectstoreplugin/objectstoreplugintest" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/volume" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" @@ -84,6 +85,7 @@ var ( type testContext struct { service *controlapi.RPCService client ateapipb.ControlClient + noOptInClient ateapipb.ControlClient // never sends preview.Header k8sClient kubernetes.Interface substrateClient versioned.Interface persistence store.Interface @@ -106,15 +108,38 @@ type testContext struct { // setupTest sets up a fully isolated test environment. func setupTest(t *testing.T, ns string) *testContext { t.Helper() - return setupTestWithVolumePlugins(t, ns, nil) + return setupTestWithOptions(t, ns, testOptions{}) } -// setupTestWithVolumePlugins is setupTest with the default mock volume plugin -// replaced by plugins, keyed by driver name. Tests that need a failure-injecting -// plugin pass it here rather than swapping it into the running RPCService, so each -// test owns its own plugin set. +type testOptions struct { + allowPreview bool // enable use of preview features over gRPC + volumePlugins map[string]volume.VolumePluginControlPlane +} + +// setupPreviewTest is setupTest with a client that opts in to preview +// features. It does not enable any gate; call preview.InitForTest for that. +func setupPreviewTest(t *testing.T, ns string) *testContext { + t.Helper() + return setupTestWithOptions(t, ns, testOptions{allowPreview: true}) +} + +// setupTestWithVolumePlugins is setupPreviewTest with the default mock volume +// plugin replaced by plugins, keyed by driver name. Tests that need a +// failure-injecting plugin pass it here rather than swapping it into the +// running RPCService, so each test owns its own plugin set. func setupTestWithVolumePlugins(t *testing.T, ns string, plugins map[string]volume.VolumePluginControlPlane) *testContext { t.Helper() + return setupTestWithOptions(t, ns, testOptions{volumePlugins: plugins, allowPreview: true}) +} + +func setupTestWithOptions(t *testing.T, ns string, options testOptions) *testContext { + t.Helper() + + var dialOpts []grpc.DialOption + if options.allowPreview { + dialOpts = preview.DialOptionsWithOptIn() + } + // 1. Start an isolated PostgreSQL-backed store. persistence, cleanupStore := storetest.SetupTestStore(t) @@ -173,7 +198,7 @@ func setupTestWithVolumePlugins(t *testing.T, ns string, plugins map[string]volu cleanupStore() t.Fatalf("failed to create metric instruments: %v", err) } - volPlugins := plugins + volPlugins := options.volumePlugins if volPlugins == nil { mockPlugin := volume.NewMockVolumePlugin() mockDriverName, err := mockPlugin.DriverName(ctx) @@ -232,6 +257,7 @@ func setupTestWithVolumePlugins(t *testing.T, ns string, plugins map[string]volu grpcServer := grpc.NewServer(grpc.ChainUnaryInterceptor( ateinterceptors.ServerUnaryInterceptor, ateinterceptors.RejectUnknownFieldsUnaryInterceptor, + preview.UnaryServerInterceptor, )) ateapipb.RegisterControlServer(grpcServer, service) @@ -243,19 +269,28 @@ func setupTestWithVolumePlugins(t *testing.T, ns string, plugins map[string]volu } }() - conn, err := grpc.NewClient( - "passthrough://bufnet", + baseDialOpts := []grpc.DialOption{ grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, s string) (net.Conn, error) { return lis.Dial() }), - ) + } + conn, err := grpc.NewClient("passthrough://bufnet", append(baseDialOpts, dialOpts...)...) + if err != nil { + grpcServer.Stop() + cancel() + cleanupStore() + t.Fatalf("failed to connect: %v", err) + } + noOptInConn, err := grpc.NewClient("passthrough://bufnet", baseDialOpts...) if err != nil { + conn.Close() grpcServer.Stop() cancel() cleanupStore() t.Fatalf("failed to connect: %v", err) } + t.Cleanup(func() { noOptInConn.Close() }) client := ateapipb.NewControlClient(conn) @@ -296,6 +331,7 @@ func setupTestWithVolumePlugins(t *testing.T, ns string, plugins map[string]volu return &testContext{ service: service, client: client, + noOptInClient: ateapipb.NewControlClient(noOptInConn), k8sClient: k8sClient, substrateClient: substrateClient, persistence: persistence, @@ -458,6 +494,7 @@ func createTemplateWithContainersAndVolumes(t *testing.T, tc *testContext, ns st WorkerSelector: &ateapipb.Selector{ MatchLabels: map[string]string{poolLabelKey: ns}, }, + Resources: resources.CPUMemory(1000, 512<<20), }, }) if err != nil { @@ -617,7 +654,7 @@ func createWorkerPool(t *testing.T, tc *testContext, ns string, name string, lab }, Spec: atev1alpha1.WorkerPoolSpec{ Replicas: 1, - WorkerImage: "ateom@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + WorkerImage: "worker@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", SandboxClasses: []atev1alpha1.WorkerPoolSandboxClass{{Name: atev1alpha1.SandboxClassGvisor}}, }, } @@ -665,6 +702,7 @@ func createTemplateWithSelector(t *testing.T, tc *testContext, name string, sele {Name: "main", Image: "main@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", Command: []string{"/main"}}, }, WorkerSelector: selector, + Resources: resources.CPUMemory(1000, 512<<20), }, }) if err != nil { @@ -734,7 +772,7 @@ func createWorkerPod(t *testing.T, tc *testContext, ns string, name string, node SandboxClass: string(pool.Spec.DefaultSandboxClass()), Labels: pool.GetLabels(), // Capacity is not settable here: a Worker gets it from its own - // ateom's report, which the reportWorkerCapacity below stands in + // worker's report, which the reportWorkerCapacity below stands in // for. These pods declare no limits, so only the actor ceiling is // reported; see setWorkerActorCapacity for tests needing more. }, diff --git a/cmd/ateapi/internal/controlapi/functionaltest/main_test.go b/cmd/ateapi/internal/controlapi/functionaltest/main_test.go index f622d4e8f9..f169e2523f 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/main_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/main_test.go @@ -87,7 +87,7 @@ func TestMain(m *testing.M) { // Start Fake Atelet Server on port 8085 ateletGrpcServer := grpc.NewServer() - ateletpb.RegisterAteomHerderServer(ateletGrpcServer, fakeAtelet) + ateletpb.RegisterAteletServer(ateletGrpcServer, fakeAtelet) ateletLis, err := net.Listen("tcp", "127.0.0.1:8085") if err != nil { log.Fatalf("listen on 127.0.0.1:8085: %v", err) @@ -111,7 +111,7 @@ func TestMain(m *testing.M) { // FakeAteletServer implements ateletpb.WorkersServer type FakeAteletServer struct { - ateletpb.UnimplementedAteomHerderServer + ateletpb.UnimplementedAteletServer Lock sync.Mutex diff --git a/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go b/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go index a212942116..8325eb9c20 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/worker_contention_test.go @@ -24,6 +24,7 @@ import ( "k8s.io/apimachinery/pkg/util/wait" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) @@ -130,7 +131,7 @@ func TestResumeActor_ConcurrentOntoOneWorker(t *testing.T) { // setWorkerActorCapacity raises the actors ceiling on every Worker in the pool. // -// The ceiling is the Worker's, reported by its ateom, so a test that wants more +// The ceiling is the Worker's, reported by its worker, so a test that wants more // than the unset default of one writes it where the reporter would. Waits for // the scheduler's cache to see it, since placement reads that and not the store. func setWorkerActorCapacity(t *testing.T, tc *testContext, pool string, actors int32) { @@ -152,9 +153,10 @@ func setWorkerActorCapacity(t *testing.T, tc *testContext, pool string, actors i } } -// reportWorkerCapacity stands in for the ateom's capacity report, which atelet -// forwards to WorkerService in a real cluster. It waits for the worker cache, -// which placement reads, to catch up. +// reportWorkerCapacity stands in for the worker's capacity report, which atelet +// forwards to WorkerService in a real cluster, with compute for actors of the +// size test templates declare. It waits for the worker cache, which placement +// reads, to catch up. func reportWorkerCapacity(t *testing.T, tc *testContext, name string, actors int32) { t.Helper() ctx := context.Background() @@ -163,7 +165,10 @@ func reportWorkerCapacity(t *testing.T, tc *testContext, name string, actors int t.Fatalf("getting worker %s: %v", name, err) } if _, err := tc.persistence.UpdateWorker(ctx, name, store.PreconditionFrom(worker), func(toUpdate *ateapipb.Worker) error { - toUpdate.Status.Capacity = &ateapipb.WorkerResources{Actors: actors} + toUpdate.Status.Capacity = &ateapipb.WorkerResources{ + Actors: actors, + Resources: resources.CPUMemory(int64(actors)*1000, int64(actors)*512<<20), + } return nil }); err != nil { t.Fatalf("setting capacity on worker %s: %v", name, err) diff --git a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go index 200a5d5035..810f5bb59f 100644 --- a/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go +++ b/cmd/ateapi/internal/controlapi/functionaltest/worker_test.go @@ -20,6 +20,7 @@ import ( "testing" "time" + "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/google/go-cmp/cmp" "google.golang.org/grpc/codes" @@ -33,7 +34,7 @@ const testWorkerName = "5f2c1a90-7b34-4e6d-8a11-0c3e9d5b7f42" // newTestWorker returns a Worker in the shape CreateWorker accepts: named after // its pod UID, with the pod coordinates filled in and no status. Status is // output-only, and that includes capacity — a Worker gets that from its own -// ateom's report to WorkerService, not from whoever registered it. These tests +// worker's report to WorkerService, not from whoever registered it. These tests // never place an actor, so the unreported default is enough. // // The Worker stands on its own, with no pod behind it. That is enough for the @@ -127,7 +128,7 @@ func TestCreateAndGetWorker(t *testing.T) { // 1. Creates a mock WorkerPool in Kubernetes. // 2. Creates a mock worker Pod in Kubernetes belonging to that pool, registers // the Worker the syncer would derive from it, and reports the capacity its -// ateom would. +// worker would. // 3. Calls ListWorkers RPC. // 4. Verifies that the worker appears in the response. func TestListWorkers(t *testing.T) { @@ -166,7 +167,10 @@ func TestListWorkers(t *testing.T) { Ips: []string{"127.0.0.1"}, SandboxClass: "gvisor", Labels: map[string]string{"foo": "bar"}, - Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}}, + Status: &ateapipb.WorkerStatus{ + State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, + Capacity: &ateapipb.WorkerResources{Actors: 1, Resources: resources.CPUMemory(1000, 512<<20)}, + }, }, } diff --git a/cmd/ateapi/internal/controlapi/hosting_test.go b/cmd/ateapi/internal/controlapi/hosting_test.go index 585913fc95..8e6e0b130e 100644 --- a/cmd/ateapi/internal/controlapi/hosting_test.go +++ b/cmd/ateapi/internal/controlapi/hosting_test.go @@ -39,7 +39,10 @@ import ( // node must not count as hosting the actor in that window, nor once the worker // hosts another actor. func TestHostingNodeAfterWorkerRelease(t *testing.T) { - const node = "node1" + // The worker is the one wireTestAssignment names, so a pause or suspend + // reaches newWireCaptureWorkflow's fake atelet on its node. + wa := wireTestAssignment() + node := wa.GetNodeName() gvisorTemplate := &ateapipb.ActorTemplate{SandboxConfig: &ateapipb.SandboxConfig{SandboxClass: "gvisor"}} tests := []struct { name string @@ -49,9 +52,9 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { // workerClass is the worker's sandbox class. A template asks for // gVisor, so "microvm" makes the worker ineligible. workerClass string - // finalize runs the step that releases the worker and then records - // the actor's new state against st. - finalize func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, actor *ateapipb.Actor) error + // finalize runs the workflow or step that releases the worker and + // then records the actor's new state against st. + finalize func(t *testing.T, ctx context.Context, st store.Interface, actorRef resources.ActorRef, actor *ateapipb.Actor) error // wantRetry is the code of the retried finalize against the real // store. wantRetry codes.Code @@ -68,8 +71,11 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { return newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local-snap-1", "") }, workerClass: "gvisor", - finalize: func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { - _, err := (&ActorWorkflow{store: st}).ensurePausedFinalized(ctx, actorRef) + // The actor is seeded SUSPENDING, so the pause skips its mark step + // and its first actor update is the finalize commit. + finalize: func(t *testing.T, ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { + w, _ := newWireCaptureWorkflow(t, st) + _, err := w.PauseActor(ctx, actorRef) return err }, wantRetry: codes.OK, @@ -84,8 +90,10 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { return newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snapshot-1", someActorSnapshotURI(t, testStorageLocation, "team-a", "snapshot-1"), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) }, workerClass: "gvisor", - finalize: func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { - _, err := (&ActorWorkflow{store: st}).ensureSuspendedFinalized(ctx, actorRef) + // As for the pause, the finalize commit is the first actor update. + finalize: func(t *testing.T, ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { + w, _ := newWireCaptureWorkflow(t, st) + _, err := w.SuspendActor(ctx, actorRef) return err }, wantRetry: codes.OK, @@ -95,7 +103,7 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { name: "crash", state: ateapipb.ActorState_ACTOR_STATE_RUNNING, workerClass: "gvisor", - finalize: func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { + finalize: func(_ *testing.T, ctx context.Context, st store.Interface, actorRef resources.ActorRef, _ *ateapipb.Actor) error { return crashActor(ctx, st, actorRef, ateattr.OperationResume, "test crash") }, wantRetry: codes.OK, @@ -108,7 +116,7 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { name: "resume onto an ineligible worker", state: ateapipb.ActorState_ACTOR_STATE_RESUMING, workerClass: "microvm", - finalize: func(ctx context.Context, st store.Interface, actorRef resources.ActorRef, actor *ateapipb.Actor) error { + finalize: func(_ *testing.T, ctx context.Context, st store.Interface, actorRef resources.ActorRef, actor *ateapipb.Actor) error { _, err := (&ActorWorkflow{store: st, scheduler: scheduling.New(nil)}).validateAssignedWorker(ctx, actorRef, actor, gvisorTemplate) return err }, @@ -120,14 +128,15 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { t.Run(tc.name, func(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} - workerName := testWorkerUID("worker-pod-1") + workerName := wa.GetWorker().GetName() if _, err := persistence.CreateWorker(ctx, &ateapipb.Worker{ Metadata: &ateapipb.ResourceMetadata{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, + WorkerNamespace: wa.GetWorkerNamespace(), + WorkerPool: wa.GetWorkerPool(), + WorkerPod: wa.GetWorkerPod(), + WorkerPodUid: wa.GetWorkerPodUid(), NodeName: node, SandboxClass: tc.workerClass, Status: &ateapipb.WorkerStatus{State: ateapipb.WorkerState_WORKER_STATE_ACTIVE, Capacity: &ateapipb.WorkerResources{Actors: 1}}, @@ -135,24 +144,18 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { t.Fatalf("CreateWorker: %v", err) } status := &ateapipb.ActorStatus{ - State: tc.state, - AssignedNode: node, - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - NodeName: node, - }, + State: tc.state, + AssignedNode: node, + WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, } if tc.snapshot != nil { status.Snapshots = []*ateapipb.Snapshot{tc.snapshot(t)} } created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, - Status: status, + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), + Status: status, }) uid := created.GetMetadata().GetUid() seedAssignment(t, persistence, workerName, &ateapipb.ActorAssignment{ @@ -181,7 +184,7 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { } // The release commits; recording the new state fails. - if err := tc.finalize(ctx, conflictingUpdateStore{Interface: persistence}, actorRef, actor); err == nil { + if err := tc.finalize(t, ctx, conflictingUpdateStore{Interface: persistence}, actorRef, actor); err == nil { t.Fatal("finalize succeeded through a store that fails every actor update") } if _, err := persistence.GetWorkerAssignment(ctx, workerName, uid); !errors.Is(err, store.ErrNotFound) { @@ -205,7 +208,7 @@ func TestHostingNodeAfterWorkerRelease(t *testing.T) { } // A retry records the new state. - if err := tc.finalize(ctx, persistence, actorRef, actor); apierror.Code(err) != tc.wantRetry { + if err := tc.finalize(t, ctx, persistence, actorRef, actor); apierror.Code(err) != tc.wantRetry { t.Fatalf("finalize retry: code %v, want %v (err: %v)", apierror.Code(err), tc.wantRetry, err) } got, actor = hostingNode("finalized") diff --git a/cmd/ateapi/internal/controlapi/template_reconciler_test.go b/cmd/ateapi/internal/controlapi/template_reconciler_test.go index d8a70e47f5..f4dfd27dd9 100644 --- a/cmd/ateapi/internal/controlapi/template_reconciler_test.go +++ b/cmd/ateapi/internal/controlapi/template_reconciler_test.go @@ -33,7 +33,6 @@ import ( "google.golang.org/grpc/codes" "google.golang.org/protobuf/proto" "google.golang.org/protobuf/types/known/timestamppb" - "k8s.io/apimachinery/pkg/api/operation" ) // fakeTemplateStore is an in-memory templateReconcilerStore. @@ -786,7 +785,7 @@ func TestFail_TruncatesErrorMessage(t *testing.T) { if !utf8.ValidString(msg) { t.Error("error_message is not valid UTF-8") } - op := operation.Operation{Type: operation.Update} + op := apivalidation.MakeUpdateOp(ctx) if errs := apivalidation.Validate_GoldenSnapshotStatus(ctx, op, nil, golden, &ateapipb.GoldenSnapshotStatus{}); len(errs) != 0 { t.Errorf("stored status fails validation: %v", errs) } diff --git a/cmd/ateapi/internal/controlapi/volumes.go b/cmd/ateapi/internal/controlapi/volumes.go index 22c65fbea7..021a79ccf9 100644 --- a/cmd/ateapi/internal/controlapi/volumes.go +++ b/cmd/ateapi/internal/controlapi/volumes.go @@ -29,9 +29,9 @@ import ( storagev1listers "k8s.io/client-go/listers/storage/v1" ) -// initialActorVolumes constructs the actor's volume objects before volume +// initialExternalVolumes constructs the actor's volume objects before volume // creation. A volume has no storage volume ID until it is provisioned. -func initialActorVolumes(ctx context.Context, scLister storagev1listers.StorageClassLister, template *ateapipb.ActorTemplate) ([]*ateapipb.ExternalVolume, error) { +func initialExternalVolumes(ctx context.Context, scLister storagev1listers.StorageClassLister, template *ateapipb.ActorTemplate) ([]*ateapipb.ExternalVolume, error) { if template == nil { return nil, apierror.InvalidArgument("template is required") } diff --git a/cmd/ateapi/internal/controlapi/volumes_test.go b/cmd/ateapi/internal/controlapi/volumes_test.go index e9be56827d..545384d569 100644 --- a/cmd/ateapi/internal/controlapi/volumes_test.go +++ b/cmd/ateapi/internal/controlapi/volumes_test.go @@ -91,12 +91,12 @@ func TestActorVolumesStorageClassErrors(t *testing.T) { t.Run(tt.name, func(t *testing.T) { lister := &fakeStorageClassLister{getErr: tt.getErr} t.Run("initial", func(t *testing.T) { - _, err := initialActorVolumes(ctx, lister, tmpl) + _, err := initialExternalVolumes(ctx, lister, tmpl) if got := apierror.Code(err); got != tt.wantCode { - t.Fatalf("initialActorVolumes() code = %v, want %v; error = %v", got, tt.wantCode, err) + t.Fatalf("initialExternalVolumes() code = %v, want %v; error = %v", got, tt.wantCode, err) } if !strings.Contains(err.Error(), `StorageClass "standard"`) { - t.Errorf("initialActorVolumes() error does not name the StorageClass: %v", err) + t.Errorf("initialExternalVolumes() error does not name the StorageClass: %v", err) } }) t.Run("create", func(t *testing.T) { @@ -163,12 +163,12 @@ func TestInitialActorVolumes(t *testing.T) { }, }, } - initVols, err := initialActorVolumes(context.Background(), scLister, tmpl) + initVols, err := initialExternalVolumes(context.Background(), scLister, tmpl) if err != nil { - t.Fatalf("initialActorVolumes failed: %v", err) + t.Fatalf("initialExternalVolumes failed: %v", err) } if diff := cmp.Diff(want, initVols, protocmp.Transform()); diff != "" { - t.Errorf("initialActorVolumes mismatch (-want +got):\n%s", diff) + t.Errorf("initialExternalVolumes mismatch (-want +got):\n%s", diff) } } diff --git a/cmd/ateapi/internal/controlapi/worker.go b/cmd/ateapi/internal/controlapi/worker.go index 0e836c4588..8fb2d47c83 100644 --- a/cmd/ateapi/internal/controlapi/worker.go +++ b/cmd/ateapi/internal/controlapi/worker.go @@ -131,7 +131,7 @@ func (s *ServiceImpl) CreateWorker(ctx context.Context, inWorker *ateapipb.Worke } // Capacity and runtimes are left unset: a Worker holds nothing and runs no - // sandbox until its own ateom says what it has, through + // sandbox until its own worker says what it has, through // WorkerService.RegisterWorker. Nothing is placed on it in the meantime, // which is the point -- the alternative is guessing on the Worker's behalf // and placing against the guess. diff --git a/cmd/ateapi/internal/controlapi/worker_assignment_reconciler.go b/cmd/ateapi/internal/controlapi/worker_assignment_reconciler.go index 2bc43b13f9..45c3c85583 100644 --- a/cmd/ateapi/internal/controlapi/worker_assignment_reconciler.go +++ b/cmd/ateapi/internal/controlapi/worker_assignment_reconciler.go @@ -41,7 +41,7 @@ type workerWatcher interface { // WorkerAssignmentReconciler keeps the Actors assigned to a Worker in line with // its epoch. It watches for Workers whose epoch has risen past -// status.observed_epoch, and crashes the Actors a restarted ateom took with it. +// status.observed_epoch, and crashes the Actors a restarted worker took with it. // // TODO: Every ateapi replica runs this reconciler and queues every Worker, with // only a per-Worker lease keeping them from releasing the same one at once. diff --git a/cmd/ateapi/internal/controlapi/worker_test.go b/cmd/ateapi/internal/controlapi/worker_test.go index dc395e12ea..ab422b2853 100644 --- a/cmd/ateapi/internal/controlapi/worker_test.go +++ b/cmd/ateapi/internal/controlapi/worker_test.go @@ -880,7 +880,7 @@ func TestCreateWorker_HoldsNoCapacityUntilReported(t *testing.T) { t.Fatalf("CreateWorker() failed: %v", err) } if capacity := got.GetStatus().GetCapacity(); capacity != nil { - t.Errorf("created worker capacity = %v, want none until its ateom reports", capacity) + t.Errorf("created worker capacity = %v, want none until its worker reports", capacity) } // Capacity is status, so a request cannot bring its own: a Worker only diff --git a/cmd/ateapi/internal/controlapi/workflow.go b/cmd/ateapi/internal/controlapi/workflow.go index ef521fbe52..b0ac809e0f 100644 --- a/cmd/ateapi/internal/controlapi/workflow.go +++ b/cmd/ateapi/internal/controlapi/workflow.go @@ -101,6 +101,10 @@ func logActorState(ctx context.Context, actor *ateapipb.Actor, opName, state str } // ActorWorkflow handles the workflows for actor's resume / suspend operations. +// +// Workflows act for the system, not the user whose request started them, so +// each exported method begins with preview.ContextWithOptIn: the request was +// already validated, and what the workflow writes is the system's own state. type ActorWorkflow struct { store actorWorkflowStore workerCache *workercache.Cache @@ -186,7 +190,8 @@ type actorWorkflowStore interface { AcquireLease(ctx context.Context, key string) (*store.Lease, error) } -// WorkerWorkflow handles the multi-step operations on a Worker. +// WorkerWorkflow handles the multi-step operations on a Worker. Like +// ActorWorkflow, each exported method begins with preview.ContextWithOptIn. // // Its steps reach across the Actor↔Worker binding, which an ActorWorkflow step // does from the other side: releasing the Actor bound to a Worker stays diff --git a/cmd/ateapi/internal/controlapi/workflow_delete.go b/cmd/ateapi/internal/controlapi/workflow_delete.go index c4c7ef0c0c..d0402c9e91 100644 --- a/cmd/ateapi/internal/controlapi/workflow_delete.go +++ b/cmd/ateapi/internal/controlapi/workflow_delete.go @@ -23,6 +23,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -32,6 +33,7 @@ import ( // DeleteActor executes the workflow to delete an actor. Idempotent. func (w *ActorWorkflow) DeleteActor(ctx context.Context, actorRef resources.ActorRef, anyState bool, precondition store.DeletePreconditions) (*ateapipb.Actor, error) { + ctx = preview.ContextWithOptIn(ctx) ctx, lease, err := w.acquireActorLease(ctx, actorRef) if err != nil { return nil, err @@ -143,7 +145,7 @@ func (w *ActorWorkflow) ensureAteletTerminated(ctx context.Context, actorRef res return err } if !hosted { - slog.InfoContext(ctx, "worker is no longer assigned to this actor, skipping ateom workload termination", + slog.InfoContext(ctx, "worker is no longer assigned to this actor, skipping worker workload termination", slog.String("worker", workerName), slog.Any("actor", actorRef)) workerPodUID = "" @@ -156,7 +158,7 @@ func (w *ActorWorkflow) ensureAteletTerminated(ctx context.Context, actorRef res return fmt.Errorf("while connecting to atelet on node %q: %w", nodeName, err) } - client := ateletpb.NewAteomHerderClient(conn) + client := ateletpb.NewAteletClient(conn) var workloadSpec *ateletpb.WorkloadSpec if actorTemplate != nil { @@ -172,19 +174,26 @@ func (w *ActorWorkflow) ensureAteletTerminated(ctx context.Context, actorRef res slog.String("actor", actorRef.Name), slog.String("templateAtespace", actor.GetActorTemplate().GetAtespace()), slog.String("templateName", actor.GetActorTemplate().GetName())) + // Given that external volumes has been in the codebase for a while and + // its preview-ness is solely around the API, and that the impact of + // disabling the gate while in use would be an orphaned volume + // attachment, we are NOT checking the ExternalVolumes gate in the + // delete/suspend path. workloadSpec = &ateletpb.WorkloadSpec{} - for _, vol := range actor.GetStatus().GetExternalVolumes() { - // StorageVolumeId is only populated once the volume is provisioned. - // Skip volumes that were never created (e.g. failed during PENDING state). - if vol.GetStorageVolumeId() != "" { - workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{ - Name: vol.GetName(), - External: &ateletpb.ExternalVolumeSource{ - StorageVolumeId: vol.GetStorageVolumeId(), - VolumeType: vol.GetVolumeType(), - VolumeContext: vol.GetVolumeContext(), - }, - }) + if preview.IsEnabled(preview.GateExternalVolumes) { + for _, vol := range actor.GetStatus().GetExternalVolumes() { + // StorageVolumeId is only populated once the volume is provisioned. + // Skip volumes that were never created (e.g. failed during PENDING state). + if vol.GetStorageVolumeId() != "" { + workloadSpec.Volumes = append(workloadSpec.Volumes, &ateletpb.Volume{ + Name: vol.GetName(), + External: &ateletpb.ExternalVolumeSource{ + StorageVolumeId: vol.GetStorageVolumeId(), + VolumeType: vol.GetVolumeType(), + VolumeContext: vol.GetVolumeContext(), + }, + }) + } } } } @@ -215,6 +224,11 @@ func (w *ActorWorkflow) ensureVolumesDetachedForDelete(ctx context.Context, acto ctx, done := stepSpan(ctx, "DetachVolumesForDelete") defer func() { err = done(err) }() + // Given that external volumes has been in the codebase for a while and + // its preview-ness is solely around the API, and that the impact of + // disabling the gate while in use would be an orphaned volume + // attachment, we are NOT checking the ExternalVolumes gate in the + // delete/suspend path. return detachActorVolumes(ctx, w.pluginRegistry, actor, actorTemplate, "delete") } @@ -354,6 +368,11 @@ func (w *ActorWorkflow) ensureVolumesDeleted(ctx context.Context, actor *ateapip return apierror.FailedPrecondition("DeleteVolumes prerequisite not met for Actor: %s (got: %v, want %s)", actor.GetMetadata().GetName(), st, ateapipb.ActorState_ACTOR_STATE_DELETING) } + // Given that external volumes has been in the codebase for a while and + // its preview-ness is solely around the API, and that the impact of + // disabling the gate while in use would be an orphaned volume + // attachment, we are NOT checking the ExternalVolumes gate in the + // delete/suspend path. if err := deleteActorVolumes(ctx, w.pluginRegistry, actor.GetMetadata().GetUid(), actor.GetStatus().GetExternalVolumes()); err != nil { return apierror.Internal("while deleting actor volumes: %v", err) } diff --git a/cmd/ateapi/internal/controlapi/workflow_delete_test.go b/cmd/ateapi/internal/controlapi/workflow_delete_test.go index 6c1d7f5a0a..67536ae114 100644 --- a/cmd/ateapi/internal/controlapi/workflow_delete_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_delete_test.go @@ -29,6 +29,7 @@ import ( "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" ) func TestDeleteActorWorkflow_ExecutionPaths(t *testing.T) { @@ -445,40 +446,21 @@ func TestDeleteActor_CollectsSnapshotsAfterWorkerDelete(t *testing.T) { objects := objectstoretest.New() actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} - workerName := testWorkerUID("pod-1") + workerName := wireTestAssignment().GetWorker().GetName() actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, ActorTemplate: &ateapipb.ObjectRef{Atespace: "team-a", Name: "sub-tmpl"}, Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_RUNNING, - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: workerName}, - WorkerNamespace: "worker-ns", - WorkerPool: "pool", - WorkerPod: "pod-1", - WorkerPodUid: workerName, - }, + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + WorkerAssignment: wireTestAssignment(), }, }) - if _, err := persistence.CreateWorker(ctx, &ateapipb.Worker{ - Metadata: &ateapipb.ResourceMetadata{Name: workerName}, - WorkerNamespace: "worker-ns", - WorkerPool: "pool", - WorkerPod: "pod-1", - WorkerPodUid: workerName, - Status: &ateapipb.WorkerStatus{}, - }); err != nil { - t.Fatalf("CreateWorker: %v", err) - } - seedAssignment(t, persistence, workerName, &ateapipb.ActorAssignment{ - Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: actorRef.Name}, - ActorUid: actor.GetMetadata().GetUid(), - }) + seedWireWorker(t, ctx, persistence, assignmentOf(actor)) if tt.hasBeenSuspended { previous := mustActorSnapshotURI(t, template, actor, "old") objects.PutSnapshot(t, previous, "manifest.json") - actor = mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapipb.ActorStatus) { + mustUpdateActorStatus(t, ctx, persistence, actor, func(s *ateapipb.ActorStatus) { s.LastAssignedGeneration = 1 s.Snapshots = []*ateapipb.Snapshot{ newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", previous.Name(), previous.String(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED), @@ -486,21 +468,30 @@ func TestDeleteActor_CollectsSnapshotsAfterWorkerDelete(t *testing.T) { }) } - actorWorkflow := NewActorWorkflow(persistence, nil, nil, nil, nil, nil, "", nil, time.Minute, objectstoreplugintest.ControlClient(objects)) - // Suspend the actor as far as it gets: MarkSuspending mints the - // in-progress URI, and the checkpoint writes under it - actor, err := actorWorkflow.ensureMarkedSuspending(ctx, actorRef, actor, template) - if err != nil { - t.Fatalf("ensureMarkedSuspending: %v", err) + actorWorkflow, atelet := newWireCaptureWorkflow(t, persistence) + actorWorkflow.objectStorePlugin = objectstoreplugintest.ControlClient(objects) + // atelet writes part of the snapshot, then the worker's pod goes + // away mid-checkpoint, so the checkpoint fails and the suspend never + // commits. + var fresh resources.SnapshotURI + atelet.onCheckpoint = func(context.Context) error { + uri, err := resources.ParseSnapshotURI(atelet.checkpointRequest().GetSnapshot().GetObject().GetSnapshotUri()) + if err != nil { + t.Errorf("ParseSnapshotURI: %v", err) + return err + } + fresh = uri + objects.PutSnapshot(t, fresh, "manifest.json") + // The worker's pod goes away with the commit still outstanding, so + // the suspend never gets to finish. + if _, err := NewWorkerWorkflow(persistence).DeleteWorker(ctx, workerName, store.DeletePreconditions{}); err != nil { + t.Errorf("DeleteWorker: %v", err) + return err + } + return status.Error(codes.Unavailable, "worker pod is gone") } - freshSnap := findLatestDurableSnapshot(actor.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) - fresh := mustParseSnapshotURI(t, freshSnap.GetDurableSnapshot().GetObject().GetSnapshotUri()) - objects.PutSnapshot(t, fresh, "manifest.json") - - // The worker's pod goes away with the commit still outstanding, so - // the suspend never gets to finish. - if _, err := NewWorkerWorkflow(persistence).DeleteWorker(ctx, workerName, store.DeletePreconditions{}); err != nil { - t.Fatalf("DeleteWorker: %v", err) + if _, err := actorWorkflow.SuspendActor(ctx, actorRef); apierror.Code(err) != codes.Unavailable { + t.Fatalf("SuspendActor err = %v, want Unavailable from the failed checkpoint", err) } // The actor is CRASHED; DeleteActor deletes the in-progress snapshot diff --git a/cmd/ateapi/internal/controlapi/workflow_pause.go b/cmd/ateapi/internal/controlapi/workflow_pause.go index d71edf3158..2701afe4c5 100644 --- a/cmd/ateapi/internal/controlapi/workflow_pause.go +++ b/cmd/ateapi/internal/controlapi/workflow_pause.go @@ -24,6 +24,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -35,6 +36,7 @@ import ( // a re-entered workflow fast-forwards past the steps a previous attempt // completed, deriving progress from the persisted actor alone. func (w *ActorWorkflow) PauseActor(ctx context.Context, actorRef resources.ActorRef) (_ *ateapipb.Actor, err error) { + ctx = preview.ContextWithOptIn(ctx) start := time.Now() var actor *ateapipb.Actor var actorTemplate *ateapipb.ActorTemplate @@ -182,7 +184,7 @@ func (w *ActorWorkflow) ensureAteletPaused(ctx context.Context, actorRef resourc if err != nil { return "", fmt.Errorf("while getting atelet conn for node %q: %w", assignment.GetNodeName(), err) } - client := ateletpb.NewAteomHerderClient(ateletConn) + client := ateletpb.NewAteletClient(ateletConn) workloadSpec, err := workloadSpecFromActorTemplate(actorTemplate, actor, nil) if err != nil { @@ -202,6 +204,7 @@ func (w *ActorWorkflow) ensureAteletPaused(ctx context.Context, actorRef resourc Spec: workloadSpec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: inProgressSnap.GetUuid(), + Generation: inProgressSnap.GetGeneration(), Fidelity: fidelityToAtelet(actorTemplate.GetSnapshotConfig().GetPreferredFidelity()), }, UploadConfig: &ateletpb.UploadConfig{ diff --git a/cmd/ateapi/internal/controlapi/workflow_pause_test.go b/cmd/ateapi/internal/controlapi/workflow_pause_test.go index 052322754b..76be4a5088 100644 --- a/cmd/ateapi/internal/controlapi/workflow_pause_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_pause_test.go @@ -16,6 +16,7 @@ package controlapi import ( "context" + "fmt" "testing" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storetest" @@ -29,50 +30,44 @@ import ( "google.golang.org/protobuf/testing/protocmp" ) -// TestEnsurePausedFinalized_WorkerGone reproduces the scenario where the worker -// pod disappears from the DB during pause finalization, so the node it ran on -// is unknown. +// TestPauseActor_Finalized_WorkerGone reproduces the scenario where the +// worker pod disappears from the DB during pause finalization, so the node it +// ran on is unknown. // // Current behavior: AssignedNode is left empty, and the actor is crashed // instead of left paused, since a local snapshot with an unknown node can -// never be safely resumed. The step fails with FailedPrecondition rather than +// never be safely resumed. The pause fails with FailedPrecondition rather than // reporting the pause as done. -func TestEnsurePausedFinalized_WorkerGone(t *testing.T) { - st, cleanup := storetest.SetupTestStore(t) - defer cleanup() - +func TestPauseActor_Finalized_WorkerGone(t *testing.T) { ctx := context.Background() + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, nil) + w, atelet := newWireCaptureWorkflow(t, st) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} records := crashRecords(t) - actor := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, - WorkerAssignment: &ateapipb.WorkerAssignment{ - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - }, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{ newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local-snap-1", ""), }, }, - } - storetest.MustCreateActor(t, ctx, st, actor) + }) // Intentionally NOT creating the worker in store, simulates worker already gone. - w := &ActorWorkflow{store: st} - if _, err := w.ensurePausedFinalized(ctx, actorRef); apierror.Code(err) != codes.FailedPrecondition { - t.Fatalf("ensurePausedFinalized error = %v, want FailedPrecondition for the actor it crashed", err) + if _, err := w.PauseActor(ctx, actorRef); apierror.Code(err) != codes.FailedPrecondition { + t.Fatalf("PauseActor error = %v, want FailedPrecondition for the actor it crashed", err) } - - got, err := st.GetActor(ctx, actorRef) - if err != nil { - t.Fatalf("GetActor: %v", err) + if checkpoints, _ := atelet.calls(); checkpoints != 1 { + t.Errorf("Checkpoint calls = %d, want 1 before finalize", checkpoints) } + got := mustReadActor(t, ctx, st, actorRef) if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_CRASHED { t.Errorf("state = %v, want CRASHED (node name unknown, cannot resume safely)", got.GetStatus().GetState()) } @@ -99,45 +94,40 @@ func TestEnsurePausedFinalized_WorkerGone(t *testing.T) { } } -// TestEnsurePausedFinalized_AlreadyCrashed verifies that if the actor was -// already crashed out-of-band when ensurePausedFinalized runs with no -// AssignedNode, its existing Crash status and snapshots are preserved, no -// duplicate crash log record is emitted, and the step still fails. -func TestEnsurePausedFinalized_AlreadyCrashed(t *testing.T) { - st, cleanup := storetest.SetupTestStore(t) - defer cleanup() - +// TestPauseActor_Finalized_AlreadyCrashed verifies that if the actor was +// crashed out-of-band while it checkpointed, and has no AssignedNode, the +// pause's finalize preserves the existing Crash status and snapshots, emits no +// duplicate crash log record, and still fails. +func TestPauseActor_Finalized_AlreadyCrashed(t *testing.T) { ctx := context.Background() + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, nil) + w, atelet := newWireCaptureWorkflow(t, st) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} records := crashRecords(t) - originalCrash := newActorCrash(ateattr.OperationPause, "original crash reason") - actor := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_CRASHED, - Crash: originalCrash, - WorkerAssignment: &ateapipb.WorkerAssignment{ - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - }, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{ newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local-snap-1", ""), }, }, - } - storetest.MustCreateActor(t, ctx, st, actor) + }) + originalCrash := newActorCrash(ateattr.OperationPause, "original crash reason") + updateDuringCheckpoint(t, atelet, st, actorRef, func(s *ateapipb.ActorStatus) { + s.State = ateapipb.ActorState_ACTOR_STATE_CRASHED + s.Crash = originalCrash + }) - w := &ActorWorkflow{store: st} - if _, err := w.ensurePausedFinalized(ctx, actorRef); apierror.Code(err) != codes.FailedPrecondition { - t.Fatalf("ensurePausedFinalized error = %v, want FailedPrecondition for a CRASHED actor", err) - } - got, err := st.GetActor(ctx, actorRef) - if err != nil { - t.Fatalf("GetActor: %v", err) + if _, err := w.PauseActor(ctx, actorRef); apierror.Code(err) != codes.FailedPrecondition { + t.Fatalf("PauseActor error = %v, want FailedPrecondition for a CRASHED actor", err) } + got := mustReadActor(t, ctx, st, actorRef) if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_CRASHED { t.Errorf("state = %v, want CRASHED", got.GetStatus().GetState()) } @@ -155,59 +145,36 @@ func TestEnsurePausedFinalized_AlreadyCrashed(t *testing.T) { } } -// TestEnsurePausedFinalized_KeepsCrashFromWorkerDelete verifies that a crash -// the DeleteWorker workflow commits while ensurePausedFinalized is freeing the -// worker is kept: the step's update is pinned to the actor version it checked, -// so it fails Aborted instead of committing SUSPENDED over the crash. -func TestEnsurePausedFinalized_KeepsCrashFromWorkerDelete(t *testing.T) { - st, cleanup := storetest.SetupTestStore(t) - defer cleanup() +// TestPauseActor_Finalized_KeepsCrashFromWorkerDelete verifies that a crash the +// DeleteWorker workflow commits while the pause's finalize is freeing the +// worker is kept: the finalize update is pinned to the actor version it +// checked, so it fails Aborted instead of committing SUSPENDED over the crash. +func TestPauseActor_Finalized_KeepsCrashFromWorkerDelete(t *testing.T) { ctx := context.Background() + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, nil) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} - workerName := testWorkerUID("worker-pod-1") created := storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_RUNNING, - AssignedNode: "node1", - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - }, + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), }, }) - if _, err := st.CreateWorker(ctx, &ateapipb.Worker{ - Metadata: &ateapipb.ResourceMetadata{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - NodeName: "node1", - Status: &ateapipb.WorkerStatus{}, - }); err != nil { - t.Fatalf("CreateWorker: %v", err) - } - seedAssignment(t, st, workerName, &ateapipb.ActorAssignment{ - Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: actorRef.Name}, - ActorUid: created.GetMetadata().GetUid(), - }) - if _, err := (&ActorWorkflow{store: st}).ensureMarkedPausing(ctx, actorRef, created, &ateapipb.ActorTemplate{}); err != nil { - t.Fatalf("ensureMarkedPausing: %v", err) - } + seedWireWorker(t, ctx, st, assignmentOf(created)) - w := &ActorWorkflow{store: workerDeletedDuringReleaseStore{Interface: st}} - _, err := w.ensurePausedFinalized(ctx, actorRef) + w, _ := newWireCaptureWorkflow(t, workerDeletedDuringReleaseStore{Interface: st}) + _, err := w.PauseActor(ctx, actorRef) assertConcurrentCrashKept(t, ctx, st, actorRef, err, "pause failed: "+crashMessageWorkerPodGone) } -// TestEnsurePausedFinalized_RecordsFidelity verifies pause records the -// template's preferred_fidelity on the local Snapshot alongside Locality and -// leaves DurableSnapshot unset until the actor is suspended. -func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { +// TestPauseActor_Finalized_RecordsFidelity verifies pause records the template's +// preferred_fidelity on the local Snapshot alongside Locality and leaves +// DurableSnapshot unset until the actor is suspended. +func TestPauseActor_Finalized_RecordsFidelity(t *testing.T) { tests := []struct { name string preferred ateapipb.SnapshotFidelity @@ -216,17 +183,18 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { {"VOLUMES", ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES}, {"ROOTFS", ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS}, {"MEMORY", ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, - {"UNSPECIFIED recorded as UNSPECIFIED", ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED}, + // The store defaults an unset preferred_fidelity to MEMORY. + {"UNSPECIFIED defaults to MEMORY", ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - st, cleanup := storetest.SetupTestStore(t) - defer cleanup() ctx := context.Background() + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, &ateapipb.SnapshotConfig{PreferredFidelity: tc.preferred}) + w, _ := newWireCaptureWorkflow(t, st) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} - workerName := testWorkerUID("worker-pod-1") wantRuntime := &ateapipb.SandboxRuntime{ SandboxClass: "gvisor", Version: &ateapipb.VersionedSandboxCompat{ @@ -236,49 +204,22 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { }, }, } + assignment := wireTestAssignment() + assignment.SandboxRuntime = wantRuntime created := storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_RUNNING, - AssignedNode: "node1", - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - SandboxRuntime: wantRuntime, - }, + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + AssignedNode: "node-1", + WorkerAssignment: assignment, }, }) - if _, err := st.CreateWorker(ctx, &ateapipb.Worker{ - Metadata: &ateapipb.ResourceMetadata{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - NodeName: "node1", - SandboxClass: "gvisor", - }); err != nil { - t.Fatalf("CreateWorker: %v", err) - } - seedAssignment(t, st, workerName, &ateapipb.ActorAssignment{ - Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: actorRef.Name}, - ActorUid: created.GetMetadata().GetUid(), - }) + seedWireWorker(t, ctx, st, assignmentOf(created)) - w := &ActorWorkflow{store: st} - tmpl := &ateapipb.ActorTemplate{ - SnapshotConfig: &ateapipb.SnapshotConfig{ - PreferredFidelity: tc.preferred, - }, - } - if _, err := w.ensureMarkedPausing(ctx, actorRef, created, tmpl); err != nil { - t.Fatalf("ensureMarkedPausing: %v", err) - } - got, err := w.ensurePausedFinalized(ctx, actorRef) + got, err := w.PauseActor(ctx, actorRef) if err != nil { - t.Fatalf("ensurePausedFinalized: %v", err) + t.Fatalf("PauseActor: %v", err) } if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { @@ -294,71 +235,59 @@ func TestEnsurePausedFinalized_RecordsFidelity(t *testing.T) { if localSnap.GetDurableSnapshot() != nil { t.Errorf("local DurableSnapshot = %v, want nil", localSnap.GetDurableSnapshot()) } - if localSnap.GetLocality() != "node1" { - t.Errorf("local Snapshot.Locality = %q, want %q", localSnap.GetLocality(), "node1") + if localSnap.GetLocality() != "node-1" { + t.Errorf("local Snapshot.Locality = %q, want %q", localSnap.GetLocality(), "node-1") } if diff := cmp.Diff(wantRuntime, localSnap.GetSandboxRuntime(), protocmp.Transform()); diff != "" { t.Errorf("Snapshot.SandboxRuntime mismatch (-want +got):\n%s", diff) } - if got.GetStatus().GetAssignedNode() != "node1" { - t.Errorf("AssignedNode = %q, want %q", got.GetStatus().GetAssignedNode(), "node1") + if got.GetStatus().GetAssignedNode() != "node-1" { + t.Errorf("AssignedNode = %q, want %q", got.GetStatus().GetAssignedNode(), "node-1") + } + if got.GetStatus().GetWorkerAssignment() != nil { + t.Errorf("WorkerAssignment = %v, want nil", got.GetStatus().GetWorkerAssignment()) + } + if a := firstAssignment(t, st, wireTestAssignment().GetWorker().GetName()); a != nil { + t.Errorf("worker assignment = %v, want released", a) } }) } } -// TestEnsurePausedFinalized_MissingLocalSnapshotClearsAssignment verifies that -// if the latest generation snapshot entry is missing when ensurePausedFinalized -// runs, the actor transitions to CRASHED and clears WorkerAssignment after -// releasing the worker rather than getting stuck SUSPENDING, and the step -// fails. -func TestEnsurePausedFinalized_MissingLocalSnapshotClearsAssignment(t *testing.T) { - st, cleanup := storetest.SetupTestStore(t) - defer cleanup() +// TestPauseActor_Finalized_MissingLocalSnapshotClearsAssignment verifies that if the +// latest generation snapshot entry is missing when the pause finalizes, the +// actor transitions to CRASHED and clears WorkerAssignment after releasing +// the worker rather than getting stuck SUSPENDING, and the pause fails. +func TestPauseActor_Finalized_MissingLocalSnapshotClearsAssignment(t *testing.T) { ctx := context.Background() + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, nil) + w, atelet := newWireCaptureWorkflow(t, st) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} records := crashRecords(t) - workerName := testWorkerUID("worker-pod-1") created := storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, - AssignedNode: "node1", - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - }, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, + Snapshots: []*ateapipb.Snapshot{ + newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "local-snap-1", ""), + }, }, }) - if _, err := st.CreateWorker(ctx, &ateapipb.Worker{ - Metadata: &ateapipb.ResourceMetadata{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - NodeName: "node1", - Status: &ateapipb.WorkerStatus{}, - }); err != nil { - t.Fatalf("CreateWorker: %v", err) - } - seedAssignment(t, st, workerName, &ateapipb.ActorAssignment{ - Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: actorRef.Name}, - ActorUid: created.GetMetadata().GetUid(), + seedWireWorker(t, ctx, st, assignmentOf(created)) + updateDuringCheckpoint(t, atelet, st, actorRef, func(s *ateapipb.ActorStatus) { + s.Snapshots = nil }) - w := &ActorWorkflow{store: st} - if _, err := w.ensurePausedFinalized(ctx, actorRef); apierror.Code(err) != codes.FailedPrecondition { - t.Fatalf("ensurePausedFinalized error = %v, want FailedPrecondition for the actor it crashed", err) - } - got, err := st.GetActor(ctx, actorRef) - if err != nil { - t.Fatalf("GetActor: %v", err) + if _, err := w.PauseActor(ctx, actorRef); apierror.Code(err) != codes.FailedPrecondition { + t.Fatalf("PauseActor error = %v, want FailedPrecondition for the actor it crashed", err) } + got := mustReadActor(t, ctx, st, actorRef) if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_CRASHED { t.Errorf("state = %v, want CRASHED", got.GetStatus().GetState()) } @@ -388,8 +317,8 @@ func TestPauseActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { wantState ateapipb.ActorState }{ { - // Pausing a SUSPENDED actor is rejected by MarkPausingStep's - // CheckPrerequisite and the actor's state is left untouched. + // Pausing a SUSPENDED actor is rejected by the pause's state gate + // and the actor's state is left untouched. name: "not running rejected", seedState: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, wantErr: true, @@ -438,84 +367,98 @@ func TestPauseActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { } } -// TestEnsureMarkedPausing_StateMatrix verifies the pause edge's state gating -// against every actor state: RUNNING takes the edge, everything else is -// rejected with FailedPrecondition. SUSPENDED is rejected whatever its -// snapshots: the orchestrator early-returns before this step for a paused -// actor. SUSPENDING without a pending node-local snapshot is a suspend in -// progress, which owns the actor; a pause in progress is covered by -// TestEnsureMarkedPausing_ReentrySkipsPauseInProgress. -func TestEnsureMarkedPausing_StateMatrix(t *testing.T) { - allowed := map[ateapipb.ActorState]bool{ - ateapipb.ActorState_ACTOR_STATE_RUNNING: true, - } +// TestPauseActor_MarkedPausing_StateMatrix verifies the pause edge's state gating against +// every actor state: RUNNING takes the edge, everything else is rejected with +// FailedPrecondition and left untouched. SUSPENDED is rejected whatever its +// snapshots short of the paused shape, which fast-forwards (see +// TestPauseActorWorkflow_RejectedAndIdempotentPaths). SUSPENDING without a +// pending node-local snapshot is a suspend in progress, which owns the actor; +// a pause in progress is covered by TestPauseActor_MarkedPausing_ReentrySkipsPauseInProgress. +func TestPauseActor_MarkedPausing_StateMatrix(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, nil) + w := newMarkOnlyWorkflow(st) for _, seedState := range allActorStates { - ctx := context.Background() - persistence := newTestPersistence(t) - w := &ActorWorkflow{store: persistence} - - actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} - - actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, - Status: &ateapipb.ActorStatus{State: seedState}, + actorRef := resources.ActorRef{Atespace: "team-a", Name: fmt.Sprintf("id%d", seedState)} + seedWorkflowActor(t, ctx, st, actorRef, "ns", "tmpl1", seedState, func(a *ateapipb.Actor) { + a.Status.WorkerAssignment = wireTestAssignment() }) - marked, err := w.ensureMarkedPausing(ctx, actorRef, actor, &ateapipb.ActorTemplate{}) - assertPrerequisiteResult(t, seedState, err, allowed[seedState]) - if err == nil && marked.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { - t.Errorf("state %v: ensureMarkedPausing returned actor in %v, want SUSPENDING", seedState, marked.GetStatus().GetState()) + _, err := w.PauseActor(ctx, actorRef) + stored := mustReadActor(t, ctx, st, actorRef) + if seedState == ateapipb.ActorState_ACTOR_STATE_RUNNING { + assertStoppedAtDial(t, err) + if got := stored.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { + t.Errorf("state %v: stored state = %v, want SUSPENDING", seedState, got) + } + if !latestSnapshotIsPendingLocal(stored.GetStatus()) { + t.Errorf("state %v: snapshots = %v, want a pending local snapshot at the latest generation", seedState, stored.GetStatus().GetSnapshots()) + } + continue + } + if got := apierror.Code(err); got != codes.FailedPrecondition { + t.Errorf("state %v: code = %v, want FailedPrecondition (err: %v)", seedState, got, err) + } + if got := stored.GetStatus().GetState(); got != seedState { + t.Errorf("state %v: stored state = %v, want it untouched", seedState, got) } } } -// TestEnsureMarkedPausing_ReentrySkipsPauseInProgress verifies a re-entered -// pause recognizes its own mark, a SUSPENDING actor whose latest generation -// is a pending node-local snapshot, and skips without minting another -// snapshot, while a running-origin suspend in progress (pending durable -// snapshot) is rejected. -func TestEnsureMarkedPausing_ReentrySkipsPauseInProgress(t *testing.T) { +// TestPauseActor_MarkedPausing_ReentrySkipsPauseInProgress verifies a re-entered pause +// recognizes its own mark, a SUSPENDING actor whose latest generation is a +// pending node-local snapshot, and checkpoints under that snapshot without +// minting another, while a running-origin suspend in progress (pending +// durable snapshot) is rejected before atelet is called. +func TestPauseActor_MarkedPausing_ReentrySkipsPauseInProgress(t *testing.T) { ctx := context.Background() - persistence := newTestPersistence(t) - w := &ActorWorkflow{store: persistence} + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, nil) + w, atelet := newWireCaptureWorkflow(t, st) actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} - pausing := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, - Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING}, - } - withPauseInProgress()(pausing) - created := storetest.MustCreateActor(t, ctx, persistence, pausing) - marked, err := w.ensureMarkedPausing(ctx, actorRef, created, &ateapipb.ActorTemplate{}) + seedWorkflowActor(t, ctx, st, actorRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDING, withPauseInProgress(), func(a *ateapipb.Actor) { + a.Status.AssignedNode = "node-1" + a.Status.WorkerAssignment = wireTestAssignment() + }) + paused, err := w.PauseActor(ctx, actorRef) if err != nil { - t.Fatalf("ensureMarkedPausing on a pause in progress: %v", err) + t.Fatalf("PauseActor on a pause in progress: %v", err) + } + if got := atelet.checkpointRequest().GetSnapshot().GetSnapshotUuid(); got != "pause-snap" { + t.Errorf("CheckpointRequest.Snapshot.SnapshotUuid = %q, want the persisted %q", got, "pause-snap") + } + if got := paused.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { + t.Errorf("state = %v, want SUSPENDED", got) } - if got := marked.GetStatus().GetLastAssignedGeneration(); got != 1 { + if got := paused.GetStatus().GetLastAssignedGeneration(); got != 1 { t.Errorf("last_assigned_generation = %d, want 1 (no new snapshot minted on re-entry)", got) } - if got := len(marked.GetStatus().GetSnapshots()); got != 1 { + if got := len(paused.GetStatus().GetSnapshots()); got != 1 { t.Errorf("len(snapshots) = %d, want 1", got) } - suspending := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: "id2"}, - Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, - LastAssignedGeneration: 1, - Snapshots: []*ateapipb.Snapshot{ - newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap", "gs://snapshots/s", ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), - }, - }, - } - created = storetest.MustCreateActor(t, ctx, persistence, suspending) - _, err = w.ensureMarkedPausing(ctx, resources.ActorRef{Atespace: actorRef.Atespace, Name: "id2"}, created, &ateapipb.ActorTemplate{}) + suspendingRef := resources.ActorRef{Atespace: actorRef.Atespace, Name: "id2"} + seedWorkflowActor(t, ctx, st, suspendingRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDING, func(a *ateapipb.Actor) { + a.Status.AssignedNode = "node-1" + a.Status.WorkerAssignment = wireTestAssignment() + a.Status.LastAssignedGeneration = 1 + a.Status.Snapshots = []*ateapipb.Snapshot{ + newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap", "gs://snapshots/s", ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + } + }) + _, err = w.PauseActor(ctx, suspendingRef) if got := apierror.Code(err); got != codes.FailedPrecondition { - t.Errorf("ensureMarkedPausing on a suspend in progress: code = %v, want FailedPrecondition (err: %v)", got, err) + t.Errorf("PauseActor on a suspend in progress: code = %v, want FailedPrecondition (err: %v)", got, err) + } + if checkpoints, _ := atelet.calls(); checkpoints != 1 { + t.Errorf("Checkpoint calls = %d, want 1 (none for the suspend in progress)", checkpoints) } } -func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { +func TestPauseActor_AteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { tests := []struct { name string prevSnapshot string @@ -534,6 +477,7 @@ func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { t.Run(tt.name, func(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, nil) var snapshots []*ateapipb.Snapshot if tt.prevSnapshot != "" { @@ -542,7 +486,8 @@ func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { snapshots = append(snapshots, newLocalSnapshot(2, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "actor-1-never-written", "")) actor := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, WorkerAssignment: &ateapipb.WorkerAssignment{ @@ -555,17 +500,13 @@ func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { Snapshots: snapshots, }, } - created := storetest.MustCreateActor(t, ctx, persistence, actor) + storetest.MustCreateActor(t, ctx, persistence, actor) - w := &ActorWorkflow{store: persistence, dialer: newDanglingDialer()} - if _, err := w.ensureAteletPaused(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}, created, &ateapipb.ActorTemplate{}); err == nil { - t.Fatal("ensureAteletPaused: want error when atelet is unreachable, got nil") - } + w := newMarkOnlyWorkflow(persistence) + _, err := w.PauseActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}) + assertStoppedAtDial(t, err) - stored, err := persistence.GetActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}) - if err != nil { - t.Fatalf("GetActor: %v", err) - } + stored := mustReadActor(t, ctx, persistence, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}) if stored.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { t.Errorf("state = %v, want unchanged SUSPENDING", stored.GetStatus().GetState()) } @@ -581,9 +522,9 @@ func TestEnsureAteletPaused_DialFailureLeavesActorRetryable(t *testing.T) { } // TestPauseActor_CrashesWhenPausingActorMissingWorkerPod verifies that a -// pausing actor with no worker pod recorded is moved to CRASHED by -// ensureAteletPaused's corrupted-assignment check and the pause fails with -// FailedPrecondition. +// pausing actor with no worker pod recorded is moved to CRASHED by the +// corrupted-assignment check before atelet is called, and the pause fails +// with FailedPrecondition. func TestPauseActor_CrashesWhenPausingActorMissingWorkerPod(t *testing.T) { ctx := context.Background() st, cleanup := storetest.SetupTestStore(t) @@ -609,43 +550,54 @@ func TestPauseActor_CrashesWhenPausingActorMissingWorkerPod(t *testing.T) { } } -// TestEnsureMarkedPausing_GoldenAtespaceRejected verifies golden actors -// cannot be paused: by design they can only be suspended (committed). -func TestEnsureMarkedPausing_GoldenAtespaceRejected(t *testing.T) { - st, cleanup := storetest.SetupTestStore(t) - defer cleanup() - w := newTestActorWorkflow(t, st, "ns", "tmpl1") - - _, err := w.ensureMarkedPausing(context.Background(), - resources.ActorRef{Atespace: resources.GoldenActorAtespace, Name: "golden-1"}, - &ateapipb.Actor{Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RUNNING}}, - &ateapipb.ActorTemplate{}) +// TestPauseActor_MarkedPausing_GoldenAtespaceRejected verifies golden actors cannot be +// paused: by design they can only be suspended (committed). +func TestPauseActor_MarkedPausing_GoldenAtespaceRejected(t *testing.T) { + ctx := context.Background() + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, nil) + w := newMarkOnlyWorkflow(st) + + actorRef := resources.ActorRef{Atespace: resources.GoldenActorAtespace, Name: "golden-1"} + seedWorkflowActor(t, ctx, st, actorRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_RUNNING, func(a *ateapipb.Actor) { + a.Status.AssignedNode = "node-1" + a.Status.WorkerAssignment = wireTestAssignment() + }) + _, err := w.PauseActor(ctx, actorRef) if got := apierror.Code(err); got != codes.FailedPrecondition { t.Fatalf("apierror.Code = %v (err %v), want FailedPrecondition", got, err) } + if got := mustReadActor(t, ctx, st, actorRef).GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_RUNNING { + t.Errorf("stored state = %v, want RUNNING untouched", got) + } } -// TestEnsureAteletPaused_ForwardsPreferredFidelity verifies that local pause -// checkpoints forward the template's PreferredFidelity to atelet. -func TestEnsureAteletPaused_ForwardsPreferredFidelity(t *testing.T) { +// TestPauseActor_AteletPaused_ForwardsPreferredFidelity verifies that local pause +// checkpoints forward the template's PreferredFidelity to atelet, and label the +// pause's lifecycle-op metric with it. +func TestPauseActor_AteletPaused_ForwardsPreferredFidelity(t *testing.T) { tests := []struct { preferred ateapipb.SnapshotFidelity want ateletpb.SnapshotFidelity }{ {ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, {ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES}, - {ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED}, + // The store defaults an unset preferred_fidelity to MEMORY. + {ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, } for _, tc := range tests { t.Run(tc.preferred.String(), func(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{PreferredFidelity: tc.preferred}) w, atelet := newWireCaptureWorkflow(t, persistence) + inst, reader := newTestInstruments(t) + w.instruments = inst actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} - created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, - ActorTemplate: &ateapipb.ObjectRef{Atespace: "ns", Name: "tmpl1"}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: "node-1", @@ -656,19 +608,9 @@ func TestEnsureAteletPaused_ForwardsPreferredFidelity(t *testing.T) { }, }, }) - tmpl := &ateapipb.ActorTemplate{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "ns", Name: "tmpl1"}, - SnapshotConfig: &ateapipb.SnapshotConfig{ - PreferredFidelity: tc.preferred, - }, - } - wireFidelity, err := w.ensureAteletPaused(ctx, actorRef, created, tmpl) - if err != nil { - t.Fatalf("ensureAteletPaused: %v", err) - } - if want := ateattr.SnapshotFidelityValue(tc.want); wireFidelity != want { - t.Errorf("wireFidelity = %q, want %q", wireFidelity, want) + if _, err := w.PauseActor(ctx, actorRef); err != nil { + t.Fatalf("PauseActor: %v", err) } req := atelet.checkpointRequest() if req == nil { @@ -680,9 +622,17 @@ func TestEnsureAteletPaused_ForwardsPreferredFidelity(t *testing.T) { if got := req.GetSnapshot().GetSnapshotUuid(); got != "local-snap-1" { t.Errorf("CheckpointRequest.Snapshot.SnapshotUuid = %q, want %q", got, "local-snap-1") } + if got := req.GetSnapshot().GetGeneration(); got != 1 { + t.Errorf("CheckpointRequest.Snapshot.Generation = %d, want 1, the in-progress snapshot's", got) + } + if got := req.GetSnapshot().GetObject(); got != nil { + t.Errorf("CheckpointRequest.Snapshot.Object = %v, want nil for a node-local checkpoint", got) + } if got := req.GetUploadConfig().GetMode(); got != ateletpb.UploadMode_UPLOAD_MODE_DEFERRED { t.Errorf("CheckpointRequest.UploadConfig.Mode = %v, want UPLOAD_MODE_DEFERRED", got) } + // The fidelity sent to atelet labels the pause's lifecycle-op metric. + assertLifecycleFidelity(t, reader, ateattr.SnapshotFidelityValue(tc.want)) }) } } diff --git a/cmd/ateapi/internal/controlapi/workflow_preview_test.go b/cmd/ateapi/internal/controlapi/workflow_preview_test.go new file mode 100644 index 0000000000..8001b2c9f1 --- /dev/null +++ b/cmd/ateapi/internal/controlapi/workflow_preview_test.go @@ -0,0 +1,93 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package controlapi + +import ( + "go/ast" + "go/parser" + "go/token" + "path/filepath" + "strings" + "testing" +) + +// TestWorkflowsOptInToPreview checks that every exported workflow method +// begins with preview.ContextWithOptIn, so a request's preview opt-out does +// not limit the system state the workflow writes. +func TestWorkflowsOptInToPreview(t *testing.T) { + files, err := filepath.Glob("*.go") + if err != nil { + t.Fatal(err) + } + fset := token.NewFileSet() + found := 0 + for _, file := range files { + if strings.HasSuffix(file, "_test.go") { + continue + } + f, err := parser.ParseFile(fset, file, nil, 0) + if err != nil { + t.Fatal(err) + } + for _, decl := range f.Decls { + fn, ok := decl.(*ast.FuncDecl) + if !ok || fn.Recv == nil || !fn.Name.IsExported() || !isWorkflowReceiver(fn.Recv) { + continue + } + found++ + if !startsWithOptIn(fn) { + t.Errorf("%s: %s must begin with `ctx = preview.ContextWithOptIn(ctx)`", fset.Position(fn.Pos()), fn.Name.Name) + } + } + } + if found == 0 { + t.Fatal("found no workflow methods; is the test looking in the right place?") + } +} + +func isWorkflowReceiver(recv *ast.FieldList) bool { + star, ok := recv.List[0].Type.(*ast.StarExpr) + if !ok { + return false + } + id, ok := star.X.(*ast.Ident) + return ok && (id.Name == "ActorWorkflow" || id.Name == "WorkerWorkflow") +} + +func startsWithOptIn(fn *ast.FuncDecl) bool { + if fn.Body == nil || len(fn.Body.List) == 0 { + return false + } + assign, ok := fn.Body.List[0].(*ast.AssignStmt) + if !ok || assign.Tok != token.ASSIGN || len(assign.Lhs) != 1 || len(assign.Rhs) != 1 { + return false + } + if lhs, ok := assign.Lhs[0].(*ast.Ident); !ok || lhs.Name != "ctx" { + return false + } + call, ok := assign.Rhs[0].(*ast.CallExpr) + if !ok || len(call.Args) != 1 { + return false + } + sel, ok := call.Fun.(*ast.SelectorExpr) + if !ok || sel.Sel.Name != "ContextWithOptIn" { + return false + } + if pkg, ok := sel.X.(*ast.Ident); !ok || pkg.Name != "preview" { + return false + } + arg, ok := call.Args[0].(*ast.Ident) + return ok && arg.Name == "ctx" +} diff --git a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments.go b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments.go index 14a8919846..6116c60591 100644 --- a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments.go +++ b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments.go @@ -20,6 +20,7 @@ import ( "fmt" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) @@ -31,13 +32,14 @@ func needsAssignmentReconcile(worker *ateapipb.Worker) bool { } // ReconcileAssignments brings a Worker's status.observed_epoch up to its epoch. -// A raised epoch means its ateom restarted, taking with it the sandboxes of the +// A raised epoch means its worker restarted, taking with it the sandboxes of the // Actors placed during an earlier epoch, so those Actors are crashed and their // assignments released before observed_epoch records it. A failure leaves // observed_epoch where it was, so the next pass redoes the release. // // Nothing to do, including a Worker that is gone, is success. func (w *WorkerWorkflow) ReconcileAssignments(ctx context.Context, name string) (err error) { + ctx = preview.ContextWithOptIn(ctx) ctx, done := stepSpan(ctx, "ReconcileAssignments") defer func() { err = done(err) }() @@ -195,7 +197,7 @@ func (w *WorkerWorkflow) releaseEarlierAssignment(ctx context.Context, worker *a return nil } if actor.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { - if err := w.crashBoundActor(ctx, worker, actorRef, actor, "Releasing actor from a worker whose ateom restarted", crashMessageAteomRestarted); err != nil { + if err := w.crashBoundActor(ctx, worker, actorRef, actor, "Releasing actor from a restarted worker", crashMessageWorkerRestarted); err != nil { return err } } diff --git a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go index 1cf4652f2e..f70e5f76ea 100644 --- a/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_reconcile_assignments_test.go @@ -49,7 +49,7 @@ func seedEpochWorker(t *testing.T, ctx context.Context, persistence store.Interf } // raiseEpoch sends the UpdateWorker the syncer sends when the worker pod's -// ateom restarts. +// worker restarts. func raiseEpoch(t *testing.T, ctx context.Context, svc *RPCService, persistence store.Interface, epoch int64) (*ateapipb.Worker, error) { t.Helper() return svc.UpdateWorker(ctx, &ateapipb.UpdateWorkerRequest{ @@ -199,10 +199,10 @@ func TestReconcileAssignments_CrashesEarlierActors(t *testing.T) { } got := mustGetActor(t, ctx, persistence) if got.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_CRASHED { - t.Errorf("actor state = %v, want CRASHED: its sandbox died with the ateom", got.GetStatus().GetState()) + t.Errorf("actor state = %v, want CRASHED: its sandbox died with the worker", got.GetStatus().GetState()) } - if msg := got.GetStatus().GetCrash().GetMessage(); msg != crashMessageAteomRestarted { - t.Errorf("crash message = %q, want %q", msg, crashMessageAteomRestarted) + if msg := got.GetStatus().GetCrash().GetMessage(); msg != crashMessageWorkerRestarted { + t.Errorf("crash message = %q, want %q", msg, crashMessageWorkerRestarted) } if got.GetStatus().GetWorkerAssignment() != nil { t.Errorf("actor worker assignment = %v, want it cleared", got.GetStatus().GetWorkerAssignment()) @@ -216,7 +216,7 @@ func TestReconcileAssignments_CrashesEarlierActors(t *testing.T) { } // An Actor placed after the raise was stamped with the new epoch, so it runs on -// the restarted ateom and is kept. +// the restarted worker and is kept. func TestReconcileAssignments_KeepsActorsPlacedAfterRaise(t *testing.T) { ctx := context.Background() svc, persistence := newWorkerAPIService(t) @@ -430,7 +430,7 @@ func (s *hookedStore) UpdateWorker(ctx context.Context, name string, preconditio } // An Actor bound again in the current epoch after the pass listed its earlier -// assignment runs on the restarted ateom, so it is kept. +// assignment runs on the restarted worker, so it is kept. func TestReconcileAssignments_KeepsActorReboundSinceList(t *testing.T) { ctx := context.Background() svc, persistence := newWorkerAPIService(t) diff --git a/cmd/ateapi/internal/controlapi/workflow_resume.go b/cmd/ateapi/internal/controlapi/workflow_resume.go index e8d1ef990b..cb82726ec9 100644 --- a/cmd/ateapi/internal/controlapi/workflow_resume.go +++ b/cmd/ateapi/internal/controlapi/workflow_resume.go @@ -27,6 +27,7 @@ import ( "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/hardware" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/volume" @@ -41,6 +42,7 @@ import ( // and passed by value to the restore step — never mutated after resolution. type resumeSnapshotSource struct { SnapshotUUID string + Generation int32 // SnapshotURI is the storage location of the durable snapshot to restore // from: the actor's latest snapshot, including a tag borrowed at creation. // Zero means cold boot from the spec (unless the actor holds a local @@ -68,6 +70,7 @@ type restoreTelemetry struct { // a re-entered workflow fast-forwards past the steps a previous attempt // completed, deriving progress from the persisted actor alone. func (w *ActorWorkflow) ResumeActor(ctx context.Context, actorRef resources.ActorRef) (_ *ateapipb.Actor, resumed bool, err error) { + ctx = preview.ContextWithOptIn(ctx) start := time.Now() var actor *ateapipb.Actor var actorTemplate *ateapipb.ActorTemplate @@ -185,6 +188,7 @@ func (w *ActorWorkflow) loadActorForResume(ctx context.Context, actorRef resourc return nil, nil, src, apierror.DataLoss("Actor %s external snapshot: %v", actorRef, err) } src.SnapshotUUID = snap.GetUuid() + src.Generation = snap.GetGeneration() src.Fidelity = snap.GetFidelity() capturedUnder := snap.GetActorTemplateUid() src.TemplateReplaced = capturedUnder != "" && capturedUnder != actorTemplate.GetMetadata().GetUid() @@ -386,7 +390,7 @@ func (w *ActorWorkflow) validateAssignedWorker(ctx context.Context, actorRef res } return nil, apierror.Aborted("actor %s crashed", actorRef.String()) } - // An unavailable worker may come back with its Actors intact. If its ateom + // An unavailable worker may come back with its Actors intact. If its worker // restarted instead, the raised epoch has this Actor crashed for it. if worker.GetStatus().GetState() == ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE { return nil, apierror.Unavailable("worker %s assigned to actor %s is unavailable", worker.GetMetadata().GetName(), actorRef) @@ -617,7 +621,7 @@ func workerAssignmentFrom(w *ateapipb.Worker, snapRuntime *ateapipb.SandboxRunti // with a runtime that did not produce it. Resolve the assets for the // matched version once SandboxConfig can name them, or treat a // restorable-only match as incompatible until it can. Latent for now: - // ateom reports no restorable_runtimes (internal/ateom/register.go). + // worker reports no restorable_runtimes (internal/ateworker/register.go). SandboxRuntime: proto.CloneOf(hardware.RuntimeFor(w.GetStatus(), snapRuntime)), } } @@ -722,6 +726,10 @@ func (w *ActorWorkflow) ensureVolumesAttached(ctx context.Context, actor *ateapi ctx, done := stepSpan(ctx, "AttachVolumes") defer func() { err = done(err) }() + if len(actor.GetStatus().GetExternalVolumes()) > 0 && !preview.IsEnabled(preview.GateExternalVolumes) { + return nil, fmt.Errorf("external volumes are not supported unless the ExternalVolumes preview gate is enabled") + } + node := worker.GetNodeName() if node == "" { return nil, fmt.Errorf("assigned worker has no node name") @@ -762,7 +770,7 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou if err != nil { return tele, err } - client := ateletpb.NewAteomHerderClient(ateletConn) + client := ateletpb.NewAteletClient(ateletConn) workloadSpec, err := workloadSpecFromActorTemplate(actorTemplate, actor, volumePublishContexts) if err != nil { @@ -801,6 +809,7 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou Spec: workloadSpec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: localSnap.GetUuid(), + Generation: localSnap.GetGeneration(), Fidelity: fidelity, }, SandboxAssets: sandboxAssets, @@ -835,6 +844,7 @@ func (w *ActorWorkflow) ensureAteletRestored(ctx context.Context, actorRef resou Spec: workloadSpec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: src.SnapshotUUID, + Generation: src.Generation, Fidelity: scope, Object: &ateletpb.ObjectStorage{ SnapshotUri: src.SnapshotURI.String(), diff --git a/cmd/ateapi/internal/controlapi/workflow_resume_test.go b/cmd/ateapi/internal/controlapi/workflow_resume_test.go index 236e842ade..3b2bdf7a7b 100644 --- a/cmd/ateapi/internal/controlapi/workflow_resume_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_resume_test.go @@ -30,6 +30,7 @@ import ( "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/installdefaults" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/volume" @@ -1047,15 +1048,23 @@ func TestLoadActorForResume_RunningActorShortCircuits(t *testing.T) { } } -// capturingAtelet records the last Restore and Run request it receives, so a -// test can assert on the exact wire request the resume workflow sends. +// capturingAtelet records the last request of each RPC it receives, so a test +// can assert on the exact wire request a workflow sends. type capturingAtelet struct { - ateletpb.UnimplementedAteomHerderServer + ateletpb.UnimplementedAteletServer - mu sync.Mutex - restore *ateletpb.RestoreRequest - run *ateletpb.RunRequest - checkpoint *ateletpb.CheckpointRequest + mu sync.Mutex + restore *ateletpb.RestoreRequest + run *ateletpb.RunRequest + checkpoint *ateletpb.CheckpointRequest + promote *ateletpb.PromoteSnapshotRequest + checkpoints int + promotes int + + // onCheckpoint, when set, runs before Checkpoint returns, standing in for + // whatever else happens while the workload checkpoints. Its error fails + // the RPC. + onCheckpoint func(ctx context.Context) error } func (f *capturingAtelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (*ateletpb.RestoreResponse, error) { @@ -1074,11 +1083,33 @@ func (f *capturingAtelet) Run(ctx context.Context, req *ateletpb.RunRequest) (*a func (f *capturingAtelet) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRequest) (*ateletpb.CheckpointResponse, error) { f.mu.Lock() - defer f.mu.Unlock() f.checkpoint = proto.Clone(req).(*ateletpb.CheckpointRequest) + f.checkpoints++ + hook := f.onCheckpoint + f.mu.Unlock() + if hook != nil { + if err := hook(ctx); err != nil { + return nil, err + } + } return &ateletpb.CheckpointResponse{}, nil } +func (f *capturingAtelet) PromoteSnapshot(ctx context.Context, req *ateletpb.PromoteSnapshotRequest) (*ateletpb.PromoteSnapshotResponse, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.promote = proto.Clone(req).(*ateletpb.PromoteSnapshotRequest) + f.promotes++ + return &ateletpb.PromoteSnapshotResponse{}, nil +} + +// calls returns how many Checkpoint and PromoteSnapshot RPCs were received. +func (f *capturingAtelet) calls() (checkpoints, promotes int) { + f.mu.Lock() + defer f.mu.Unlock() + return f.checkpoints, f.promotes +} + func (f *capturingAtelet) checkpointRequest() *ateletpb.CheckpointRequest { f.mu.Lock() defer f.mu.Unlock() @@ -1088,6 +1119,15 @@ func (f *capturingAtelet) checkpointRequest() *ateletpb.CheckpointRequest { return proto.Clone(f.checkpoint).(*ateletpb.CheckpointRequest) } +func (f *capturingAtelet) promoteRequest() *ateletpb.PromoteSnapshotRequest { + f.mu.Lock() + defer f.mu.Unlock() + if f.promote == nil { + return nil + } + return proto.Clone(f.promote).(*ateletpb.PromoteSnapshotRequest) +} + // requests returns the recorded Restore and Run requests, nil for an RPC that // was never called. func (f *capturingAtelet) requests() (*ateletpb.RestoreRequest, *ateletpb.RunRequest) { @@ -1126,7 +1166,7 @@ func newWireCaptureWorkflow(t *testing.T, persistence store.Interface) (*ActorWo fake := &capturingAtelet{} srv := grpc.NewServer() - ateletpb.RegisterAteomHerderServer(srv, fake) + ateletpb.RegisterAteletServer(srv, fake) lis := bufconn.Listen(1 << 20) go func() { if err := srv.Serve(lis); err != nil { @@ -1167,7 +1207,7 @@ func newWireCaptureWorkflow(t *testing.T, persistence store.Interface) (*ActorWo }, }}) - return &ActorWorkflow{store: persistence, dialer: dialer, sandboxConfigLister: lister}, fake + return &ActorWorkflow{store: persistence, dialer: dialer, sandboxConfigLister: lister, workflowDeadline: time.Minute}, fake } // TestResumeActor_AteletWireRequest is the characteristic test for the @@ -1581,6 +1621,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { // Paused and suspended actors are both SUSPENDED; a pause // checkpoint is what makes the resume local. actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} + var extGen, localGen int32 seedWorkflowActor(t, ctx, persistence, actorRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDED, func(a *ateapipb.Actor) { a.SourceTag = tt.actor.sourceTag a.Status.WorkerAssignment = wireTestAssignment() @@ -1592,6 +1633,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { var gen int32 if tt.actor.externalSnapshot != nil { gen++ + extGen = gen ext := proto.CloneOf(tt.actor.externalSnapshot) ext.Generation = gen if ext.ActorTemplateUid == "" { @@ -1601,6 +1643,7 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { } if tt.actor.localSnapshot != nil { gen++ + localGen = gen a.Status.AssignedNode = "node-1" loc := proto.CloneOf(tt.actor.localSnapshot) loc.Generation = gen @@ -1652,6 +1695,13 @@ func TestResumeActor_AteletWireRequest(t *testing.T) { if got := restore.GetSnapshot().GetFidelity(); got != tt.want.scope { t.Errorf("restore scope = %v, want %v", got, tt.want.scope) } + wantGen := extGen + if tt.want.snapshotName == localSnapshotName { + wantGen = localGen + } + if got := restore.GetSnapshot().GetGeneration(); got != wantGen { + t.Errorf("Snapshot.Generation = %d, want %d, the restored snapshot's", got, wantGen) + } }) } } @@ -1670,6 +1720,7 @@ func (p *publishContextVolumePlugin) AttachVolume(ctx context.Context, req volum // The attach step hands the publish context to the Restore call in memory and // leaves the stored actor untouched. func TestEnsureVolumesAttached_ReturnsPublishContext(t *testing.T) { + preview.InitForTest(t, "ExternalVolumes") ctx := context.Background() persistence := newTestPersistence(t) actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} diff --git a/cmd/ateapi/internal/controlapi/workflow_revert.go b/cmd/ateapi/internal/controlapi/workflow_revert.go index 44d6d30263..696a0d3799 100644 --- a/cmd/ateapi/internal/controlapi/workflow_revert.go +++ b/cmd/ateapi/internal/controlapi/workflow_revert.go @@ -25,6 +25,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "go.opentelemetry.io/otel/attribute" @@ -56,6 +57,7 @@ func isRevertable(actor *ateapipb.Actor) bool { // actor was reverted from, so a re-entered workflow fast-forwards to wherever // the previous attempt stopped. func (w *ActorWorkflow) RevertActor(ctx context.Context, actorRef resources.ActorRef) (_ *ateapipb.Actor, err error) { + ctx = preview.ContextWithOptIn(ctx) start := time.Now() var actor *ateapipb.Actor var actorTemplate *ateapipb.ActorTemplate diff --git a/cmd/ateapi/internal/controlapi/workflow_revert_test.go b/cmd/ateapi/internal/controlapi/workflow_revert_test.go index c7fe7ca6bb..ecf052fb4f 100644 --- a/cmd/ateapi/internal/controlapi/workflow_revert_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_revert_test.go @@ -189,20 +189,25 @@ func TestRevertActor_ReturnsActorToItsSnapshot(t *testing.T) { // allocate generation 4 rather than reusing 2 or 3. running := mustUpdateActorStatus(t, ctx, st, reverted, func(s *ateapipb.ActorStatus) { s.State = ateapipb.ActorState_ACTOR_STATE_RUNNING + s.AssignedNode = "node-1" + s.WorkerAssignment = wireTestAssignment() }) - tmpl := &ateapipb.ActorTemplate{ - SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots/"}, - } - suspending, err := w.ensureMarkedSuspending(ctx, actorRef, running, tmpl) + seedWireWorker(t, ctx, st, assignmentOf(running)) + wire, atelet := newWireCaptureWorkflow(t, st) + suspended, err := wire.SuspendActor(ctx, actorRef) if err != nil { - t.Fatalf("ensureMarkedSuspending after revert: %v", err) + t.Fatalf("SuspendActor after revert: %v", err) } - if got := suspending.GetStatus().GetLastAssignedGeneration(); got != 4 { + if got := suspended.GetStatus().GetLastAssignedGeneration(); got != 4 { t.Errorf("last_assigned_generation after post-revert suspend = %d, want 4", got) } - if got := snapshotAtLatestGeneration(suspending.GetStatus()).GetGeneration(); got != 4 { + newSnap := snapshotAtLatestGeneration(suspended.GetStatus()) + if got := newSnap.GetGeneration(); got != 4 { t.Errorf("new snapshot generation after post-revert suspend = %d, want 4", got) } + if got, want := atelet.checkpointRequest().GetSnapshot().GetSnapshotUuid(), newSnap.GetUuid(); got != want { + t.Errorf("CheckpointRequest.Snapshot.SnapshotUuid = %q, want the generation 4 snapshot %q", got, want) + } }) } } diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend.go b/cmd/ateapi/internal/controlapi/workflow_suspend.go index 400b9ce376..b2655daa16 100644 --- a/cmd/ateapi/internal/controlapi/workflow_suspend.go +++ b/cmd/ateapi/internal/controlapi/workflow_suspend.go @@ -24,6 +24,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -37,6 +38,7 @@ import ( // the steps a previous attempt completed, deriving progress from the // persisted actor alone. func (w *ActorWorkflow) SuspendActor(ctx context.Context, actorRef resources.ActorRef) (_ *ateapipb.Actor, err error) { + ctx = preview.ContextWithOptIn(ctx) start := time.Now() var actor *ateapipb.Actor var actorTemplate *ateapipb.ActorTemplate @@ -249,7 +251,7 @@ func (w *ActorWorkflow) ensureAteletSuspended(ctx context.Context, actorRef reso if err != nil { return "", fmt.Errorf("while getting atelet conn for node %q: %w", assignment.GetNodeName(), err) } - client := ateletpb.NewAteomHerderClient(ateletConn) + client := ateletpb.NewAteletClient(ateletConn) workloadSpec, err := workloadSpecFromActorTemplate(actorTemplate, actor, nil) if err != nil { @@ -270,6 +272,7 @@ func (w *ActorWorkflow) ensureAteletSuspended(ctx context.Context, actorRef reso Spec: workloadSpec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: inProgressSnap.GetUuid(), + Generation: inProgressSnap.GetGeneration(), Fidelity: fidelity, Object: &ateletpb.ObjectStorage{ SnapshotUri: inProgressSnap.GetDurableSnapshot().GetObject().GetSnapshotUri(), @@ -295,7 +298,7 @@ func (w *ActorWorkflow) ensureAteletSuspended(ctx context.Context, actorRef reso // ensurePausedSnapshotUploaded suspends a paused actor by telling the atelet // on the node holding the local pause snapshot to upload it to the actor's // persisted in-progress snapshot location; no workload runs, so there is no -// ateom to checkpoint. Retries re-send the same semantic request: the +// worker to checkpoint. Retries re-send the same semantic request: the // destination is minted once and the upload overwrites deterministic object // names, with the remote manifest as the commit marker. func (w *ActorWorkflow) ensurePausedSnapshotUploaded(ctx context.Context, actorRef resources.ActorRef, actor *ateapipb.Actor, actorTemplate *ateapipb.ActorTemplate) (wireFidelity string, err error) { @@ -320,7 +323,7 @@ func (w *ActorWorkflow) ensurePausedSnapshotUploaded(ctx context.Context, actorR // retryable rather than crash. return "", fmt.Errorf("while getting atelet conn for node %q: %w", nodeName, err) } - client := ateletpb.NewAteomHerderClient(ateletConn) + client := ateletpb.NewAteletClient(ateletConn) // The template's preferred fidelity, as a running-origin suspend commits. fidelity := fidelityToAtelet(preferredFidelity(actor.GetMetadata().GetAtespace(), actorTemplate)) @@ -332,6 +335,7 @@ func (w *ActorWorkflow) ensurePausedSnapshotUploaded(ctx context.Context, actorR ActorTemplateName: actor.GetActorTemplate().GetName(), Snapshot: &ateletpb.Snapshot{ SnapshotUuid: localSnap.GetUuid(), + Generation: localSnap.GetGeneration(), Fidelity: fidelity, Object: &ateletpb.ObjectStorage{ SnapshotUri: snapshotAtLatestGeneration(actor.GetStatus()).GetDurableSnapshot().GetObject().GetSnapshotUri(), @@ -366,6 +370,11 @@ func (w *ActorWorkflow) ensureVolumesDetached(ctx context.Context, actor *ateapi ctx, done := stepSpan(ctx, spanName) defer func() { err = done(err) }() + // Given that external volumes has been in the codebase for a while and + // its preview-ness is solely around the API, and that the impact of + // disabling the gate while in use would be an orphaned volume + // attachment, we are NOT checking the ExternalVolumes gate in the + // delete/suspend path. return detachActorVolumes(ctx, w.pluginRegistry, actor, actorTemplate, op) } diff --git a/cmd/ateapi/internal/controlapi/workflow_suspend_test.go b/cmd/ateapi/internal/controlapi/workflow_suspend_test.go index 202d9169cc..8bed842237 100644 --- a/cmd/ateapi/internal/controlapi/workflow_suspend_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_suspend_test.go @@ -17,15 +17,15 @@ package controlapi import ( "context" "errors" + "fmt" "strings" "testing" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storetest" "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/installdefaults" - "github.com/agent-substrate/substrate/internal/objectstore/objectstoretest" - "github.com/agent-substrate/substrate/internal/objectstoreplugin/objectstoreplugintest" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -36,10 +36,10 @@ import ( "k8s.io/client-go/tools/cache" ) -func TestEnsureMarkedSuspending_SnapshotURI(t *testing.T) { +func TestSuspendActor_MarkedSuspending_SnapshotURI(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) - workerName := testWorkerUID("worker-pod-1") + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: "gs://bucket/root/"}) wantRuntime := &ateapipb.SandboxRuntime{ SandboxClass: "gvisor", Version: &ateapipb.VersionedSandboxCompat{ @@ -49,28 +49,21 @@ func TestEnsureMarkedSuspending_SnapshotURI(t *testing.T) { }, }, } - actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + assignment := wireTestAssignment() + assignment.SandboxRuntime = wantRuntime + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_RUNNING, - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - SandboxRuntime: wantRuntime, - }, + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + WorkerAssignment: assignment, }, }) - tmpl := &ateapipb.ActorTemplate{ - SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://bucket/root/"}, - } - w := &ActorWorkflow{store: persistence} - marked, err := w.ensureMarkedSuspending(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}, actor, tmpl) - if err != nil { - t.Fatalf("ensureMarkedSuspending: %v", err) - } + w := newMarkOnlyWorkflow(persistence) + _, err := w.SuspendActor(ctx, actorRef) + assertStoppedAtDial(t, err) + marked := mustReadActor(t, ctx, persistence, actorRef) inProgressSnap := findLatestDurableSnapshot(marked.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) uri := mustParseSnapshotURI(t, inProgressSnap.GetDurableSnapshot().GetObject().GetSnapshotUri()) @@ -89,31 +82,44 @@ func TestEnsureMarkedSuspending_SnapshotURI(t *testing.T) { } } -// TestEnsureMarkedSuspending_ReentryKeepsPersistedSnapshotLocation verifies a +// TestSuspendActor_MarkedSuspending_ReentryKeepsPersistedSnapshotLocation verifies a // re-entered workflow does not mint a second snapshot location: the location -// persisted by the first attempt stays authoritative. -func TestEnsureMarkedSuspending_ReentryKeepsPersistedSnapshotLocation(t *testing.T) { +// persisted by the first attempt stays authoritative, both on the wire and in +// the snapshot the suspend commits. +func TestSuspendActor_MarkedSuspending_ReentryKeepsPersistedSnapshotLocation(t *testing.T) { const firstAttempt = "gs://bucket/root/atespaces/team-a/actors/actor-uid/snapshots/first-attempt" ctx := context.Background() persistence := newTestPersistence(t) - actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}) + w, atelet := newWireCaptureWorkflow(t, persistence) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{ newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "first-attempt", firstAttempt, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), }, }, }) - w := &ActorWorkflow{store: persistence} - marked, err := w.ensureMarkedSuspending(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}, actor, &ateapipb.ActorTemplate{}) + suspended, err := w.SuspendActor(ctx, actorRef) if err != nil { - t.Fatalf("ensureMarkedSuspending: %v", err) + t.Fatalf("SuspendActor: %v", err) + } + if got := atelet.checkpointRequest().GetSnapshot().GetObject().GetSnapshotUri(); got != firstAttempt { + t.Errorf("CheckpointRequest.Snapshot.Object.SnapshotUri = %q, want the first attempt's location", got) + } + if got := suspended.GetStatus().GetLastAssignedGeneration(); got != 1 { + t.Errorf("last_assigned_generation = %d, want 1 (no new snapshot minted on re-entry)", got) } - if gotSnap := findLatestDurableSnapshot(marked.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING); gotSnap.GetDurableSnapshot().GetObject().GetSnapshotUri() != firstAttempt { - t.Errorf("in-progress durable snapshot URI = %q, want the first attempt's location", gotSnap.GetDurableSnapshot().GetObject().GetSnapshotUri()) + gotSnap := findLatestDurableSnapshot(suspended.GetStatus(), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED) + if got := gotSnap.GetDurableSnapshot().GetObject().GetSnapshotUri(); got != firstAttempt { + t.Errorf("committed durable snapshot URI = %q, want the first attempt's location", got) } } @@ -171,88 +177,96 @@ func TestSuspendActorWorkflow_RejectedAndIdempotentPaths(t *testing.T) { } } -// TestEnsureMarkedSuspending_StateMatrix verifies the suspend edge's state -// gating against every actor state: RUNNING takes the edge (checkpoint the -// workload), SUSPENDING skips (a previous attempt already marked the actor), -// everything else is rejected with FailedPrecondition. SUSPENDED without a -// node-local snapshot is rejected here because the orchestrator -// early-returns before this step for a fully suspended actor; SUSPENDED with -// one is the paused shape, covered by -// TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot. -func TestEnsureMarkedSuspending_StateMatrix(t *testing.T) { - allowed := map[ateapipb.ActorState]bool{ - ateapipb.ActorState_ACTOR_STATE_RUNNING: true, - ateapipb.ActorState_ACTOR_STATE_SUSPENDING: true, // skipped, not re-marked - } +// TestSuspendActor_MarkedSuspending_StateMatrix verifies the suspend edge's state gating +// against every actor state: RUNNING takes the edge (checkpoint the +// workload), SUSPENDING skips the mark (a previous attempt already marked the +// actor) and goes on to atelet, everything else is rejected with +// FailedPrecondition and left untouched. SUSPENDED without a node-local +// snapshot fast-forwards (see TestSuspendActorWorkflow_RejectedAndIdempotentPaths); +// SUSPENDED with one is the paused shape, covered by +// TestSuspendActor_MarkedSuspending_PausedPromotesLocalSnapshot. +func TestSuspendActor_MarkedSuspending_StateMatrix(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}) + w := newMarkOnlyWorkflow(persistence) for _, seedState := range allActorStates { - ctx := context.Background() - persistence := newTestPersistence(t) - w := &ActorWorkflow{store: persistence} - - actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} - actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, - Status: &ateapipb.ActorStatus{State: seedState}, + actorRef := resources.ActorRef{Atespace: "team-a", Name: fmt.Sprintf("id%d", seedState)} + seedWorkflowActor(t, ctx, persistence, actorRef, "ns", "tmpl1", seedState, func(a *ateapipb.Actor) { + a.Status.WorkerAssignment = wireTestAssignment() }) - tmpl := &ateapipb.ActorTemplate{ - SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}, - } - marked, err := w.ensureMarkedSuspending(ctx, actorRef, actor, tmpl) - assertPrerequisiteResult(t, seedState, err, allowed[seedState]) - if err == nil && marked.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { - t.Errorf("state %v: ensureMarkedSuspending returned actor in %v, want SUSPENDING", seedState, marked.GetStatus().GetState()) + _, err := w.SuspendActor(ctx, actorRef) + stored := mustReadActor(t, ctx, persistence, actorRef) + switch seedState { + case ateapipb.ActorState_ACTOR_STATE_RUNNING, ateapipb.ActorState_ACTOR_STATE_SUSPENDING: + assertStoppedAtDial(t, err) + if got := stored.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { + t.Errorf("state %v: stored state = %v, want SUSPENDING", seedState, got) + } + case ateapipb.ActorState_ACTOR_STATE_SUSPENDED: + if err != nil { + t.Errorf("state %v: SuspendActor = %v, want the fast-forward", seedState, err) + } + if got := stored.GetStatus().GetState(); got != seedState { + t.Errorf("state %v: stored state = %v, want it untouched", seedState, got) + } + default: + if got := apierror.Code(err); got != codes.FailedPrecondition { + t.Errorf("state %v: code = %v, want FailedPrecondition (err: %v)", seedState, got, err) + } + if got := stored.GetStatus().GetState(); got != seedState { + t.Errorf("state %v: stored state = %v, want it untouched", seedState, got) + } } } } -// TestEnsureMarkedSuspending_RejectsPauseInProgress verifies a suspend does -// not adopt a SUSPENDING actor whose latest generation is a pending -// node-local snapshot: that actor is mid-pause, and the pause owns it until -// it is paused. -func TestEnsureMarkedSuspending_RejectsPauseInProgress(t *testing.T) { +// TestSuspendActor_MarkedSuspending_RejectsPauseInProgress verifies a suspend does not adopt a +// SUSPENDING actor whose latest generation is a pending node-local snapshot: +// that actor is mid-pause, and the pause owns it until it is paused. +func TestSuspendActor_MarkedSuspending_RejectsPauseInProgress(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) - w := &ActorWorkflow{store: persistence} + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}) + w := newMarkOnlyWorkflow(persistence) actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} - pausing := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, - Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING}, - } - withPauseInProgress()(pausing) - created := storetest.MustCreateActor(t, ctx, persistence, pausing) + seedWorkflowActor(t, ctx, persistence, actorRef, "ns", "tmpl1", ateapipb.ActorState_ACTOR_STATE_SUSPENDING, withPauseInProgress(), func(a *ateapipb.Actor) { + a.Status.WorkerAssignment = wireTestAssignment() + }) - tmpl := &ateapipb.ActorTemplate{SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}} - _, err := w.ensureMarkedSuspending(ctx, actorRef, created, tmpl) + _, err := w.SuspendActor(ctx, actorRef) if got := apierror.Code(err); got != codes.FailedPrecondition { - t.Errorf("ensureMarkedSuspending on a pause in progress: code = %v, want FailedPrecondition (err: %v)", got, err) - } - stored, err := persistence.GetActor(ctx, actorRef) - if err != nil { - t.Fatalf("GetActor: %v", err) + t.Errorf("SuspendActor on a pause in progress: code = %v, want FailedPrecondition (err: %v)", got, err) } + stored := mustReadActor(t, ctx, persistence, actorRef) if got := stored.GetStatus().GetLastAssignedGeneration(); got != 1 { t.Errorf("last_assigned_generation = %d, want 1 (the pause's snapshot untouched)", got) } + if !latestSnapshotIsPendingLocal(stored.GetStatus()) { + t.Errorf("snapshots = %v, want the pause's pending local snapshot untouched", stored.GetStatus().GetSnapshots()) + } } -// TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot verifies the suspend -// edge from a paused actor (SUSPENDED whose latest snapshot is node-local): -// it takes the edge and promotes that snapshot in place to a pending durable -// one, keeping its generation, uuid, and sandbox runtime stamp, with the -// upload destination derived from the uuid. -func TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot(t *testing.T) { +// TestSuspendActor_MarkedSuspending_PausedPromotesLocalSnapshot verifies the suspend edge from +// a paused actor (SUSPENDED whose latest snapshot is node-local): it takes the +// edge and promotes that snapshot in place to a pending durable one, keeping +// its generation, uuid, and sandbox runtime stamp, with the upload destination +// derived from the uuid. +func TestSuspendActor_MarkedSuspending_PausedPromotesLocalSnapshot(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) - w := &ActorWorkflow{store: persistence} + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}) + w := newMarkOnlyWorkflow(persistence) actorRef := resources.ActorRef{Atespace: "team-a", Name: "id1"} localSnap := newLocalSnapshot(1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap-1", "node-1") localSnap.SandboxRuntime = testSandboxRuntime() actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDED, AssignedNode: "node-1", @@ -261,13 +275,10 @@ func TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot(t *testing.T) { }, }) - tmpl := &ateapipb.ActorTemplate{ - SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}, - } - marked, err := w.ensureMarkedSuspending(ctx, actorRef, actor, tmpl) - if err != nil { - t.Fatalf("ensureMarkedSuspending: %v", err) - } + // The upload's dial fails, so the actor stays as the mark left it. + _, err := w.SuspendActor(ctx, actorRef) + assertStoppedAtDial(t, err) + marked := mustReadActor(t, ctx, persistence, actorRef) if got := marked.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { t.Errorf("state = %v, want SUSPENDING", got) } @@ -299,9 +310,9 @@ func TestEnsureMarkedSuspending_PausedPromotesLocalSnapshot(t *testing.T) { } // TestSuspendActor_CrashesWhenSuspendingActorMissingWorkerPod verifies that a -// SUSPENDING actor with no worker pod recorded is moved to CRASHED by -// CallAteletSuspendStep's prerequisite check and the suspend fails with -// FailedPrecondition. +// SUSPENDING actor with no worker pod recorded is moved to CRASHED by the +// corrupted-assignment check before atelet is called, and the suspend fails +// with FailedPrecondition. func TestSuspendActor_CrashesWhenSuspendingActorMissingWorkerPod(t *testing.T) { ctx := context.Background() st, cleanup := storetest.SetupTestStore(t) @@ -343,7 +354,7 @@ func newDanglingDialer() *AteletDialer { return NewAteletDialer(empty, installdefaults.AteletSPIFFEID(installdefaults.SystemNamespace), "", "") } -func TestEnsureAteletSuspended_DialFailureLeavesActorRetryable(t *testing.T) { +func TestSuspendActor_AteletSuspended_DialFailureLeavesActorRetryable(t *testing.T) { neverWritten := someActorSnapshotURI(t, testStorageLocation, "team-a", "never-written") tests := []struct { @@ -364,6 +375,7 @@ func TestEnsureAteletSuspended_DialFailureLeavesActorRetryable(t *testing.T) { t.Run(tt.name, func(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}) var snapshots []*ateapipb.Snapshot if tt.prevSnapshot != "" { snapshots = append(snapshots, newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "prev-snapshot", tt.prevSnapshot, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED)) @@ -371,7 +383,8 @@ func TestEnsureAteletSuspended_DialFailureLeavesActorRetryable(t *testing.T) { snapshots = append(snapshots, newDurableSnapshot(2, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "never-written", neverWritten, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING)) actor := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, WorkerAssignment: &ateapipb.WorkerAssignment{ @@ -384,12 +397,11 @@ func TestEnsureAteletSuspended_DialFailureLeavesActorRetryable(t *testing.T) { Snapshots: snapshots, }, } - created := storetest.MustCreateActor(t, ctx, persistence, actor) + storetest.MustCreateActor(t, ctx, persistence, actor) - w := &ActorWorkflow{store: persistence, dialer: newDanglingDialer()} - if _, err := w.ensureAteletSuspended(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}, created, &ateapipb.ActorTemplate{}); err == nil { - t.Fatal("ensureAteletSuspended: want error when atelet is unreachable, got nil") - } + w := newMarkOnlyWorkflow(persistence) + _, err := w.SuspendActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}) + assertStoppedAtDial(t, err) // A dial failure is transient from this workflow's point of view: it // must not crash the actor or touch its snapshot state. A worker that @@ -412,20 +424,23 @@ func TestEnsureAteletSuspended_DialFailureLeavesActorRetryable(t *testing.T) { } } -// TestEnsureAteletSuspended_WireRequest verifies a running-origin suspend -// checkpoints to the actor's in-progress durable snapshot with an immediate -// upload, since FinalizeSuspended then marks that snapshot COMPLETED. -func TestEnsureAteletSuspended_WireRequest(t *testing.T) { +// TestSuspendActor_AteletSuspended_WireRequest verifies a running-origin suspend checkpoints +// to the actor's in-progress durable snapshot with an immediate upload, since +// the suspend's finalize then marks that snapshot COMPLETED. +func TestSuspendActor_AteletSuspended_WireRequest(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}) w, atelet := newWireCaptureWorkflow(t, persistence) + inst, reader := newTestInstruments(t) + w.instruments = inst actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} snapshotName := "2026-01-01t00-00-00z-abc" snapshotURI := someActorSnapshotURI(t, testStorageLocation, "team-a", snapshotName) - created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, - ActorTemplate: &ateapipb.ObjectRef{Atespace: "ns", Name: "tmpl1"}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: "node-1", @@ -436,13 +451,9 @@ func TestEnsureAteletSuspended_WireRequest(t *testing.T) { }, }, }) - tmpl := &ateapipb.ActorTemplate{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "ns", Name: "tmpl1"}, - SnapshotConfig: &ateapipb.SnapshotConfig{PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, - } - if _, err := w.ensureAteletSuspended(ctx, actorRef, created, tmpl); err != nil { - t.Fatalf("ensureAteletSuspended: %v", err) + if _, err := w.SuspendActor(ctx, actorRef); err != nil { + t.Fatalf("SuspendActor: %v", err) } req := atelet.checkpointRequest() if req == nil { @@ -451,6 +462,9 @@ func TestEnsureAteletSuspended_WireRequest(t *testing.T) { if got := req.GetSnapshot().GetSnapshotUuid(); got != snapshotName { t.Errorf("CheckpointRequest.Snapshot.SnapshotUuid = %q, want %q", got, snapshotName) } + if got := req.GetSnapshot().GetGeneration(); got != 1 { + t.Errorf("CheckpointRequest.Snapshot.Generation = %d, want 1, the in-progress snapshot's", got) + } if got := req.GetSnapshot().GetObject().GetSnapshotUri(); got != snapshotURI { t.Errorf("CheckpointRequest.Snapshot.Object.SnapshotUri = %q, want %q", got, snapshotURI) } @@ -458,37 +472,59 @@ func TestEnsureAteletSuspended_WireRequest(t *testing.T) { if got := req.GetUploadConfig(); !proto.Equal(got, wantConfig) { t.Errorf("CheckpointRequest.UploadConfig = %v, want %v", got, wantConfig) } + // The fidelity sent to atelet labels the suspend's lifecycle-op metric. + assertLifecycleFidelity(t, reader, ateattr.SnapshotFidelityMemory) } -// TestEnsureSuspendedFinalized_NoAssignment verifies finalization runs even when -// the actor has no worker assignment: the external snapshot must be recorded on -// the actor and the actor moved to SUSPENDED rather than silently left -// SUSPENDING. This is the shape a paused-origin suspend (#791) produces — a -// paused actor has no worker — and the regression test for finalization +// TestSuspendActor_Finalized_NoAssignment verifies finalization runs even +// when the actor has no worker assignment: the external snapshot must be +// recorded on the actor and the actor moved to SUSPENDED rather than silently +// left SUSPENDING. This is the shape a paused-origin suspend (#791) produces — +// a paused actor has no worker — and the regression test for finalization // previously living inside the worker-freeing branch. -func TestEnsureSuspendedFinalized_NoAssignment(t *testing.T) { +func TestSuspendActor_Finalized_NoAssignment(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}) + w, atelet := newWireCaptureWorkflow(t, persistence) + inst, reader := newTestInstruments(t) + w.instruments = inst + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} snapshotURI := someActorSnapshotURI(t, testStorageLocation, "team-a", "2026-01-01t00-00-00z-abc") snap := newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "tmpl-uid-1", "actor-1-pause-snapshot", snapshotURI, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) - snap.Locality = "node1" - actor := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + snap.Locality = "node-1" + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, - AssignedNode: "node1", + AssignedNode: "node-1", LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{snap}, }, - } - storetest.MustCreateActor(t, ctx, persistence, actor) + }) - w := &ActorWorkflow{store: persistence} - stored, err := w.ensureSuspendedFinalized(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}) + stored, err := w.SuspendActor(ctx, actorRef) if err != nil { - t.Fatalf("ensureSuspendedFinalized: %v", err) + t.Fatalf("SuspendActor: %v", err) + } + + // With no worker there is no workload to checkpoint: the suspend uploads + // the node-local snapshot instead. + if checkpoints, promotes := atelet.calls(); checkpoints != 0 || promotes != 1 { + t.Errorf("atelet calls = %d Checkpoint, %d PromoteSnapshot; want 0 and 1", checkpoints, promotes) + } + promote := atelet.promoteRequest() + if got := promote.GetSnapshot().GetSnapshotUuid(); got != "actor-1-pause-snapshot" { + t.Errorf("PromoteSnapshotRequest.Snapshot.SnapshotUuid = %q, want %q", got, "actor-1-pause-snapshot") } + if got := promote.GetSnapshot().GetObject().GetSnapshotUri(); got != snapshotURI { + t.Errorf("PromoteSnapshotRequest.Snapshot.Object.SnapshotUri = %q, want %q", got, snapshotURI) + } + // The template's preferred fidelity, which the store defaults to MEMORY, + // labels the suspend's lifecycle-op metric. + assertLifecycleFidelity(t, reader, ateattr.SnapshotFidelityMemory) if stored.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { t.Errorf("state = %v, want SUSPENDED", stored.GetStatus().GetState()) @@ -513,11 +549,11 @@ func TestEnsureSuspendedFinalized_NoAssignment(t *testing.T) { } } -// TestEnsureSuspendedFinalized_ReleasesReplacedSnapshot verifies which external +// TestSuspendActor_Finalized_ReleasesReplacedSnapshot verifies which external // snapshot a suspend collects. The actor's previous one goes away, because // nothing else can name it — unless the actor was only borrowing it from a tag, // in which case the tag owns those objects and the suspend must leave them be. -func TestEnsureSuspendedFinalized_ReleasesReplacedSnapshot(t *testing.T) { +func TestSuspendActor_Finalized_ReleasesReplacedSnapshot(t *testing.T) { // The name of the snapshot the suspend has just written, in place of the // one the actor was running from. const snapshotName = "2026-01-01t00-00-00z-new" @@ -546,13 +582,17 @@ func TestEnsureSuspendedFinalized_ReleasesReplacedSnapshot(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) template := seedSubstrateTemplate(t, ctx, persistence, "sub-tmpl") - w, objects := newFinalizeWorkflow(persistence) + w, _, objects := newWireFinalizeWorkflow(t, persistence) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, ActorTemplate: &ateapipb.ObjectRef{Atespace: "team-a", Name: "sub-tmpl"}, - Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), + }, }) previous := mustActorSnapshotURI(t, template, actor, "old") @@ -573,9 +613,9 @@ func TestEnsureSuspendedFinalized_ReleasesReplacedSnapshot(t *testing.T) { } }) - stored, err := w.ensureSuspendedFinalized(ctx, actorRef) + stored, err := w.SuspendActor(ctx, actorRef) if err != nil { - t.Fatalf("ensureSuspendedFinalized: %v", err) + t.Fatalf("SuspendActor: %v", err) } // Whichever way the previous snapshot went, the actor now owns the // one it just wrote and is no longer borrowing. @@ -603,23 +643,27 @@ func isObjectStoreErr(err error) bool { return err != nil && strings.Contains(err.Error(), errObjectStore.Error()) } -// TestEnsureSuspendedFinalized_CommitsDespiteObjectStoreFailure verifies that +// TestSuspendActor_Finalized_CommitsDespiteObjectStoreFailure verifies that // failing to collect the snapshot a suspend replaced does not fail the // suspend. The worker is already released by then, so aborting would leave the // actor SUSPENDING with no way to resume; it commits SUSPENDED on the new // snapshot instead and leaves the replaced one in storage. -func TestEnsureSuspendedFinalized_CommitsDespiteObjectStoreFailure(t *testing.T) { +func TestSuspendActor_Finalized_CommitsDespiteObjectStoreFailure(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) template := seedSubstrateTemplate(t, ctx, persistence, "sub-tmpl") - w, objects := newFinalizeWorkflow(persistence) + w, _, objects := newWireFinalizeWorkflow(t, persistence) const snapshotName = "2026-01-01t00-00-00z-new" actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, ActorTemplate: &ateapipb.ObjectRef{Atespace: "team-a", Name: "sub-tmpl"}, - Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), + }, }) previous := mustActorSnapshotURI(t, template, actor, "old") @@ -636,9 +680,9 @@ func TestEnsureSuspendedFinalized_CommitsDespiteObjectStoreFailure(t *testing.T) }) objects.OnDelete = func(string, string) error { return errObjectStore } - stored, err := w.ensureSuspendedFinalized(ctx, actorRef) + stored, err := w.SuspendActor(ctx, actorRef) if err != nil { - t.Fatalf("ensureSuspendedFinalized: %v", err) + t.Fatalf("SuspendActor: %v", err) } if got := stored.GetStatus().GetState(); got != ateapipb.ActorState_ACTOR_STATE_SUSPENDED { t.Errorf("state = %v, want SUSPENDED", got) @@ -658,21 +702,27 @@ func TestEnsureSuspendedFinalized_CommitsDespiteObjectStoreFailure(t *testing.T) } } -// TestEnsureSuspendedFinalized_KeepsReplacedSnapshotOnConflict verifies that a +// TestSuspendActor_Finalized_KeepsReplacedSnapshotOnConflict verifies that a // commit lost to a concurrent update releases nothing: the actor record still // names its old snapshot, so collecting it would leave the actor pointing at // deleted objects. -func TestEnsureSuspendedFinalized_KeepsReplacedSnapshotOnConflict(t *testing.T) { +func TestSuspendActor_Finalized_KeepsReplacedSnapshotOnConflict(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) template := seedSubstrateTemplate(t, ctx, persistence, "sub-tmpl") - objects := objectstoretest.New() + // The actor is seeded SUSPENDING, so the mark step skips and the first + // actor update the suspend makes is its finalize commit. + w, atelet, objects := newWireFinalizeWorkflow(t, &conflictingUpdateStore{Interface: persistence}) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} actor := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, ActorTemplate: &ateapipb.ObjectRef{Atespace: "team-a", Name: "sub-tmpl"}, - Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING}, + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), + }, }) previous := mustActorSnapshotURI(t, template, actor, "old") @@ -687,9 +737,13 @@ func TestEnsureSuspendedFinalized_KeepsReplacedSnapshotOnConflict(t *testing.T) } }) - w := &ActorWorkflow{store: &conflictingUpdateStore{Interface: persistence}, objectStorePlugin: objectstoreplugintest.ControlClient(objects)} - if _, err := w.ensureSuspendedFinalized(ctx, actorRef); apierror.Code(err) != codes.Aborted { - t.Fatalf("ensureSuspendedFinalized = %v, want code Aborted", err) + if _, err := w.SuspendActor(ctx, actorRef); apierror.Code(err) != codes.Aborted { + t.Fatalf("SuspendActor = %v, want code Aborted", err) + } + // Every actor update conflicts, so pin that the Aborted came from finalize + // and not from an earlier step that never reached atelet. + if checkpoints, _ := atelet.calls(); checkpoints != 1 { + t.Errorf("Checkpoint calls = %d, want 1 before finalize", checkpoints) } if len(objects.Snapshot(t, previous)) == 0 { t.Error("replaced external snapshot was collected though the commit lost") @@ -706,57 +760,37 @@ func (conflictingUpdateStore) UpdateActor(context.Context, resources.ActorRef, s return nil, store.ErrVersionConflict } -// TestEnsureSuspendedFinalized_KeepsCrashFromWorkerDelete verifies that a -// crash the DeleteWorker workflow commits while ensureSuspendedFinalized is -// freeing the worker is kept: the step's update is pinned to the actor version -// it checked, so it fails Aborted instead of committing SUSPENDED over the -// crash. -func TestEnsureSuspendedFinalized_KeepsCrashFromWorkerDelete(t *testing.T) { +// TestSuspendActor_Finalized_KeepsCrashFromWorkerDelete verifies that a crash the +// DeleteWorker workflow commits while the suspend's finalize is freeing the +// worker is kept: the finalize update is pinned to the actor version it +// checked, so it fails Aborted instead of committing SUSPENDED over the crash. +func TestSuspendActor_Finalized_KeepsCrashFromWorkerDelete(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} - workerName := testWorkerUID("worker-pod-1") created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, - AssignedNode: "node1", - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - }, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{ newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snapshot-1", someActorSnapshotURI(t, testStorageLocation, "team-a", "snapshot-1"), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), }, }, }) - if _, err := persistence.CreateWorker(ctx, &ateapipb.Worker{ - Metadata: &ateapipb.ResourceMetadata{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - NodeName: "node1", - Status: &ateapipb.WorkerStatus{}, - }); err != nil { - t.Fatalf("CreateWorker: %v", err) - } - seedAssignment(t, persistence, workerName, &ateapipb.ActorAssignment{ - Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: actorRef.Name}, - ActorUid: created.GetMetadata().GetUid(), - }) + seedWireWorker(t, ctx, persistence, assignmentOf(created)) - w := &ActorWorkflow{store: workerDeletedDuringReleaseStore{Interface: persistence}} - _, err := w.ensureSuspendedFinalized(ctx, actorRef) + w, _ := newWireCaptureWorkflow(t, workerDeletedDuringReleaseStore{Interface: persistence}) + _, err := w.SuspendActor(ctx, actorRef) assertConcurrentCrashKept(t, ctx, persistence, actorRef, err, "suspend failed: "+crashMessageWorkerPodGone) } -func TestEnsureSuspendedFinalized_ReleasesOnlyOwnWorker(t *testing.T) { +func TestSuspendActor_Finalized_ReleasesOnlyOwnWorker(t *testing.T) { tests := []struct { name string assignmentAtespace string @@ -785,17 +819,15 @@ func TestEnsureSuspendedFinalized_ReleasesOnlyOwnWorker(t *testing.T) { t.Run(tt.name, func(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}) + w, _ := newWireCaptureWorkflow(t, persistence) actor := &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "shared"}, + Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "shared"}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: testWorkerUID("pod-1")}, - WorkerNamespace: "worker-ns", - WorkerPool: "pool", - WorkerPod: "pod-1", - WorkerPodUid: testWorkerUID("pod-1"), - }, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{ newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snapshot-1", someActorSnapshotURI(t, testStorageLocation, "team-a", "snapshot-1"), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), @@ -808,28 +840,16 @@ func TestEnsureSuspendedFinalized_ReleasesOnlyOwnWorker(t *testing.T) { if tt.assignmentAtespace != "team-a" || tt.mismatchedUID { uid = "other-actor-uid-b" } - worker := &ateapipb.Worker{ - Metadata: &ateapipb.ResourceMetadata{Name: testWorkerUID("pod-1")}, - WorkerNamespace: "worker-ns", - WorkerPool: "pool", - WorkerPod: "pod-1", - WorkerPodUid: testWorkerUID("pod-1"), - Status: &ateapipb.WorkerStatus{}, - } - if _, err := persistence.CreateWorker(ctx, worker); err != nil { - t.Fatalf("CreateWorker: %v", err) - } - seedAssignment(t, persistence, testWorkerUID("pod-1"), &ateapipb.ActorAssignment{ + seedWireWorker(t, ctx, persistence, &ateapipb.ActorAssignment{ Actor: &ateapipb.ObjectRef{Atespace: tt.assignmentAtespace, Name: "shared"}, ActorUid: uid, }) - w := &ActorWorkflow{store: persistence} - if _, err := w.ensureSuspendedFinalized(ctx, resources.ActorRef{Atespace: "team-a", Name: "shared"}); err != nil { - t.Fatalf("ensureSuspendedFinalized: %v", err) + if _, err := w.SuspendActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "shared"}); err != nil { + t.Fatalf("SuspendActor: %v", err) } - stored := firstAssignment(t, persistence, testWorkerUID("pod-1")) + stored := firstAssignment(t, persistence, wireTestAssignment().GetWorker().GetName()) if released := stored == nil; released != tt.wantReleased { t.Errorf("worker released = %t, want %t (assignment: %v)", released, tt.wantReleased, stored) } @@ -898,18 +918,23 @@ func TestIsPausedOriginSuspend(t *testing.T) { } } -// TestEnsurePausedSnapshotUploaded_Preconditions covers the paused branch's -// failure handling: a lost node record crashes the actor (the snapshot can -// never be found), while an unreachable atelet stays retryable (the bytes -// are likely still on the node's disk). -func TestEnsurePausedSnapshotUploaded_Preconditions(t *testing.T) { +// TestSuspendActor_PausedSnapshotUploaded_Preconditions covers the paused +// branch's failure handling: a lost node record crashes the actor (the +// snapshot can never be found), while an unreachable atelet stays retryable +// (the bytes are likely still on the node's disk). Both actors are SUSPENDING +// with no worker, so the mark step skips and the suspend takes the paused +// branch. +func TestSuspendActor_PausedSnapshotUploaded_Preconditions(t *testing.T) { t.Run("no node recorded crashes", func(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) - w := &ActorWorkflow{store: persistence, dialer: newDanglingDialer()} + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}) + w := newMarkOnlyWorkflow(persistence) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} - created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, LastAssignedGeneration: 1, @@ -919,29 +944,32 @@ func TestEnsurePausedSnapshotUploaded_Preconditions(t *testing.T) { }, }) - _, err := w.ensurePausedSnapshotUploaded(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}, created, &ateapipb.ActorTemplate{}) + _, err := w.SuspendActor(ctx, actorRef) if got := apierror.Code(err); got != codes.FailedPrecondition { t.Fatalf("apierror.Code(err) = %v, want %v for a missing node record (err: %v)", got, codes.FailedPrecondition, err) } - stored, err := persistence.GetActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}) - if err != nil { - t.Fatalf("GetActor: %v", err) - } + stored := mustReadActor(t, ctx, persistence, actorRef) if stored.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_CRASHED { t.Errorf("state = %v, want CRASHED", stored.GetStatus().GetState()) } + if msg, want := stored.GetStatus().GetCrash().GetMessage(), "suspend failed: "+crashMessageLocalSnapshotNodeUnknown; msg != want { + t.Errorf("crash message = %q, want %q", msg, want) + } }) t.Run("no atelet on node stays retryable", func(t *testing.T) { ctx := context.Background() persistence := newTestPersistence(t) - w := &ActorWorkflow{store: persistence, dialer: newDanglingDialer()} + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}) + w := newMarkOnlyWorkflow(persistence) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} snap := newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snap", someActorSnapshotURI(t, testStorageLocation, "team-a", "snap-dest"), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) snap.Locality = "node1" - created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: "team-a", Name: "actor-1"}, + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: "node1", @@ -950,69 +978,94 @@ func TestEnsurePausedSnapshotUploaded_Preconditions(t *testing.T) { }, }) - tmpl := &ateapipb.ActorTemplate{SnapshotConfig: &ateapipb.SnapshotConfig{StorageLocation: "gs://snapshots"}} - _, err := w.ensurePausedSnapshotUploaded(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}, created, tmpl) + _, err := w.SuspendActor(ctx, actorRef) if !errors.Is(err, ErrNoAteletOnNode) { - t.Fatalf("ensurePausedSnapshotUploaded = %v, want ErrNoAteletOnNode", err) + t.Fatalf("SuspendActor = %v, want ErrNoAteletOnNode", err) } - stored, err := persistence.GetActor(ctx, resources.ActorRef{Atespace: "team-a", Name: "actor-1"}) - if err != nil { - t.Fatalf("GetActor: %v", err) - } + stored := mustReadActor(t, ctx, persistence, actorRef) if stored.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_SUSPENDING { t.Errorf("state = %v, want SUSPENDING (retryable, not crashed)", stored.GetStatus().GetState()) } }) } -// TestEnsureSuspendedFinalized_MissingDurableSnapshotClearsAssignment verifies -// that if the latest generation snapshot entry is missing when -// ensureSuspendedFinalized runs, the actor transitions to CRASHED and clears -// WorkerAssignment after releasing the worker rather than getting stuck in -// SUSPENDING, and the step fails. -func TestEnsureSuspendedFinalized_MissingDurableSnapshotClearsAssignment(t *testing.T) { - st, cleanup := storetest.SetupTestStore(t) - defer cleanup() +// TestSuspendActor_PausedSnapshotUploaded_WireRequest verifies the +// PromoteSnapshot request names the paused actor's local snapshot, with its +// generation, and the destination the suspend recorded for it. +func TestSuspendActor_PausedSnapshotUploaded_WireRequest(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation, PreferredFidelity: ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}) + w, atelet := newWireCaptureWorkflow(t, persistence) + + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} + destURI := someActorSnapshotURI(t, testStorageLocation, "team-a", "snap-dest") + snap := newDurableSnapshot(3, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "pause-snap", destURI, ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING) + snap.Locality = "node-1" + // A suspend from paused in progress: SUSPENDING with no worker, so the + // suspend skips its mark step and promotes the local snapshot. + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + LastAssignedGeneration: 3, + Snapshots: []*ateapipb.Snapshot{snap}, + }, + }) + + if _, err := w.SuspendActor(ctx, actorRef); err != nil { + t.Fatalf("SuspendActor: %v", err) + } + req := atelet.promoteRequest() + if req == nil { + t.Fatal("expected PromoteSnapshot RPC to be called") + } + want := &ateletpb.Snapshot{ + SnapshotUuid: "pause-snap", + Generation: 3, + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{SnapshotUri: destURI}, + } + if got := req.GetSnapshot(); !proto.Equal(got, want) { + t.Errorf("PromoteSnapshotRequest.Snapshot = %v, want %v", got, want) + } +} + +// TestSuspendActor_Finalized_MissingDurableSnapshotClearsAssignment verifies that if the +// latest generation snapshot entry is missing when the suspend finalizes, the +// actor transitions to CRASHED and clears WorkerAssignment after releasing the +// worker rather than getting stuck in SUSPENDING, and the suspend fails. +func TestSuspendActor_Finalized_MissingDurableSnapshotClearsAssignment(t *testing.T) { ctx := context.Background() + st := newTestPersistence(t) + seedSuspendTemplate(t, ctx, st, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}) + w, atelet := newWireCaptureWorkflow(t, st) actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} records := crashRecords(t) - workerName := testWorkerUID("worker-pod-1") created := storetest.MustCreateActor(t, ctx, st, &ateapipb.Actor{ - Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), Status: &ateapipb.ActorStatus{ - State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, - AssignedNode: "node1", - WorkerAssignment: &ateapipb.WorkerAssignment{ - Worker: &ateapipb.ObjectRef{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - }, + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), LastAssignedGeneration: 1, + Snapshots: []*ateapipb.Snapshot{ + newDurableSnapshot(1, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", "snapshot-1", someActorSnapshotURI(t, testStorageLocation, "team-a", "snapshot-1"), ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING), + }, }, }) - if _, err := st.CreateWorker(ctx, &ateapipb.Worker{ - Metadata: &ateapipb.ResourceMetadata{Name: workerName}, - WorkerNamespace: "default", - WorkerPool: "pool1", - WorkerPod: "worker-pod-1", - WorkerPodUid: workerName, - NodeName: "node1", - Status: &ateapipb.WorkerStatus{}, - }); err != nil { - t.Fatalf("CreateWorker: %v", err) - } - seedAssignment(t, st, workerName, &ateapipb.ActorAssignment{ - Actor: &ateapipb.ObjectRef{Atespace: actorRef.Atespace, Name: actorRef.Name}, - ActorUid: created.GetMetadata().GetUid(), + seedWireWorker(t, ctx, st, assignmentOf(created)) + updateDuringCheckpoint(t, atelet, st, actorRef, func(s *ateapipb.ActorStatus) { + s.Snapshots = nil }) - w := &ActorWorkflow{store: st} - if _, err := w.ensureSuspendedFinalized(ctx, actorRef); apierror.Code(err) != codes.FailedPrecondition { - t.Fatalf("ensureSuspendedFinalized error = %v, want FailedPrecondition for the actor it crashed", err) + if _, err := w.SuspendActor(ctx, actorRef); apierror.Code(err) != codes.FailedPrecondition { + t.Fatalf("SuspendActor error = %v, want FailedPrecondition for the actor it crashed", err) } got, err := st.GetActor(ctx, actorRef) if err != nil { @@ -1031,3 +1084,337 @@ func TestEnsureSuspendedFinalized_MissingDurableSnapshotClearsAssignment(t *test t.Fatalf("got %d crash records, want 1", len(*records)) } } + +// suspendModes runs a table test against both suspend workflows: sync, whose +// snapshot is durable before it returns, and async, whose snapshot stays +// node-local. +var suspendModes = []struct { + name string + run func(*ActorWorkflow, context.Context, resources.ActorRef) (*ateapipb.Actor, error) +}{ + {"sync", (*ActorWorkflow).SuspendActor}, + {"async", (*ActorWorkflow).PauseActor}, +} + +// suspendOutcome is what one suspend workflow does to a seeded actor. Rows +// omit zero-valued fields, which the matrix still pins. +type suspendOutcome struct { + code codes.Code + atelet ateletCall + state ateapipb.ActorState + assignedNode string + // workerHeld is whether the worker wireTestAssignment names still holds + // the actor's assignment. + workerHeld bool + generation int32 + snapshots []snapshotOutcome +} + +// ateletCall is the suspend RPC atelet received; the zero value is none. +type ateletCall struct { + checkpoints, promotes int + // fidelity and object describe the request sent; uploadMode is set only + // on a Checkpoint. + fidelity ateletpb.SnapshotFidelity + uploadMode ateletpb.UploadMode + object bool +} + +// snapshotOutcome is the part of a stored snapshot the matrix pins. +type snapshotOutcome struct { + generation int32 + fidelity ateapipb.SnapshotFidelity + locality string + // durable is the durable copy's status, UNSPECIFIED when there is none. + durable ateapipb.SnapshotStorageStatus +} + +// TestSuspendAndPause_CharacteristicMatrix pins, for each actor shape either +// workflow can be called on, what SuspendActor (sync) and PauseActor (async) +// do: the code returned, the atelet RPC sent, the record left behind, and +// whether the actor's worker was released. +func TestSuspendAndPause_CharacteristicMatrix(t *testing.T) { + const node = "node-1" + volumes := ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + durable := func(gen int32, status ateapipb.SnapshotStorageStatus) *ateapipb.Snapshot { + name := fmt.Sprintf("snap-%d", gen) + return newDurableSnapshot(gen, ateapipb.SnapshotOwner_SNAPSHOT_OWNER_ACTOR, volumes, "", name, someActorSnapshotURI(t, testStorageLocation, "team-a", name), status) + } + local := func(gen int32, locality string) *ateapipb.Snapshot { + return newLocalSnapshot(gen, volumes, "", fmt.Sprintf("snap-%d", gen), locality) + } + completed := ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_COMPLETED + pending := ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_PENDING + noDurable := ateapipb.SnapshotStorageStatus_SNAPSHOT_STORAGE_STATUS_UNSPECIFIED + promoting := durable(1, pending) + promoting.Locality = node + + ateletVolumes := ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + immediate := ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE + checkpointSync := ateletCall{checkpoints: 1, fidelity: ateletVolumes, uploadMode: immediate, object: true} + checkpointAsync := ateletCall{checkpoints: 1, fidelity: ateletVolumes, uploadMode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED} + promote := ateletCall{promotes: 1, fidelity: ateletVolumes, object: true} + // rejected is a FailedPrecondition outcome: no atelet call, and the + // record left as want describes it. + rejected := func(want suspendOutcome) suspendOutcome { + want.code = codes.FailedPrecondition + return want + } + suspended := ateapipb.ActorState_ACTOR_STATE_SUSPENDED + + tests := []struct { + name string + // atespace defaults to team-a. + atespace string + status *ateapipb.ActorStatus + sync, async suspendOutcome + }{ + { + name: "RUNNING", + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RUNNING, AssignedNode: node, WorkerAssignment: wireTestAssignment()}, + sync: suspendOutcome{code: codes.OK, atelet: checkpointSync, state: suspended, generation: 1, snapshots: []snapshotOutcome{{1, volumes, "", completed}}}, + async: suspendOutcome{code: codes.OK, atelet: checkpointAsync, state: suspended, assignedNode: node, generation: 1, snapshots: []snapshotOutcome{{1, volumes, node, noDurable}}}, + }, + { + name: "RUNNING resumed from pause", + status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, AssignedNode: node, WorkerAssignment: wireTestAssignment(), + LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{local(1, node)}, + }, + sync: suspendOutcome{code: codes.OK, atelet: checkpointSync, state: suspended, generation: 2, snapshots: []snapshotOutcome{{2, volumes, "", completed}}}, + async: suspendOutcome{code: codes.OK, atelet: checkpointAsync, state: suspended, assignedNode: node, generation: 2, snapshots: []snapshotOutcome{{2, volumes, node, noDurable}}}, + }, + { + name: "RUNNING resumed from a durable snapshot", + status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, AssignedNode: node, WorkerAssignment: wireTestAssignment(), + LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{durable(1, completed)}, + }, + sync: suspendOutcome{code: codes.OK, atelet: checkpointSync, state: suspended, generation: 2, snapshots: []snapshotOutcome{{2, volumes, "", completed}}}, + async: suspendOutcome{code: codes.OK, atelet: checkpointAsync, state: suspended, assignedNode: node, generation: 2, snapshots: []snapshotOutcome{{1, volumes, "", completed}, {2, volumes, node, noDurable}}}, + }, + { + name: "SUSPENDED with a durable snapshot", + status: &ateapipb.ActorStatus{State: suspended, LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{durable(1, completed)}}, + sync: suspendOutcome{code: codes.OK, state: suspended, generation: 1, snapshots: []snapshotOutcome{{1, volumes, "", completed}}}, + async: rejected(suspendOutcome{state: suspended, generation: 1, snapshots: []snapshotOutcome{{1, volumes, "", completed}}}), + }, + { + name: "paused", + status: &ateapipb.ActorStatus{State: suspended, AssignedNode: node, LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{local(1, node)}}, + sync: suspendOutcome{code: codes.OK, atelet: promote, state: suspended, generation: 1, snapshots: []snapshotOutcome{{1, volumes, "", completed}}}, + async: suspendOutcome{code: codes.OK, state: suspended, assignedNode: node, generation: 1, snapshots: []snapshotOutcome{{1, volumes, node, noDurable}}}, + }, + { + name: "suspend from RUNNING in progress", + status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: node, WorkerAssignment: wireTestAssignment(), + LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{durable(1, pending)}, + }, + sync: suspendOutcome{code: codes.OK, atelet: checkpointSync, state: suspended, generation: 1, snapshots: []snapshotOutcome{{1, volumes, "", completed}}}, + async: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, assignedNode: node, workerHeld: true, generation: 1, snapshots: []snapshotOutcome{{1, volumes, "", pending}}}), + }, + { + name: "suspend from paused in progress", + status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: node, + LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{promoting}, + }, + sync: suspendOutcome{code: codes.OK, atelet: promote, state: suspended, generation: 1, snapshots: []snapshotOutcome{{1, volumes, "", completed}}}, + async: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, assignedNode: node, generation: 1, snapshots: []snapshotOutcome{{1, volumes, node, pending}}}), + }, + { + name: "pause in progress", + status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, AssignedNode: node, WorkerAssignment: wireTestAssignment(), + LastAssignedGeneration: 1, Snapshots: []*ateapipb.Snapshot{local(1, "")}, + }, + sync: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_SUSPENDING, assignedNode: node, workerHeld: true, generation: 1, snapshots: []snapshotOutcome{{1, volumes, "", noDurable}}}), + async: suspendOutcome{code: codes.OK, atelet: checkpointAsync, state: suspended, assignedNode: node, generation: 1, snapshots: []snapshotOutcome{{1, volumes, node, noDurable}}}, + }, + { + name: "RESUMING", + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RESUMING}, + sync: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_RESUMING}), + async: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_RESUMING}), + }, + { + name: "CRASHED", + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_CRASHED}, + sync: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_CRASHED}), + async: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_CRASHED}), + }, + { + name: "DELETING", + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_DELETING}, + sync: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_DELETING}), + async: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_DELETING}), + }, + { + // A golden actor commits MEMORY whatever the template prefers, and + // cannot be paused at all. + name: "golden RUNNING", + atespace: resources.GoldenActorAtespace, + status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_RUNNING, AssignedNode: node, WorkerAssignment: wireTestAssignment()}, + sync: suspendOutcome{code: codes.OK, atelet: ateletCall{checkpoints: 1, fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, uploadMode: immediate, object: true}, state: suspended, generation: 1, snapshots: []snapshotOutcome{{1, ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, "", completed}}}, + async: rejected(suspendOutcome{state: ateapipb.ActorState_ACTOR_STATE_RUNNING, assignedNode: node, workerHeld: true}), + }, + } + + for i, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + for _, mode := range suspendModes { + t.Run(mode.name, func(t *testing.T) { + want := tc.sync + if mode.name == "async" { + want = tc.async + } + atespace := tc.atespace + if atespace == "" { + atespace = "team-a" + } + // Each run gets its own store, so it can seed the one worker + // wireTestAssignment names. + ctx := context.Background() + persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation, PreferredFidelity: volumes}) + actorRef := resources.ActorRef{Atespace: atespace, Name: fmt.Sprintf("row%d-%s", i, mode.name)} + created := storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), + Status: proto.CloneOf(tc.status), + }) + var assignment *ateapipb.ActorAssignment + if tc.status.GetWorkerAssignment() != nil { + assignment = assignmentOf(created) + } + seedWireWorker(t, ctx, persistence, assignment) + w, atelet := newWireCaptureWorkflow(t, persistence) + + _, err := mode.run(w, ctx, actorRef) + stored := mustReadActor(t, ctx, persistence, actorRef) + got := suspendOutcome{ + code: apierror.Code(err), + atelet: describeAteletCall(atelet), + state: stored.GetStatus().GetState(), + assignedNode: stored.GetStatus().GetAssignedNode(), + workerHeld: firstAssignment(t, persistence, wireTestAssignment().GetWorker().GetName()) != nil, + generation: stored.GetStatus().GetLastAssignedGeneration(), + snapshots: describeSnapshots(stored.GetStatus()), + } + if diff := cmp.Diff(want, got, cmp.AllowUnexported(suspendOutcome{}, ateletCall{}, snapshotOutcome{})); diff != "" { + t.Errorf("outcome mismatch (-want +got):\n%s (err: %v)", diff, err) + } + if got.state == ateapipb.ActorState_ACTOR_STATE_SUSPENDED && stored.GetStatus().GetWorkerAssignment() != nil { + t.Errorf("WorkerAssignment = %v, want cleared on SUSPENDED", stored.GetStatus().GetWorkerAssignment()) + } + assertRequestNamesLatestSnapshot(t, atelet, stored) + }) + } + }) + } +} + +// TestSuspendAndPause_FinalizeWithoutAssignment pins where the two finalize +// steps differ when the DeleteWorker workflow crashes the actor, clearing its +// worker assignment, between the atelet call and finalize. Both sides pin +// bugs tracked in #2676: the suspend commits SUSPENDED over the crash, and +// the pause keeps the crash but returns OK for an actor left CRASHED. The fix +// changes the sync wantState to CRASHED and both codes to FailedPrecondition, +// replacing the err != nil fatal below. +func TestSuspendAndPause_FinalizeWithoutAssignment(t *testing.T) { + wantState := map[string]ateapipb.ActorState{ + "sync": ateapipb.ActorState_ACTOR_STATE_SUSPENDED, + "async": ateapipb.ActorState_ACTOR_STATE_CRASHED, + } + for _, mode := range suspendModes { + t.Run(mode.name, func(t *testing.T) { + ctx := context.Background() + persistence := newTestPersistence(t) + seedSuspendTemplate(t, ctx, persistence, &ateapipb.SnapshotConfig{StorageLocation: testStorageLocation}) + w, atelet := newWireCaptureWorkflow(t, persistence) + actorRef := resources.ActorRef{Atespace: "team-a", Name: "actor-1"} + storetest.MustCreateActor(t, ctx, persistence, &ateapipb.Actor{ + Metadata: &ateapipb.ResourceMetadata{Atespace: actorRef.Atespace, Name: actorRef.Name}, + ActorTemplate: suspendTestTemplateRef(), + Status: &ateapipb.ActorStatus{ + State: ateapipb.ActorState_ACTOR_STATE_RUNNING, + AssignedNode: "node-1", + WorkerAssignment: wireTestAssignment(), + }, + }) + updateDuringCheckpoint(t, atelet, persistence, actorRef, func(s *ateapipb.ActorStatus) { + s.State = ateapipb.ActorState_ACTOR_STATE_CRASHED + s.WorkerAssignment = nil + }) + + returned, err := mode.run(w, ctx, actorRef) + // #2676: both modes should return FailedPrecondition here. + if err != nil { + t.Fatalf("%s: %v", mode.name, err) + } + if got := returned.GetStatus().GetState(); got != wantState[mode.name] { + t.Errorf("returned state = %v, want %v", got, wantState[mode.name]) + } + if got := mustReadActor(t, ctx, persistence, actorRef).GetStatus().GetState(); got != wantState[mode.name] { + t.Errorf("stored state = %v, want %v", got, wantState[mode.name]) + } + }) + } +} + +// describeAteletCall summarizes the suspend RPCs atelet received: how many +// of each, and the fidelity, upload mode, and object destination requested. +func describeAteletCall(atelet *capturingAtelet) ateletCall { + var call ateletCall + call.checkpoints, call.promotes = atelet.calls() + var snap *ateletpb.Snapshot + if req := atelet.checkpointRequest(); req != nil { + snap = req.GetSnapshot() + call.uploadMode = req.GetUploadConfig().GetMode() + } else if req := atelet.promoteRequest(); req != nil { + snap = req.GetSnapshot() + } + call.fidelity = snap.GetFidelity() + call.object = snap.GetObject() != nil + return call +} + +// describeSnapshots summarizes each stored snapshot as its generation, +// fidelity, locality, and durable copy's status. +func describeSnapshots(status *ateapipb.ActorStatus) []snapshotOutcome { + var out []snapshotOutcome + for _, snap := range status.GetSnapshots() { + out = append(out, snapshotOutcome{ + generation: snap.GetGeneration(), + fidelity: snap.GetFidelity(), + locality: snap.GetLocality(), + durable: snap.GetDurableSnapshot().GetStatus(), + }) + } + return out +} + +// assertRequestNamesLatestSnapshot verifies the suspend RPC atelet received, +// if any, names the snapshot at the stored actor's latest generation, and that +// an object destination is that snapshot's durable copy. +func assertRequestNamesLatestSnapshot(t *testing.T, atelet *capturingAtelet, stored *ateapipb.Actor) { + t.Helper() + var snap *ateletpb.Snapshot + if req := atelet.checkpointRequest(); req != nil { + snap = req.GetSnapshot() + } else if req := atelet.promoteRequest(); req != nil { + snap = req.GetSnapshot() + } else { + return + } + latest := snapshotAtLatestGeneration(stored.GetStatus()) + if got, want := snap.GetSnapshotUuid(), latest.GetUuid(); got != want { + t.Errorf("request Snapshot.SnapshotUuid = %q, want the latest generation's %q", got, want) + } + if obj := snap.GetObject(); obj != nil { + if got, want := obj.GetSnapshotUri(), latest.GetDurableSnapshot().GetObject().GetSnapshotUri(); got != want { + t.Errorf("request Snapshot.Object.SnapshotUri = %q, want the latest generation's durable copy %q", got, want) + } + } +} diff --git a/cmd/ateapi/internal/controlapi/workflow_tag.go b/cmd/ateapi/internal/controlapi/workflow_tag.go index cc4d9a6e67..384e0084da 100644 --- a/cmd/ateapi/internal/controlapi/workflow_tag.go +++ b/cmd/ateapi/internal/controlapi/workflow_tag.go @@ -21,6 +21,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "google.golang.org/protobuf/proto" @@ -47,6 +48,7 @@ import ( // AlreadyExists. To retry, delete the tag, which collects whatever the failed // attempt stranded, and create it again. func (w *ActorWorkflow) TagActorSnapshot(ctx context.Context, tag *ateapipb.Tag) (*ateapipb.Tag, error) { + ctx = preview.ContextWithOptIn(ctx) actorRef := resources.ActorRefFromObjectRef(tag.GetSourceActor()) // Serializes against a suspend of the same actor, which would otherwise @@ -107,6 +109,7 @@ func (w *ActorWorkflow) TagActorSnapshot(ctx context.Context, tag *ateapipb.Tag) // and never suspended is still borrowing it and becomes unrecoverable. Do not // delete a tag while clones of it exist. func (w *ActorWorkflow) DeleteTag(ctx context.Context, tagRef resources.TagRef, precondition store.DeletePreconditions) (*ateapipb.Tag, error) { + ctx = preview.ContextWithOptIn(ctx) // Serializes against a create of the same tag, whose copy would otherwise // keep writing into the prefix this is collecting. ctx, lease, err := acquireTagLease(ctx, w.store, tagRef) diff --git a/cmd/ateapi/internal/controlapi/workflow_template_delete.go b/cmd/ateapi/internal/controlapi/workflow_template_delete.go index 2884de261e..91fce8361b 100644 --- a/cmd/ateapi/internal/controlapi/workflow_template_delete.go +++ b/cmd/ateapi/internal/controlapi/workflow_template_delete.go @@ -21,6 +21,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "google.golang.org/grpc/codes" @@ -30,6 +31,7 @@ import ( // golden actor and tag go first and the row last, so a failed attempt leaves // the row for a retry to rediscover them by. func (w *ActorWorkflow) DeleteActorTemplate(ctx context.Context, templateRef resources.ActorTemplateRef, precondition store.DeletePreconditions) (*ateapipb.ActorTemplate, error) { + ctx = preview.ContextWithOptIn(ctx) // Serializes against the reconciler, which creates the golden actor and // tag under the same lease. ctx, lease, err := acquireLease(ctx, w.store, "lease:actortemplate:"+templateRef.Atespace+":"+templateRef.Name, "ActorTemplate "+templateRef.String()) diff --git a/cmd/ateapi/internal/controlapi/workflow_test.go b/cmd/ateapi/internal/controlapi/workflow_test.go index c0e1876941..61c53f42b0 100644 --- a/cmd/ateapi/internal/controlapi/workflow_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_test.go @@ -47,10 +47,8 @@ func TestActorStateChangeRecords(t *testing.T) { { name: "suspend marks the actor suspending", seedState: ateapipb.ActorState_ACTOR_STATE_RUNNING, - transition: func(t *testing.T, w *ActorWorkflow, ref resources.ActorRef, actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) { - if _, err := w.ensureMarkedSuspending(context.Background(), ref, actor, tmpl); err != nil { - t.Fatalf("ensureMarkedSuspending: %v", err) - } + transition: func(t *testing.T, w *ActorWorkflow, ref resources.ActorRef, actor *ateapipb.Actor, _ *ateapipb.ActorTemplate) { + markOnly(t, w, ref, actor, w.SuspendActor) }, wantOp: ateattr.OperationSuspend, wantState: ateattr.ActorStateSuspending, @@ -59,10 +57,8 @@ func TestActorStateChangeRecords(t *testing.T) { { name: "pause marks the actor suspending", seedState: ateapipb.ActorState_ACTOR_STATE_RUNNING, - transition: func(t *testing.T, w *ActorWorkflow, ref resources.ActorRef, actor *ateapipb.Actor, tmpl *ateapipb.ActorTemplate) { - if _, err := w.ensureMarkedPausing(context.Background(), ref, actor, tmpl); err != nil { - t.Fatalf("ensureMarkedPausing: %v", err) - } + transition: func(t *testing.T, w *ActorWorkflow, ref resources.ActorRef, actor *ateapipb.Actor, _ *ateapipb.ActorTemplate) { + markOnly(t, w, ref, actor, w.PauseActor) }, wantOp: ateattr.OperationPause, wantState: ateattr.ActorStateSuspending, @@ -110,7 +106,7 @@ func TestActorStateChangeRecords(t *testing.T) { t.Fatalf("get template: %v", err) } - w := &ActorWorkflow{store: persistence} + w := newMarkOnlyWorkflow(persistence) tt.transition(t, w, actorRef, actor, tmpl) // The logs exporter is on, so the record goes to OTLP only. @@ -152,6 +148,23 @@ func TestActorStateChangeRecords(t *testing.T) { } } +// markOnly runs a suspend or pause on a RUNNING actor only as far as its mark +// step: the actor gets a worker to checkpoint on, but the atelet dial fails, +// so the SUSPENDING mark is the only transition the workflow commits. +func markOnly(t *testing.T, w *ActorWorkflow, ref resources.ActorRef, actor *ateapipb.Actor, run func(context.Context, resources.ActorRef) (*ateapipb.Actor, error)) { + t.Helper() + ctx := context.Background() + if _, err := w.store.UpdateActor(ctx, ref, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error { + toUpdate.Status.WorkerAssignment = wireTestAssignment() + return nil + }); err != nil { + t.Fatalf("UpdateActor: %v", err) + } + w.dialer = newDanglingDialer() + _, err := run(ctx, ref) + assertStoppedAtDial(t, err) +} + // TestActorCreatedRecord covers creation. An actor created and never resumed // makes no other transition, so without this it has no record at all, at any // retention. diff --git a/cmd/ateapi/internal/controlapi/workflow_testutil_test.go b/cmd/ateapi/internal/controlapi/workflow_testutil_test.go index e7a33bddc3..4e1f95c86d 100644 --- a/cmd/ateapi/internal/controlapi/workflow_testutil_test.go +++ b/cmd/ateapi/internal/controlapi/workflow_testutil_test.go @@ -24,11 +24,13 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/cmd/ateapi/internal/store/storetest" "github.com/agent-substrate/substrate/internal/apierror" + "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/objectstore/objectstoretest" "github.com/agent-substrate/substrate/internal/objectstoreplugin/objectstoreplugintest" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/google/uuid" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" "google.golang.org/grpc/codes" ) @@ -70,6 +72,17 @@ func newFinalizeWorkflow(persistence store.Interface) (*ActorWorkflow, *objectst return &ActorWorkflow{store: persistence, workflowDeadline: time.Minute, objectStorePlugin: objectstoreplugintest.ControlClient(objects)}, objects } +// newWireFinalizeWorkflow is newWireCaptureWorkflow with an in-memory object +// store behind the control object-store plugin, for the suspend tests that +// check which external snapshots a finalize collects. +func newWireFinalizeWorkflow(t *testing.T, persistence store.Interface) (*ActorWorkflow, *capturingAtelet, *objectstoretest.Fake) { + t.Helper() + w, atelet := newWireCaptureWorkflow(t, persistence) + objects := objectstoretest.New() + w.objectStorePlugin = objectstoreplugintest.ControlClient(objects) + return w, atelet, objects +} + // mustActorSnapshotURI builds the URI of a snapshot the actor took under // template's storage location, the way the suspend workflow does. func mustActorSnapshotURI(t *testing.T, template *ateapipb.ActorTemplate, actor *ateapipb.Actor, name string) resources.SnapshotURI { @@ -161,9 +174,9 @@ func withPausedShape(node string) func(*ateapipb.Actor) { } } -// withPauseInProgress seeds the shape a pause has between MarkPausing and -// FinalizePaused: a node-local snapshot at the latest generation that has no -// locality yet. Pair it with ACTOR_STATE_SUSPENDING. +// withPauseInProgress seeds the shape a pause has between marking the actor +// SUSPENDING and finalizing it: a node-local snapshot at the latest generation +// that has no locality yet. Pair it with ACTOR_STATE_SUSPENDING. func withPauseInProgress() func(*ateapipb.Actor) { return func(a *ateapipb.Actor) { a.Status.LastAssignedGeneration = 1 @@ -208,6 +221,121 @@ func assertPrerequisiteResult(t *testing.T, st ateapipb.ActorState, err error, w } } +// suspendTestTemplateRef names the template seedSuspendTemplate stores. +func suspendTestTemplateRef() *ateapipb.ObjectRef { + return &ateapipb.ObjectRef{Atespace: "ns", Name: "tmpl1"} +} + +// seedSuspendTemplate stores the ActorTemplate the suspend and pause workflow +// tests run their actors from, with the given snapshot config, and returns it +// as stored. +func seedSuspendTemplate(t *testing.T, ctx context.Context, st store.Interface, snapshotConfig *ateapipb.SnapshotConfig) *ateapipb.ActorTemplate { + t.Helper() + ref := suspendTestTemplateRef() + storetest.MustCreateAtespace(t, ctx, st, ref.GetAtespace()) + created, err := st.CreateActorTemplate(ctx, &ateapipb.ActorTemplate{ + Metadata: &ateapipb.ResourceMetadata{Atespace: ref.GetAtespace(), Name: ref.GetName()}, + SnapshotConfig: snapshotConfig, + SandboxConfig: &ateapipb.SandboxConfig{ + SandboxClass: "gvisor", + ConfigName: "gvisor", + }, + }) + if err != nil { + t.Fatalf("CreateActorTemplate: %v", err) + } + stored, err := st.GetActorTemplate(ctx, resources.ActorTemplateRefFromActorTemplate(created)) + if err != nil { + t.Fatalf("GetActorTemplate: %v", err) + } + return stored +} + +// seedWireWorker stores the Worker wireTestAssignment names and places +// assignment on it (none when nil), so a finalize step has a real assignment +// to release. +func seedWireWorker(t *testing.T, ctx context.Context, st store.Interface, assignment *ateapipb.ActorAssignment) { + t.Helper() + wa := wireTestAssignment() + if _, err := st.CreateWorker(ctx, &ateapipb.Worker{ + Metadata: &ateapipb.ResourceMetadata{Name: wa.GetWorker().GetName()}, + WorkerNamespace: wa.GetWorkerNamespace(), + WorkerPool: wa.GetWorkerPool(), + WorkerPod: wa.GetWorkerPod(), + WorkerPodUid: wa.GetWorkerPodUid(), + NodeName: wa.GetNodeName(), + Status: &ateapipb.WorkerStatus{}, + }); err != nil { + t.Fatalf("CreateWorker: %v", err) + } + seedAssignment(t, st, wa.GetWorker().GetName(), assignment) +} + +// assignmentOf is the worker assignment that places actor on a worker. +func assignmentOf(actor *ateapipb.Actor) *ateapipb.ActorAssignment { + return &ateapipb.ActorAssignment{ + Actor: &ateapipb.ObjectRef{Atespace: actor.GetMetadata().GetAtespace(), Name: actor.GetMetadata().GetName()}, + ActorUid: actor.GetMetadata().GetUid(), + } +} + +// newMarkOnlyWorkflow builds an ActorWorkflow whose atelet dial always fails, +// so a suspend or pause stops right after its mark step and leaves the actor +// as that step committed it. +func newMarkOnlyWorkflow(st store.Interface) *ActorWorkflow { + return &ActorWorkflow{store: st, dialer: newDanglingDialer(), workflowDeadline: time.Minute} +} + +// assertStoppedAtDial verifies a workflow built by newMarkOnlyWorkflow failed +// dialing atelet: it got past its mark step, and the failure is retryable. +func assertStoppedAtDial(t *testing.T, err error) { + t.Helper() + if !errors.Is(err, ErrNoAteletOnNode) { + t.Fatalf("err = %v, want ErrNoAteletOnNode from the atelet dial", err) + } +} + +// assertLifecycleFidelity verifies the one lifecycle-op datapoint reader +// collected carries want as its snapshot fidelity label. +func assertLifecycleFidelity(t *testing.T, reader *sdkmetric.ManualReader, want string) { + t.Helper() + dp := singleHistogramDP(t, reader, lifecycleOpDurationMetric) + if got, ok := attrString(dp, ateattr.SnapshotFidelityKey); !ok || got != want { + t.Errorf("lifecycle-op snapshot fidelity = %q (present %t), want %q", got, ok, want) + } +} + +// updateDuringCheckpoint makes atelet's Checkpoint mutate the stored actor +// before it returns, standing in for a writer that lands between a workflow's +// atelet call and its finalize step. +func updateDuringCheckpoint(t *testing.T, atelet *capturingAtelet, st store.Interface, actorRef resources.ActorRef, mutate func(*ateapipb.ActorStatus)) { + atelet.onCheckpoint = func(ctx context.Context) error { + actor, err := st.GetActor(ctx, actorRef) + if err != nil { + t.Errorf("GetActor during Checkpoint: %v", err) + return err + } + if _, err := st.UpdateActor(ctx, actorRef, store.PreconditionFrom(actor), func(toUpdate *ateapipb.Actor) error { + mutate(toUpdate.Status) + return nil + }); err != nil { + t.Errorf("UpdateActor during Checkpoint: %v", err) + return err + } + return nil + } +} + +// mustReadActor reads the stored actor. +func mustReadActor(t *testing.T, ctx context.Context, st store.Interface, actorRef resources.ActorRef) *ateapipb.Actor { + t.Helper() + actor, err := st.GetActor(ctx, actorRef) + if err != nil { + t.Fatalf("GetActor(%s): %v", actorRef, err) + } + return actor +} + // workerDeletedDuringReleaseStore runs the DeleteWorker workflow's release of // an actor right after a finalize step frees the actor's worker. That workflow // takes no actor lease, so its crash can land between the step's read of the diff --git a/cmd/ateapi/internal/controlapi/workflow_worker_delete.go b/cmd/ateapi/internal/controlapi/workflow_worker_delete.go index d3919f8987..eddf952f6c 100644 --- a/cmd/ateapi/internal/controlapi/workflow_worker_delete.go +++ b/cmd/ateapi/internal/controlapi/workflow_worker_delete.go @@ -23,6 +23,7 @@ import ( "github.com/agent-substrate/substrate/cmd/ateapi/internal/store" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) @@ -36,6 +37,7 @@ import ( // already did. An absent Worker is NOT_FOUND rather than success; idempotency // belongs to the caller, which knows whether that is the state it wanted. func (w *WorkerWorkflow) DeleteWorker(ctx context.Context, name string, precondition store.DeletePreconditions) (*ateapipb.Worker, error) { + ctx = preview.ContextWithOptIn(ctx) worker, err := w.loadWorkerForDelete(ctx, name) if err != nil { return nil, err diff --git a/cmd/ateapi/internal/controlapi/workload_spec.go b/cmd/ateapi/internal/controlapi/workload_spec.go index 749cb0b497..bed3c3f602 100644 --- a/cmd/ateapi/internal/controlapi/workload_spec.go +++ b/cmd/ateapi/internal/controlapi/workload_spec.go @@ -23,7 +23,7 @@ import ( ) // toAteletResources resolves a container's declared limits into the scalars -// atelet and the ateoms consume; everything downstream compares numbers. +// atelet and the workers consume; everything downstream compares numbers. // Returns nil when the container declares no limits, so the OCI spec stays // untouched for templates that do not use them. func toAteletResources(r *ateapipb.Resources) (*ateletpb.ResourceLimits, error) { @@ -109,6 +109,8 @@ func workloadSpecFromActorTemplate(actorTemplate *ateapipb.ActorTemplate, actor // TODO: order may be important for nested mounts. Also need to think about // nested mount support in general. + // This function is called from several places, so we do NOT check the + // ExternalVolumes gate here. Callers must check it, if appropriate. if err := appendExternalVolumes(workloadSpec, actorTemplate, actor, volumePublishContexts); err != nil { return nil, err } diff --git a/cmd/ateapi/internal/scheduling/scheduling_test.go b/cmd/ateapi/internal/scheduling/scheduling_test.go index e1775d15bf..28a0b5b73c 100644 --- a/cmd/ateapi/internal/scheduling/scheduling_test.go +++ b/cmd/ateapi/internal/scheduling/scheduling_test.go @@ -180,7 +180,7 @@ func TestSchedule(t *testing.T) { }, { // A worker that has not said it can hold more admits one, so this is - // the behavior every worker has until an ateom reports otherwise. + // the behavior every worker has until a worker reports otherwise. name: "unset actor capacity admits one actor", fleet: fleet{ worker("w-busy", "gvisor", "node-a", tierTwo, assigned("demo", "other")), diff --git a/cmd/ateapi/internal/workerservice/certificate.go b/cmd/ateapi/internal/workerservice/certificate.go index 457d8b2a13..c6737bf419 100644 --- a/cmd/ateapi/internal/workerservice/certificate.go +++ b/cmd/ateapi/internal/workerservice/certificate.go @@ -32,10 +32,10 @@ import ( "github.com/agent-substrate/substrate/pkg/proto/ateapipb" ) -// MintAteomActorCertificate mints a Substrate-issued SPIFFE certificate that asserts -// an ateom acting on behalf of a particular actor. -func (s *Server) MintAteomActorCertificate(ctx context.Context, req *ateapipb.MintAteomActorCertificateRequest) (*ateapipb.MintAteomActorCertificateResponse, error) { - if errs := apivalidation.ValidateMintAteomActorCertificateRequest(ctx, req); len(errs) > 0 { +// MintWorkerActorCertificate mints a Substrate-issued SPIFFE certificate that asserts +// a worker acting on behalf of a particular actor. +func (s *Server) MintWorkerActorCertificate(ctx context.Context, req *ateapipb.MintWorkerActorCertificateRequest) (*ateapipb.MintWorkerActorCertificateResponse, error) { + if errs := apivalidation.ValidateMintWorkerActorCertificateRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } @@ -76,7 +76,7 @@ func (s *Server) MintAteomActorCertificate(ctx context.Context, req *ateapipb.Mi Scheme: "spiffe", // TODO(identity): Must be configurable per-install, so that each install can set it to a unique value. Host: "substrate-actor.local", - Path: path.Join("ateom-for-actor", dbActor.GetMetadata().GetAtespace(), dbActor.GetMetadata().GetName()), + Path: path.Join("worker-for-actor", dbActor.GetMetadata().GetAtespace(), dbActor.GetMetadata().GetName()), }, }, NotBefore: time.Now().Add(-5 * time.Minute), @@ -105,7 +105,7 @@ func (s *Server) MintAteomActorCertificate(ctx context.Context, req *ateapipb.Mi return nil, fmt.Errorf("while signing certificate: %w", err) } - return &ateapipb.MintAteomActorCertificateResponse{ + return &ateapipb.MintWorkerActorCertificateResponse{ ActorCertificates: chain, }, nil } diff --git a/cmd/ateapi/internal/workerservice/certificate_test.go b/cmd/ateapi/internal/workerservice/certificate_test.go index 6982d389ef..5ff978b5aa 100644 --- a/cmd/ateapi/internal/workerservice/certificate_test.go +++ b/cmd/ateapi/internal/workerservice/certificate_test.go @@ -123,7 +123,7 @@ func newTestCAPool(t *testing.T) *localca.ConcretePool { } } -func TestMintAteomActorCertificate(t *testing.T) { +func TestMintWorkerActorCertificate(t *testing.T) { st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21") caPool := newTestCAPool(t) s := New(st, &fakeSuspender{}, testAteletSPIFFEID, caPool, "", nil) @@ -131,7 +131,7 @@ func TestMintAteomActorCertificate(t *testing.T) { csr, key := generateTestCSR(t) ctx := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1")) - resp, err := s.MintAteomActorCertificate(ctx, &ateapipb.MintAteomActorCertificateRequest{ + resp, err := s.MintWorkerActorCertificate(ctx, &ateapipb.MintWorkerActorCertificateRequest{ Actor: &ateapipb.ObjectRef{ Atespace: "team-a", Name: "my-actor", @@ -140,12 +140,12 @@ func TestMintAteomActorCertificate(t *testing.T) { CertificateSigningRequest: csr, }) if err != nil { - t.Fatalf("MintAteomActorCertificate() failed: %v", err) + t.Fatalf("MintWorkerActorCertificate() failed: %v", err) } chain := resp.GetActorCertificates() if len(chain) == 0 { - t.Fatal("MintAteomActorCertificate() returned empty chain") + t.Fatal("MintWorkerActorCertificate() returned empty chain") } leaf, err := x509.ParseCertificate(chain[0]) @@ -156,7 +156,7 @@ func TestMintAteomActorCertificate(t *testing.T) { t.Error("leaf public key does not match CSR key") } - wantURI := "spiffe://substrate-actor.local/ateom-for-actor/team-a/my-actor" + wantURI := "spiffe://substrate-actor.local/worker-for-actor/team-a/my-actor" if len(leaf.URIs) != 1 || leaf.URIs[0].String() != wantURI { t.Errorf("leaf URIs = %v, want [%s]", leaf.URIs, wantURI) } @@ -175,7 +175,7 @@ func TestMintAteomActorCertificate(t *testing.T) { } } -func TestMintAteomActorCertificate_Errors(t *testing.T) { +func TestMintWorkerActorCertificate_Errors(t *testing.T) { st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21") caPool := newTestCAPool(t) s := New(st, &fakeSuspender{}, testAteletSPIFFEID, caPool, "", nil) @@ -186,13 +186,13 @@ func TestMintAteomActorCertificate_Errors(t *testing.T) { tests := []struct { name string ctx context.Context - req *ateapipb.MintAteomActorCertificateRequest + req *ateapipb.MintWorkerActorCertificateRequest want codes.Code }{ { name: "unauthenticated", ctx: ateletauthtest.ContextWith(nil), - req: &ateapipb.MintAteomActorCertificateRequest{ + req: &ateapipb.MintWorkerActorCertificateRequest{ Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}, ActorUid: actor.GetMetadata().GetUid(), CertificateSigningRequest: csr, @@ -202,7 +202,7 @@ func TestMintAteomActorCertificate_Errors(t *testing.T) { { name: "missing actor", ctx: authed, - req: &ateapipb.MintAteomActorCertificateRequest{ + req: &ateapipb.MintWorkerActorCertificateRequest{ ActorUid: actor.GetMetadata().GetUid(), CertificateSigningRequest: csr, }, @@ -211,7 +211,7 @@ func TestMintAteomActorCertificate_Errors(t *testing.T) { { name: "missing actor uid", ctx: authed, - req: &ateapipb.MintAteomActorCertificateRequest{ + req: &ateapipb.MintWorkerActorCertificateRequest{ Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}, CertificateSigningRequest: csr, }, @@ -220,7 +220,7 @@ func TestMintAteomActorCertificate_Errors(t *testing.T) { { name: "missing csr", ctx: authed, - req: &ateapipb.MintAteomActorCertificateRequest{ + req: &ateapipb.MintWorkerActorCertificateRequest{ Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}, ActorUid: actor.GetMetadata().GetUid(), }, @@ -229,7 +229,7 @@ func TestMintAteomActorCertificate_Errors(t *testing.T) { { name: "actor not found", ctx: authed, - req: &ateapipb.MintAteomActorCertificateRequest{ + req: &ateapipb.MintWorkerActorCertificateRequest{ Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "nonexistent"}, ActorUid: actor.GetMetadata().GetUid(), CertificateSigningRequest: csr, @@ -239,7 +239,7 @@ func TestMintAteomActorCertificate_Errors(t *testing.T) { { name: "actor uid mismatch", ctx: authed, - req: &ateapipb.MintAteomActorCertificateRequest{ + req: &ateapipb.MintWorkerActorCertificateRequest{ Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}, ActorUid: "00000000-0000-0000-0000-000000000000", CertificateSigningRequest: csr, @@ -250,7 +250,7 @@ func TestMintAteomActorCertificate_Errors(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - _, err := s.MintAteomActorCertificate(tc.ctx, tc.req) + _, err := s.MintWorkerActorCertificate(tc.ctx, tc.req) if got := apierror.Code(err); got != tc.want { t.Errorf("code = %v (err %v), want %v", got, err, tc.want) } diff --git a/cmd/ateapi/internal/workerservice/jwt.go b/cmd/ateapi/internal/workerservice/jwt.go index 131fe5af37..be85b4666b 100644 --- a/cmd/ateapi/internal/workerservice/jwt.go +++ b/cmd/ateapi/internal/workerservice/jwt.go @@ -33,13 +33,13 @@ import ( "google.golang.org/protobuf/types/known/timestamppb" ) -// ateomActorJWTLifetime is the fixed lifetime of an ateom-for-actor JWT. It +// workerActorJWTLifetime is the fixed lifetime of a worker-for-actor JWT. It // is not chosen by the caller: the token's only consumers are node-side // plugins that exchange it for short-lived credentials. -const ateomActorJWTLifetime = time.Hour +const workerActorJWTLifetime = time.Hour -func (s *Server) MintAteomActorJwt(ctx context.Context, req *ateapipb.MintAteomActorJwtRequest) (*ateapipb.MintAteomActorJwtResponse, error) { - if errs := apivalidation.ValidateMintAteomActorJwtRequest(ctx, req); len(errs) > 0 { +func (s *Server) MintWorkerActorJwt(ctx context.Context, req *ateapipb.MintWorkerActorJwtRequest) (*ateapipb.MintWorkerActorJwtResponse, error) { + if errs := apivalidation.ValidateMintWorkerActorJwtRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } @@ -62,7 +62,7 @@ func (s *Server) MintAteomActorJwt(ctx context.Context, req *ateapipb.MintAteomA if err := s.checkActorOnNode(ctx, dbActor, caller.NodeName); err != nil { return nil, err } - // A stale UID is Aborted, as in MintAteomActorCertificate. + // A stale UID is Aborted, as in MintWorkerActorCertificate. if dbActor.GetMetadata().GetUid() != req.GetActorUid() { return nil, apierror.Aborted("conflict; actor has been deleted and recreated") } @@ -70,12 +70,12 @@ func (s *Server) MintAteomActorJwt(ctx context.Context, req *ateapipb.MintAteomA // JWT timestamps have one-second resolution. Truncating keeps expire_time // equal to the exp claim. now := time.Now().Truncate(time.Second) - expiresAt := now.Add(ateomActorJWTLifetime) + expiresAt := now.Add(workerActorJWTLifetime) meta := dbActor.GetMetadata() claims := &actoridjwt.Claims{ - Kind: actoridjwt.KindAteomForActor, + Kind: actoridjwt.KindWorkerForActor, Issuer: s.actorJWTIssuer, - Subject: fmt.Sprintf("%s/%s/%s", actoridjwt.KindAteomForActor.SubjectPrefix(), meta.GetAtespace(), meta.GetName()), + Subject: fmt.Sprintf("%s/%s/%s", actoridjwt.KindWorkerForActor.SubjectPrefix(), meta.GetAtespace(), meta.GetName()), Audiences: req.GetAudiences(), Expiration: expiresAt, NotBefore: now.Add(-5 * time.Minute), @@ -90,9 +90,9 @@ func (s *Server) MintAteomActorJwt(ctx context.Context, req *ateapipb.MintAteomA } token, err := s.actorIDJWTPool.SignJWT(claims) if err != nil { - return nil, fmt.Errorf("while signing ateom-for-actor JWT: %w", err) + return nil, fmt.Errorf("while signing worker-for-actor JWT: %w", err) } - return &ateapipb.MintAteomActorJwtResponse{ + return &ateapipb.MintWorkerActorJwtResponse{ ActorJwt: token, ExpireTime: timestamppb.New(expiresAt), }, nil @@ -108,7 +108,7 @@ func (s *Server) checkActorOnNode(ctx context.Context, actor *ateapipb.Actor, ca return err } if hostingNode == "" || hostingNode != callerNode { - slog.WarnContext(ctx, "Refusing an ateom-for-actor JWT for an actor the caller's node does not host", + slog.WarnContext(ctx, "Refusing a worker-for-actor JWT for an actor the caller's node does not host", slog.String("actor", resources.ActorRefFromActor(actor).String()), slog.String("hosting_node", hostingNode), slog.String("caller_node", callerNode)) diff --git a/cmd/ateapi/internal/workerservice/jwt_test.go b/cmd/ateapi/internal/workerservice/jwt_test.go index 0725f21306..4f17f2b459 100644 --- a/cmd/ateapi/internal/workerservice/jwt_test.go +++ b/cmd/ateapi/internal/workerservice/jwt_test.go @@ -61,26 +61,26 @@ func decodeJWTClaims(t *testing.T, token string) actoridjwt.WireClaims { return claims } -func TestMintAteomActorJwt(t *testing.T) { +func TestMintWorkerActorJwt(t *testing.T) { st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21") s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, testActorJWTIssuer, newTestJWTPool(t)) ctx := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1")) before := time.Now().Truncate(time.Second) - resp, err := s.MintAteomActorJwt(ctx, &ateapipb.MintAteomActorJwtRequest{ + resp, err := s.MintWorkerActorJwt(ctx, &ateapipb.MintWorkerActorJwtRequest{ Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}, ActorUid: actor.GetMetadata().GetUid(), Audiences: []string{"snapshot-plugin"}, }) if err != nil { - t.Fatalf("MintAteomActorJwt() failed: %v", err) + t.Fatalf("MintWorkerActorJwt() failed: %v", err) } claims := decodeJWTClaims(t, resp.GetActorJwt()) if claims.Issuer != testActorJWTIssuer { t.Errorf("iss = %q, want %q", claims.Issuer, testActorJWTIssuer) } - if want := "ateom-for-actor/team-a/my-actor"; claims.Subject != want { + if want := "worker-for-actor/team-a/my-actor"; claims.Subject != want { t.Errorf("sub = %q, want %q", claims.Subject, want) } var audiences []string @@ -91,11 +91,11 @@ func TestMintAteomActorJwt(t *testing.T) { t.Errorf("aud = %v, want [snapshot-plugin]", audiences) } kind, substrate, ok := claims.Substrate() - if !ok || kind != actoridjwt.KindAteomForActor { - t.Fatalf("token kind = %v (ok %v), want KindAteomForActor under ateom-for-actor.ate.dev", kind, ok) + if !ok || kind != actoridjwt.KindWorkerForActor { + t.Fatalf("token kind = %v (ok %v), want KindWorkerForActor under worker-for-actor.ate.dev", kind, ok) } if got, want := *substrate, (actoridjwt.WireSubstrateClaims{Atespace: "team-a", ActorName: "my-actor", ActorUID: actor.GetMetadata().GetUid()}); got != want { - t.Errorf("ateom-for-actor.ate.dev = %+v, want %+v", got, want) + t.Errorf("worker-for-actor.ate.dev = %+v, want %+v", got, want) } expiresAt := resp.GetExpireTime().AsTime() @@ -103,10 +103,10 @@ func TestMintAteomActorJwt(t *testing.T) { t.Errorf("exp = %v, expire_time = %v; want equal", got, expiresAt) } lifetime := expiresAt.Sub(time.Unix(int64(claims.IssuedAt), 0)) - if lifetime != ateomActorJWTLifetime { - t.Errorf("exp - iat = %v, want %v", lifetime, ateomActorJWTLifetime) + if lifetime != workerActorJWTLifetime { + t.Errorf("exp - iat = %v, want %v", lifetime, workerActorJWTLifetime) } - if expiresAt.Before(before.Add(ateomActorJWTLifetime)) { + if expiresAt.Before(before.Add(workerActorJWTLifetime)) { t.Errorf("expire_time = %v is earlier than %v + lifetime", expiresAt, before) } } @@ -123,13 +123,13 @@ func pausedStatus(state ateapipb.ActorState, node string) *ateapipb.ActorStatus } } -func TestMintAteomActorJwt_Errors(t *testing.T) { +func TestMintWorkerActorJwt_Errors(t *testing.T) { st, actor := newFakeStoreWithActor("team-a", "my-actor", "3b9f1e77-2c4d-4a80-91be-6d5c8f0a7e21") s := New(st, &fakeSuspender{}, testAteletSPIFFEID, nil, testActorJWTIssuer, newTestJWTPool(t)) authed := ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-1")) - valid := func() *ateapipb.MintAteomActorJwtRequest { - return &ateapipb.MintAteomActorJwtRequest{ + valid := func() *ateapipb.MintWorkerActorJwtRequest { + return &ateapipb.MintWorkerActorJwtRequest{ Actor: &ateapipb.ObjectRef{Atespace: "team-a", Name: "my-actor"}, ActorUid: actor.GetMetadata().GetUid(), Audiences: []string{"snapshot-plugin"}, @@ -154,7 +154,7 @@ func TestMintAteomActorJwt_Errors(t *testing.T) { tests := []struct { name string ctx context.Context - mutate func(*ateapipb.MintAteomActorJwtRequest) + mutate func(*ateapipb.MintWorkerActorJwtRequest) // status, when set, replaces the actor's status for the case. status *ateapipb.ActorStatus // assignments, when set, replaces the store's worker assignments @@ -278,25 +278,25 @@ func TestMintAteomActorJwt_Errors(t *testing.T) { { name: "missing actor", ctx: authed, - mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.Actor = nil }, + mutate: func(r *ateapipb.MintWorkerActorJwtRequest) { r.Actor = nil }, want: codes.InvalidArgument, }, { name: "missing actor uid", ctx: authed, - mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.ActorUid = "" }, + mutate: func(r *ateapipb.MintWorkerActorJwtRequest) { r.ActorUid = "" }, want: codes.InvalidArgument, }, { name: "missing audiences", ctx: authed, - mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.Audiences = nil }, + mutate: func(r *ateapipb.MintWorkerActorJwtRequest) { r.Audiences = nil }, want: codes.InvalidArgument, }, { name: "empty audience", ctx: authed, - mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.Audiences = []string{""} }, + mutate: func(r *ateapipb.MintWorkerActorJwtRequest) { r.Audiences = []string{""} }, want: codes.InvalidArgument, }, { @@ -310,7 +310,7 @@ func TestMintAteomActorJwt_Errors(t *testing.T) { // check answering Aborted: the node check runs first. name: "actor hosted on another node, stale uid", ctx: ateletauthtest.ContextWith(ateletauthtest.CertOn(t, "node-2")), - mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.ActorUid = "7d1c0b2a-9e4f-4c3d-8a6b-5f2e1d0c9b8a" }, + mutate: func(r *ateapipb.MintWorkerActorJwtRequest) { r.ActorUid = "7d1c0b2a-9e4f-4c3d-8a6b-5f2e1d0c9b8a" }, want: codes.NotFound, }, { @@ -373,15 +373,15 @@ func TestMintAteomActorJwt_Errors(t *testing.T) { { name: "actor not found", ctx: authed, - mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.Actor.Name = "nonexistent" }, + mutate: func(r *ateapipb.MintWorkerActorJwtRequest) { r.Actor.Name = "nonexistent" }, want: codes.NotFound, }, { // A recreated actor has a new UID, and a token must not be minted - // for the old incarnation. Aborted, as MintAteomActorCertificate answers. + // for the old incarnation. Aborted, as MintWorkerActorCertificate answers. name: "stale actor uid", ctx: authed, - mutate: func(r *ateapipb.MintAteomActorJwtRequest) { r.ActorUid = "7d1c0b2a-9e4f-4c3d-8a6b-5f2e1d0c9b8a" }, + mutate: func(r *ateapipb.MintWorkerActorJwtRequest) { r.ActorUid = "7d1c0b2a-9e4f-4c3d-8a6b-5f2e1d0c9b8a" }, want: codes.Aborted, }, } @@ -402,9 +402,9 @@ func TestMintAteomActorJwt_Errors(t *testing.T) { if tc.mutate != nil { tc.mutate(req) } - _, err := s.MintAteomActorJwt(tc.ctx, req) + _, err := s.MintWorkerActorJwt(tc.ctx, req) if got := apierror.Code(err); got != tc.want { - t.Errorf("MintAteomActorJwt() code = %v, want %v (err: %v)", got, tc.want, err) + t.Errorf("MintWorkerActorJwt() code = %v, want %v (err: %v)", got, tc.want, err) } }) } diff --git a/cmd/ateapi/internal/workerservice/register.go b/cmd/ateapi/internal/workerservice/register.go index 2ae394e889..e17f090548 100644 --- a/cmd/ateapi/internal/workerservice/register.go +++ b/cmd/ateapi/internal/workerservice/register.go @@ -109,7 +109,7 @@ func (s *Server) RegisterWorker(ctx context.Context, req *ateapipb.RegisterWorke // validateRuntimeClasses rejects a runtime of a class other than the Worker's // own. The class is fixed when the pool creates the Worker and names the -// sandbox its ateom runs, so a runtime of another class contradicts the +// sandbox its worker runs, so a runtime of another class contradicts the // recorded Worker state. A Worker recorded without a class has nothing for a // report to contradict. func validateRuntimeClasses(class string, req *ateapipb.RegisterWorkerRequest) error { diff --git a/cmd/ateapi/internal/workerservice/register_test.go b/cmd/ateapi/internal/workerservice/register_test.go index de99e677a1..2209d25668 100644 --- a/cmd/ateapi/internal/workerservice/register_test.go +++ b/cmd/ateapi/internal/workerservice/register_test.go @@ -57,7 +57,7 @@ func setRequest(actors int32) *ateapipb.RegisterWorkerRequest { } // The point of the whole path: a Worker moves from what it reported before to -// what its ateom reports now. +// what its worker reports now. func TestRegisterWorker(t *testing.T) { st, cleanup := storetest.SetupTestStore(t) defer cleanup() @@ -238,7 +238,7 @@ func TestRegisterWorker_OtherNodeIsNotFound(t *testing.T) { } } -// Re-sending the same capacity is not an update. An ateom reports once, but it +// Re-sending the same capacity is not an update. A worker reports once, but it // retries until accepted and reports again if it restarts, so a repeat must not // churn the Worker's version. func TestRegisterWorker_UnchangedDoesNotWrite(t *testing.T) { diff --git a/cmd/ateapi/internal/workerservice/service.go b/cmd/ateapi/internal/workerservice/service.go index 7ea05eb484..435b1113ff 100644 --- a/cmd/ateapi/internal/workerservice/service.go +++ b/cmd/ateapi/internal/workerservice/service.go @@ -44,7 +44,7 @@ type Server struct { actorIDCAPool localca.Pool // actorJWTIssuer is copied verbatim into the iss claim of every - // ateom-for-actor JWT, and actorIDJWTPool signs them. Both are the ones + // worker-for-actor JWT, and actorIDJWTPool signs them. Both are the ones // Control.MintActorJwt uses. actorJWTIssuer string actorIDJWTPool localjwtauthority.Pool diff --git a/cmd/ateapi/internal/workerservice/suspend_test.go b/cmd/ateapi/internal/workerservice/suspend_test.go index 408432e2ea..7fa4c4a679 100644 --- a/cmd/ateapi/internal/workerservice/suspend_test.go +++ b/cmd/ateapi/internal/workerservice/suspend_test.go @@ -93,7 +93,7 @@ func TestRequestActorSuspend(t *testing.T) { if diff := cmp.Diff(want, suspender.calls, protocmp.Transform()); diff != "" { t.Errorf("suspend calls mismatch (-want +got):\n%s", diff) } - // The caller gets the actor as the workflow left it, so an ateom can see + // The caller gets the actor as the workflow left it, so a worker can see // whether the suspend it asked for actually happened. if diff := cmp.Diff(suspended, got.GetActor(), protocmp.Transform()); diff != "" { t.Errorf("returned actor mismatch (-want +got):\n%s", diff) diff --git a/cmd/ateapi/main.go b/cmd/ateapi/main.go index 4293d38dd7..e7fb5b610d 100644 --- a/cmd/ateapi/main.go +++ b/cmd/ateapi/main.go @@ -41,6 +41,7 @@ import ( "github.com/agent-substrate/substrate/internal/oidcdiscovery" "github.com/agent-substrate/substrate/internal/oidcjwt" "github.com/agent-substrate/substrate/internal/pluginsocket" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/version" @@ -112,6 +113,7 @@ var ( showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") + previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.") ) func main() { @@ -128,6 +130,9 @@ func main() { if err := loadFlagsFromEnv(); err != nil { serverboot.Fatal(ctx, "Invalid PostgreSQL configuration", err) } + if err := preview.Init(*previewFlags...); err != nil { + serverboot.Fatal(ctx, "Invalid --preview", err) + } if err := rejectStorageEnv(); err != nil { serverboot.Fatal(ctx, "Storage settings moved to the objectstore-plugin sidecar", err) } @@ -333,7 +338,7 @@ func main() { templateReconciler := controlapi.NewActorTemplateReconciler(persistence, controlSrv, *templateResyncInterval) templateReconciler.Start(shutdownCtx) - // Crash the Actors lost when a Worker's ateom restarts. + // Crash the Actors lost when a Worker's worker restarts. workerAssignmentReconciler := controlapi.NewWorkerAssignmentReconciler(persistence, workerCache) workerAssignmentReconciler.Start(shutdownCtx) @@ -353,6 +358,7 @@ func main() { ateinterceptors.ServerUnaryInterceptor, authz.UnaryServerInterceptor(authorizer, *experimentalEnableAuthz), ateinterceptors.RejectUnknownFieldsUnaryInterceptor, + preview.UnaryServerInterceptor, } mux := grpc.NewServer( @@ -368,6 +374,7 @@ func main() { grpc.ChainUnaryInterceptor(unaryInterceptors...), grpc.ChainStreamInterceptor( apiauthn.StreamServerInterceptor(authCfg), + preview.StreamServerInterceptor, ), ) reflection.Register(mux) @@ -487,6 +494,7 @@ func logFlagValues(ctx context.Context) { slog.Duration("drain-delay", *drainDelay), slog.Duration("drain-timeout", *drainTimeout), slog.Duration("actor-workflow-deadline", *actorWorkflowDeadline), + slog.Any("preview", *previewFlags), ) } diff --git a/cmd/atecontroller/internal/controllers/networkpolicy_controller_test.go b/cmd/atecontroller/internal/controllers/networkpolicy_controller_test.go index f1ff854d10..9d939ea33c 100644 --- a/cmd/atecontroller/internal/controllers/networkpolicy_controller_test.go +++ b/cmd/atecontroller/internal/controllers/networkpolicy_controller_test.go @@ -27,7 +27,7 @@ import ( func TestWorkerPoolCreatesNetworkPolicy(t *testing.T) { ctx := t.Context() - wp := makeWorkerPool("test-netpolicy-create", "default", 2, "ateom:v1") + wp := makeWorkerPool("test-netpolicy-create", "default", 2, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } diff --git a/cmd/atecontroller/internal/controllers/sandboxconfig_protection_controller_test.go b/cmd/atecontroller/internal/controllers/sandboxconfig_protection_controller_test.go index fd65d90f23..bd2bf5648f 100644 --- a/cmd/atecontroller/internal/controllers/sandboxconfig_protection_controller_test.go +++ b/cmd/atecontroller/internal/controllers/sandboxconfig_protection_controller_test.go @@ -46,7 +46,7 @@ func TestSandboxConfigProtectionReconcile(t *testing.T) { // using returns a pool whose status has one sandbox class per name, each // using the named config, or no config when the name is "". using := func(names ...string) *atev1alpha1.WorkerPool { - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") for _, name := range names { s := atev1alpha1.WorkerPoolSandboxClassStatus{Name: atev1alpha1.SandboxClassGvisor} if name != "" { diff --git a/cmd/atecontroller/internal/controllers/workerpool_apply.go b/cmd/atecontroller/internal/controllers/workerpool_apply.go index c282f71bdb..728e106a9a 100644 --- a/cmd/atecontroller/internal/controllers/workerpool_apply.go +++ b/cmd/atecontroller/internal/controllers/workerpool_apply.go @@ -16,6 +16,7 @@ package controllers import ( "slices" + "strings" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" @@ -25,16 +26,17 @@ import ( corev1ac "k8s.io/client-go/applyconfigurations/core/v1" metav1ac "k8s.io/client-go/applyconfigurations/meta/v1" - "github.com/agent-substrate/substrate/internal/ateom" "github.com/agent-substrate/substrate/internal/deviceplugin" "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/nodepath" + "github.com/agent-substrate/substrate/internal/preview" + "github.com/agent-substrate/substrate/internal/workercapacity" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" ) -// ateomOTelResourceAttributes mirrors atelet.yaml; service.instance.id is the pod +// workerOTelResourceAttributes mirrors atelet.yaml; service.instance.id is the pod // uid so each worker pod is a distinct telemetry source. -const ateomOTelResourceAttributes = "k8s.namespace.name=$(POD_NAMESPACE),k8s.pod.name=$(POD_NAME),k8s.pod.uid=$(POD_UID),k8s.node.name=$(NODE_NAME),service.instance.id=$(POD_UID)" +const workerOTelResourceAttributes = "k8s.namespace.name=$(POD_NAMESPACE),k8s.pod.name=$(POD_NAME),k8s.pod.uid=$(POD_UID),k8s.node.name=$(NODE_NAME),service.instance.id=$(POD_UID)" // workerTerminationGracePeriodSeconds is the hardcoded pod termination grace // period for worker pods (60 minutes). @@ -55,17 +57,17 @@ const ( workerRolloutProgressDeadlineSeconds = int32(4800) ) -// ateomOTelSettings is the telemetry configuration propagated to ateom worker +// workerOTelSettings is the telemetry configuration propagated to worker // pods. A zero value leaves the pods without telemetry env. -type ateomOTelSettings struct { - // Endpoint is the OTLP collector address. Empty disables ateom telemetry +type workerOTelSettings struct { + // Endpoint is the OTLP collector address. Empty disables worker telemetry // entirely, so the other fields are ignored. Endpoint string // MetricExportInterval overrides the SDK's 60s PeriodicReader interval. It is // the raw OTEL_METRIC_EXPORT_INTERVAL value, i.e. whole milliseconds; the SDK // falls back to its default when it does not parse. Empty keeps the default. // - // ateom registers no instruments of its own, so its only telemetry is + // worker registers no instruments of its own, so its only telemetry is // otelgrpc's rpc.server.* and it stays invisible to the collector until the // first export tick fires. Shortening the interval is what keeps that // startup gap inside an e2e budget; production leaves it unset. @@ -74,13 +76,13 @@ type ateomOTelSettings struct { // whole-millisecond form as MetricExportInterval. Empty keeps the default. MetricExportTimeout string // TracesSampler and TracesSamplerArg are the raw OTEL_TRACES_SAMPLER and - // OTEL_TRACES_SAMPLER_ARG values, passed through untouched: ateom's own + // OTEL_TRACES_SAMPLER_ARG values, passed through untouched: worker's own // serverboot resolution validates them. Empty sampler keeps the worker's // default and drops the arg, which is dead config on its own. TracesSampler string TracesSamplerArg string // LogsExporter is the raw OTEL_LOGS_EXPORTER value. otlp sends the usage - // records over OTLP instead of stdout; empty keeps ateom's default, none. + // records over OTLP instead of stdout; empty keeps worker's default, none. LogsExporter string } @@ -92,14 +94,14 @@ const ( atunnelIdentityMountPath = "/run/podidentity.podcert.ate.dev" atunnelEgressTrustVolume = "atunnel-egress-trust" atunnelEgressTrustMountPath = "/run/servicedns.podcert.ate.dev" - ateomCapacityVolume = "ateom-capacity" + workerCapacityVolume = "worker-capacity" ) // buildDeploymentApplyConfig constructs the SSA apply configuration for the // Deployment managed by a WorkerPool. Only fields owned by this controller // are declared here. otel, when it carries an endpoint, is propagated to the -// ateom container so it pushes telemetry to that collector. -func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettings, systemNamespace, ateletServiceAccount, routerServiceAccount, dnsEgressPolicy string) *appsv1ac.DeploymentApplyConfiguration { +// worker container so it pushes telemetry to that collector. +func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel workerOTelSettings, systemNamespace, ateletServiceAccount, routerServiceAccount, dnsEgressPolicy string) *appsv1ac.DeploymentApplyConfiguration { labels := map[string]string{} annotations := map[string]string{} if wp.Spec.Template != nil { @@ -120,15 +122,15 @@ func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettin "--atunnel-trust-bundle=" + atunnelIdentityMountPath + "/trust-bundle.pem", // The peers atunnel authenticates live in substrate's namespace, not // the worker's, so the controller passes their identities rather than - // letting ateom assume the default install. --atunnel-client-identity - // has been accepted by every ateom that carries this controller's + // letting worker assume the default install. --atunnel-client-identity + // has been accepted by every worker that carries this controller's // contemporaries, so it is always safe to pass. "--atunnel-client-identity=" + installdefaults.SPIFFEID(systemNamespace, routerServiceAccount), } - // --atunnel-broker-identity is newer than the oldest ateom a WorkerPool - // may still pin, and an ateom exits on an unrecognized flag. An ateom - // without the flag hardcodes the canonical identity, and an ateom with it + // --atunnel-broker-identity is newer than the oldest worker a WorkerPool + // may still pin, and a worker exits on an unrecognized flag. A worker + // without the flag hardcodes the canonical identity, and a worker with it // defaults to the same, so omitting the flag when it carries that value is // equivalent for both. A relocated or renamed install passes something // else and needs an image new enough to accept it, which it necessarily @@ -143,8 +145,23 @@ func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettin "--dns-egress-policy="+dnsEgressPolicy, ) + // worker gets preview features when the controller has them. + enabledGates := func() []string { + ret := []string{} + for k, v := range preview.AsMap() { + if v { + ret = append(ret, k) + } + } + // Map order is random; a stable arg avoids needless rollouts. + slices.Sort(ret) + return ret + } + previewArg := "--preview=" + strings.Join(enabledGates(), ",") + args = append(args, previewArg) + containerAC := corev1ac.Container(). - WithName("ateom"). + WithName("worker"). WithImage(wp.Spec.WorkerImage). WithArgs(args...). WithPorts(corev1ac.ContainerPort(). @@ -163,15 +180,15 @@ func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettin WithHTTPGet(corev1ac.HTTPGetAction(). WithPath("/readyz"). WithPort(intstr.FromString("readyz")))). - WithSecurityContext(ateomSecurityContext(wp.Spec.DefaultSandboxClass())). - WithEnv(ateomContainerEnv(otel)...). + WithSecurityContext(workerSecurityContext(wp.Spec.DefaultSandboxClass())). + WithEnv(workerContainerEnv(otel)...). WithVolumeMounts( corev1ac.VolumeMount(). - WithName(ateomCapacityVolume). - WithMountPath(ateom.CapacityMountPath). + WithName(workerCapacityVolume). + WithMountPath(workercapacity.MountPath). WithReadOnly(true), corev1ac.VolumeMount(). - WithName("run-ateom"). + WithName("ate-base"). WithMountPath(nodepath.BasePath). WithMountPropagation(corev1.MountPropagationHostToContainer), corev1ac.VolumeMount(). @@ -190,14 +207,14 @@ func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettin WithRunAsGroup(0)). WithVolumes( corev1ac.Volume(). - WithName(ateomCapacityVolume). + WithName(workerCapacityVolume). WithDownwardAPI(corev1ac.DownwardAPIVolumeSource(). WithItems( - resourceFieldRefFile(ateom.CPULimitFile, "limits.cpu", milliCores), - resourceFieldRefFile(ateom.MemoryLimitFile, "limits.memory", wholeBytes), + resourceFieldRefFile(workercapacity.CPULimitFile, "limits.cpu", milliCores), + resourceFieldRefFile(workercapacity.MemoryLimitFile, "limits.memory", wholeBytes), )), corev1ac.Volume(). - WithName("run-ateom"). + WithName("ate-base"). WithHostPath(corev1ac.HostPathVolumeSource(). WithPath(nodepath.BasePath). WithType(corev1.HostPathDirectoryOrCreate)), @@ -264,10 +281,10 @@ func buildDeploymentApplyConfig(wp *atev1alpha1.WorkerPool, otel ateomOTelSettin WithSpec(podSpecAC))) } -// ateomContainerEnv adds the OTLP endpoint and resource identity only when +// workerContainerEnv adds the OTLP endpoint and resource identity only when // telemetry is configured. Every ref precedes OTEL_RESOURCE_ATTRIBUTES so its // $(...) substitutions resolve. -func ateomContainerEnv(otel ateomOTelSettings) []*corev1ac.EnvVarApplyConfiguration { +func workerContainerEnv(otel workerOTelSettings) []*corev1ac.EnvVarApplyConfiguration { // The pool pair labels every usage record, telemetry export or not. A // worker pod runs in its WorkerPool's namespace. envs := []*corev1ac.EnvVarApplyConfiguration{ @@ -282,7 +299,7 @@ func ateomContainerEnv(otel ateomOTelSettings) []*corev1ac.EnvVarApplyConfigurat fieldRefEnv("POD_NAME", "metadata.name"), fieldRefEnv("NODE_NAME", "spec.nodeName"), corev1ac.EnvVar().WithName("OTEL_EXPORTER_OTLP_ENDPOINT").WithValue(otel.Endpoint), - corev1ac.EnvVar().WithName("OTEL_RESOURCE_ATTRIBUTES").WithValue(ateomOTelResourceAttributes), + corev1ac.EnvVar().WithName("OTEL_RESOURCE_ATTRIBUTES").WithValue(workerOTelResourceAttributes), ) if otel.MetricExportInterval != "" { envs = append(envs, corev1ac.EnvVar(). @@ -325,7 +342,7 @@ func resourceFieldRefFile(path, resourceName, divisor string) *corev1ac.Downward return corev1ac.DownwardAPIVolumeFile(). WithPath(path). WithResourceFieldRef(corev1ac.ResourceFieldSelector(). - WithContainerName("ateom"). + WithContainerName("worker"). WithResource(resourceName). WithDivisor(resource.MustParse(divisor))) } @@ -338,7 +355,7 @@ func fieldRefEnv(name, fieldPath string) *corev1ac.EnvVarApplyConfiguration { WithFieldPath(fieldPath))) } -// ateomGvisorCapabilities is the capability set an unprivileged gVisor worker +// workerGvisorCapabilities is the capability set an unprivileged gVisor worker // needs. runsc's gofer maps a full-range identity in a user namespace // (SETUID/SETGID/SETPCAP/SETFCAP), the sandbox pivots root and traces the // application (SYS_ADMIN/SYS_CHROOT/SYS_PTRACE), actor networking programs the @@ -346,14 +363,14 @@ func fieldRefEnv(name, fieldPath string) *corev1ac.EnvVarApplyConfiguration { // and device nodes created as root over image-owned trees // (DAC_OVERRIDE/FOWNER/CHOWN/MKNOD). This replaces the former privileged worker; // seccomp stays at the runtime default, but AppArmor must be Unconfined (see -// ateomSecurityContext) since runsc's own mounts trip the default profile. -var ateomGvisorCapabilities = []corev1.Capability{ +// workerSecurityContext) since runsc's own mounts trip the default profile. +var workerGvisorCapabilities = []corev1.Capability{ "NET_ADMIN", "SYS_ADMIN", "SYS_CHROOT", "SYS_PTRACE", "SETUID", "SETGID", "SETPCAP", "DAC_OVERRIDE", "FOWNER", "CHOWN", "MKNOD", "NET_RAW", "SETFCAP", } -// ateomMicroVMCapabilities is the capability set an unprivileged micro-VM worker +// workerMicroVMCapabilities is the capability set an unprivileged micro-VM worker // needs: the gVisor set, which covers the same worker-side work, plus FSETID and // DAC_READ_SEARCH for virtiofsd. virtiofsd re-applies a fixed set to its // sandboxed child, and the kernel refuses to raise a capability the bounding set @@ -362,15 +379,15 @@ var ateomGvisorCapabilities = []corev1.Capability{ // // The hypervisor devices are not capabilities: they come from atelet's device // plugin (see maybeApplyMicroVMPodShape). -var ateomMicroVMCapabilities = slices.Concat(ateomGvisorCapabilities, []corev1.Capability{ +var workerMicroVMCapabilities = slices.Concat(workerGvisorCapabilities, []corev1.Capability{ "FSETID", "DAC_READ_SEARCH", }) -// ateomSecurityContext returns the ateom container security context for a sandbox +// workerSecurityContext returns the worker container security context for a sandbox // class. Neither class runs privileged; they differ in their capability set. // Both declare seccomp Unconfined because their sandbox child pivot_root()s, // which the default profile denies. An empty class defaults to gVisor. -func ateomSecurityContext(class atev1alpha1.SandboxClass) *corev1ac.SecurityContextApplyConfiguration { +func workerSecurityContext(class atev1alpha1.SandboxClass) *corev1ac.SecurityContextApplyConfiguration { // Both runtimes mount inside the worker — runsc pivots root and the worker // remounts /sys/fs/cgroup to nest per-actor cgroups; the micro-VM worker // shares /run/kata-containers and remounts /proc/sys — and the default @@ -390,14 +407,14 @@ func ateomSecurityContext(class atev1alpha1.SandboxClass) *corev1ac.SecurityCont // which pivot_root()s — a syscall the profile denies whatever capabilities // the worker holds. The trade favors containing the guest: virtiofsd parses // requests from it, so its namespaces are part of the guest-to-host - // boundary, while the pod-wide profile mostly confines ateom, which we + // boundary, while the pod-wide profile mostly confines worker, which we // trust. Both guest-facing processes confine themselves anyway, virtiofsd // with its own seccomp filter and nine capabilities, cloud-hypervisor with // per-thread filters. return sc. WithCapabilities(corev1ac.Capabilities(). WithDrop("ALL"). - WithAdd(ateomMicroVMCapabilities...)). + WithAdd(workerMicroVMCapabilities...)). WithSeccompProfile(corev1ac.SeccompProfile(). WithType(corev1.SeccompProfileTypeUnconfined)) } @@ -409,7 +426,7 @@ func ateomSecurityContext(class atev1alpha1.SandboxClass) *corev1ac.SecurityCont return sc. WithCapabilities(corev1ac.Capabilities(). WithDrop("ALL"). - WithAdd(ateomGvisorCapabilities...)). + WithAdd(workerGvisorCapabilities...)). WithSeccompProfile(corev1ac.SeccompProfile(). WithType(corev1.SeccompProfileTypeUnconfined)) } diff --git a/cmd/atecontroller/internal/controllers/workerpool_apply_test.go b/cmd/atecontroller/internal/controllers/workerpool_apply_test.go index 94bc08954e..237811ba14 100644 --- a/cmd/atecontroller/internal/controllers/workerpool_apply_test.go +++ b/cmd/atecontroller/internal/controllers/workerpool_apply_test.go @@ -30,11 +30,12 @@ import ( metav1ac "k8s.io/client-go/applyconfigurations/meta/v1" "k8s.io/utils/ptr" - "github.com/agent-substrate/substrate/internal/ateom" "github.com/agent-substrate/substrate/internal/deviceplugin" "github.com/agent-substrate/substrate/internal/egresspolicy" "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/nodepath" + "github.com/agent-substrate/substrate/internal/preview" + "github.com/agent-substrate/substrate/internal/workercapacity" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" ) @@ -223,7 +224,7 @@ func TestBuildDeploymentApplyConfig(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - got := buildDeploymentApplyConfig(tt.wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) + got := buildDeploymentApplyConfig(tt.wp, workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) if diff := cmp.Diff(tt.want, got); diff != "" { t.Fatalf("buildDeploymentApplyConfig() mismatch (-want +got):\n%s", diff) } @@ -243,7 +244,7 @@ func TestBuildDeploymentApplyConfigMetadata(t *testing.T) { }, }) - got := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) + got := buildDeploymentApplyConfig(wp, workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) wantLabels := map[string]string{ "project": "agent-substrate", "team": "compute", @@ -301,7 +302,7 @@ func TestSandboxClassToleration(t *testing.T) { }}, }) wp.Spec.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClass{{Name: tt.class}} - ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec + ps := buildDeploymentApplyConfig(wp, workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec var classValues []string hasTemplateToleration := false @@ -349,7 +350,7 @@ func TestMicroVMPodShape(t *testing.T) { t.Run(tt.name, func(t *testing.T) { wp := testWorkerPoolApplyConfig(nil) wp.Spec.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClass{{Name: tt.class}} - ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec + ps := buildDeploymentApplyConfig(wp, workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec // /dev/kvm must come from the device plugin, never a hostPath: a // hostPath mount carries no cgroup device allow rule, and the @@ -376,7 +377,7 @@ func TestMicroVMPodShape(t *testing.T) { } } if hasTunMount && !containerMountsPath(ps.Containers[0], tunDevicePath) { - t.Errorf("%s is a pod volume but the ateom container does not mount it", tunDevicePath) + t.Errorf("%s is a pod volume but the worker container does not mount it", tunDevicePath) } hasDeviceRequest := true @@ -436,7 +437,7 @@ func TestMicroVMDeviceRequestsPreserveTemplateResources(t *testing.T) { }, }) wp.Spec.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClass{{Name: atev1alpha1.SandboxClassMicroVM}} - c := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec.Containers[0] + c := buildDeploymentApplyConfig(wp, workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec.Containers[0] if got, ok := deviceLimit(c, string(corev1.ResourceMemory)); !ok || got != "2Gi" { t.Errorf("memory limit = %q (present=%v), want 2Gi", got, ok) @@ -446,11 +447,11 @@ func TestMicroVMDeviceRequestsPreserveTemplateResources(t *testing.T) { } } -// TestAteomSecurityContextByClass asserts no worker runs privileged: every class +// TestWorkerSecurityContextByClass asserts no worker runs privileged: every class // drops ALL capabilities and adds back an explicit set. Only the micro-VM class // gives up the runtime's default seccomp profile, and only so virtiofsd can keep // its own sandbox. An empty class defaults to gVisor. -func TestAteomSecurityContextByClass(t *testing.T) { +func TestWorkerSecurityContextByClass(t *testing.T) { tests := []struct { name string class atev1alpha1.SandboxClass @@ -459,13 +460,13 @@ func TestAteomSecurityContextByClass(t *testing.T) { // virtiofsd's sandbox pivot_root(), which the default profile denies. wantSeccompUnconfined bool }{ - {"gvisor default", "", ateomGvisorCapabilities, true}, - {"gvisor explicit", atev1alpha1.SandboxClassGvisor, ateomGvisorCapabilities, true}, - {"microvm", atev1alpha1.SandboxClassMicroVM, ateomMicroVMCapabilities, true}, + {"gvisor default", "", workerGvisorCapabilities, true}, + {"gvisor explicit", atev1alpha1.SandboxClassGvisor, workerGvisorCapabilities, true}, + {"microvm", atev1alpha1.SandboxClassMicroVM, workerMicroVMCapabilities, true}, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - sc := ateomSecurityContext(tt.class) + sc := workerSecurityContext(tt.class) if sc.Privileged == nil || *sc.Privileged { t.Errorf("Privileged = %v, want false for every class", sc.Privileged) } @@ -501,7 +502,7 @@ func TestAteomSecurityContextByClass(t *testing.T) { // TestTerminationGracePeriodSeconds asserts the pod's grace period is hardcoded to 3600s. func TestTerminationGracePeriodSeconds(t *testing.T) { wp := testWorkerPoolApplyConfig(nil) - ps := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec + ps := buildDeploymentApplyConfig(wp, workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec.Template.Spec if ps.TerminationGracePeriodSeconds == nil { t.Fatalf("TerminationGracePeriodSeconds not set") } @@ -515,7 +516,7 @@ func TestTerminationGracePeriodSeconds(t *testing.T) { // actors' drain window. func TestRolloutStrategy(t *testing.T) { wp := testWorkerPoolApplyConfig(nil) - spec := buildDeploymentApplyConfig(wp, ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec + spec := buildDeploymentApplyConfig(wp, workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel).Spec if spec.Strategy == nil || spec.Strategy.RollingUpdate == nil { t.Fatalf("Strategy.RollingUpdate not set") } @@ -535,7 +536,7 @@ func TestRolloutStrategy(t *testing.T) { } // TestBuildDeploymentApplyConfigOTelEndpoint asserts the OTLP endpoint and the -// resource identity are set on the ateom container only when an endpoint is +// resource identity are set on the worker container only when an endpoint is // configured, and that every ref the value substitutes is declared ahead of it. func TestBuildDeploymentApplyConfigOTelEndpoint(t *testing.T) { const endpoint = "http://collector.otel-system.svc:4317" @@ -549,7 +550,7 @@ func TestBuildDeploymentApplyConfigOTelEndpoint(t *testing.T) { } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{Endpoint: tt.endpoint}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), workerOTelSettings{Endpoint: tt.endpoint}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] env := envByName(c.Env) @@ -597,8 +598,8 @@ func TestBuildDeploymentApplyConfigOTelEndpoint(t *testing.T) { } // TestBuildDeploymentApplyConfigMetricExportTuning asserts the export interval and -// per-export timeout reach the ateom container only when each is set alongside an -// endpoint. ateom is invisible to the collector until its first successful export +// per-export timeout reach the worker container only when each is set alongside an +// endpoint. worker is invisible to the collector until its first successful export // tick, so the kind stack shortens the SDK's 60s interval to keep that gap inside // the e2e budget, and its 30s timeout so a failing tick cannot swallow three // shortened intervals. @@ -606,32 +607,32 @@ func TestBuildDeploymentApplyConfigMetricExportTuning(t *testing.T) { const endpoint = "http://collector.otel-system.svc:4317" tests := []struct { name string - otel ateomOTelSettings + otel workerOTelSettings want map[string]string // env name -> value; absent key means must not be set }{ { name: "unset keeps SDK defaults", - otel: ateomOTelSettings{Endpoint: endpoint}, + otel: workerOTelSettings{Endpoint: endpoint}, want: nil, }, { name: "both set with endpoint", - otel: ateomOTelSettings{Endpoint: endpoint, MetricExportInterval: "10000", MetricExportTimeout: "10000"}, + otel: workerOTelSettings{Endpoint: endpoint, MetricExportInterval: "10000", MetricExportTimeout: "10000"}, want: map[string]string{"OTEL_METRIC_EXPORT_INTERVAL": "10000", "OTEL_METRIC_EXPORT_TIMEOUT": "10000"}, }, { name: "interval alone", - otel: ateomOTelSettings{Endpoint: endpoint, MetricExportInterval: "10000"}, + otel: workerOTelSettings{Endpoint: endpoint, MetricExportInterval: "10000"}, want: map[string]string{"OTEL_METRIC_EXPORT_INTERVAL": "10000"}, }, { name: "timeout alone", - otel: ateomOTelSettings{Endpoint: endpoint, MetricExportTimeout: "10000"}, + otel: workerOTelSettings{Endpoint: endpoint, MetricExportTimeout: "10000"}, want: map[string]string{"OTEL_METRIC_EXPORT_TIMEOUT": "10000"}, }, { name: "ignored without endpoint", - otel: ateomOTelSettings{MetricExportInterval: "10000", MetricExportTimeout: "10000"}, + otel: workerOTelSettings{MetricExportInterval: "10000", MetricExportTimeout: "10000"}, want: nil, }, } @@ -656,39 +657,39 @@ func TestBuildDeploymentApplyConfigMetricExportTuning(t *testing.T) { } // TestBuildDeploymentApplyConfigTracesSamplerPropagation asserts the sampler -// env pair reaches the ateom container only alongside an endpoint, and the arg +// env pair reaches the worker container only alongside an endpoint, and the arg // only alongside a sampler: an arg without a sampler name is dead config the // SDK ignores. func TestBuildDeploymentApplyConfigTracesSamplerPropagation(t *testing.T) { const endpoint = "http://collector.otel-system.svc:4317" tests := []struct { name string - otel ateomOTelSettings + otel workerOTelSettings want map[string]string // value by env name; absent key means must not be set }{ { name: "unset keeps binary default", - otel: ateomOTelSettings{Endpoint: endpoint}, + otel: workerOTelSettings{Endpoint: endpoint}, want: nil, }, { name: "sampler and arg with endpoint", - otel: ateomOTelSettings{Endpoint: endpoint, TracesSampler: "parentbased_traceidratio", TracesSamplerArg: "0.25"}, + otel: workerOTelSettings{Endpoint: endpoint, TracesSampler: "parentbased_traceidratio", TracesSamplerArg: "0.25"}, want: map[string]string{"OTEL_TRACES_SAMPLER": "parentbased_traceidratio", "OTEL_TRACES_SAMPLER_ARG": "0.25"}, }, { name: "sampler alone", - otel: ateomOTelSettings{Endpoint: endpoint, TracesSampler: "parentbased_always_on"}, + otel: workerOTelSettings{Endpoint: endpoint, TracesSampler: "parentbased_always_on"}, want: map[string]string{"OTEL_TRACES_SAMPLER": "parentbased_always_on"}, }, { name: "arg alone stays unset", - otel: ateomOTelSettings{Endpoint: endpoint, TracesSamplerArg: "0.25"}, + otel: workerOTelSettings{Endpoint: endpoint, TracesSamplerArg: "0.25"}, want: nil, }, { name: "ignored without endpoint", - otel: ateomOTelSettings{TracesSampler: "parentbased_traceidratio", TracesSamplerArg: "0.25"}, + otel: workerOTelSettings{TracesSampler: "parentbased_traceidratio", TracesSamplerArg: "0.25"}, want: nil, }, } @@ -764,7 +765,7 @@ func testWorkerPoolApplyConfig(tmpl *atev1alpha1.WorkerPoolPodTemplate) *atev1al ObjectMeta: metav1.ObjectMeta{Name: "pool", Namespace: "default", UID: "uid"}, Spec: atev1alpha1.WorkerPoolSpec{ Replicas: 2, - WorkerImage: "ateom:v1", + WorkerImage: "worker:v1", Template: tmpl, }, } @@ -779,14 +780,14 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf WithRunAsGroup(0)). WithVolumes( corev1ac.Volume(). - WithName(ateomCapacityVolume). + WithName(workerCapacityVolume). WithDownwardAPI(corev1ac.DownwardAPIVolumeSource(). WithItems( - resourceFieldRefFile(ateom.CPULimitFile, "limits.cpu", milliCores), - resourceFieldRefFile(ateom.MemoryLimitFile, "limits.memory", wholeBytes), + resourceFieldRefFile(workercapacity.CPULimitFile, "limits.cpu", milliCores), + resourceFieldRefFile(workercapacity.MemoryLimitFile, "limits.memory", wholeBytes), )), corev1ac.Volume(). - WithName("run-ateom"). + WithName("ate-base"). WithHostPath(corev1ac.HostPathVolumeSource(). WithPath(nodepath.BasePath). WithType(corev1.HostPathDirectoryOrCreate)), @@ -821,7 +822,7 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf ), ). WithContainers(corev1ac.Container(). - WithName("ateom"). + WithName("worker"). WithImage(wp.Spec.WorkerImage). WithArgs( "--pod-uid=$(POD_UID)", @@ -833,6 +834,7 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf "--atunnel-egress-listen-address=0.0.0.0:15001", "--atunnel-egress-trust-bundle="+atunnelEgressTrustMountPath+"/trust-bundle.pem", "--dns-egress-policy="+egresspolicy.DNSEgressPolicyEnforceInAtunnel, + "--preview=", ). WithPorts(corev1ac.ContainerPort(). WithName("https"). @@ -856,7 +858,7 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf WithPrivileged(false). WithCapabilities(corev1ac.Capabilities(). WithDrop("ALL"). - WithAdd(ateomGvisorCapabilities...)). + WithAdd(workerGvisorCapabilities...)). WithAppArmorProfile(corev1ac.AppArmorProfile(). WithType(corev1.AppArmorProfileTypeUnconfined)). WithSeccompProfile(corev1ac.SeccompProfile(). @@ -880,11 +882,11 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf ). WithVolumeMounts( corev1ac.VolumeMount(). - WithName(ateomCapacityVolume). - WithMountPath(ateom.CapacityMountPath). + WithName(workerCapacityVolume). + WithMountPath(workercapacity.MountPath). WithReadOnly(true), corev1ac.VolumeMount(). - WithName("run-ateom"). + WithName("ate-base"). WithMountPath(nodepath.BasePath). WithMountPropagation(corev1.MountPropagationHostToContainer), corev1ac.VolumeMount(). @@ -935,7 +937,7 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf } // TestBuildDeploymentAtunnelIdentitiesRelocatedNamespace pins the SPIFFE -// identities handed to ateom to the controller's namespace. atunnel runs in +// identities handed to worker to the controller's namespace. atunnel runs in // the actor's pod, so it cannot derive atelet's or the router's namespace // itself; if these carry the wrong one, the credential broker handshake and // actor ingress both fail closed. Every other case here passes the canonical @@ -943,7 +945,7 @@ func expectedDeploymentApplyConfig(mutatePodSpec func(*corev1ac.PodSpecApplyConf func TestBuildDeploymentAtunnelIdentitiesRelocatedNamespace(t *testing.T) { const relocated = "substrate-test" - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, relocated, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), workerOTelSettings{}, relocated, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] want := map[string]string{ @@ -978,7 +980,7 @@ func TestBuildDeploymentAtunnelIdentitiesPrefixedServiceAccounts(t *testing.T) { router = "kagent-atenet-router" ) - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, namespace, atelet, router, egresspolicy.DNSEgressPolicyEnforceInAtunnel). + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), workerOTelSettings{}, namespace, atelet, router, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] want := map[string]string{ @@ -999,21 +1001,21 @@ func TestBuildDeploymentAtunnelIdentitiesPrefixedServiceAccounts(t *testing.T) { } // TestBuildDeploymentOmitsBrokerIdentityForCanonicalInstall pins the flag's -// absence. A WorkerPool may still pin an ateom from before +// absence. A WorkerPool may still pin a worker from before // --atunnel-broker-identity existed, which exits on the unrecognized flag, so // passing it would crashloop that pool's workers. func TestBuildDeploymentOmitsBrokerIdentityForCanonicalInstall(t *testing.T) { - c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). Spec.Template.Spec.Containers[0] for _, arg := range c.Args { if strings.HasPrefix(arg, "--atunnel-broker-identity=") { - t.Fatalf("canonical install passed %q; an ateom predating the flag exits on it", arg) + t.Fatalf("canonical install passed %q; a worker predating the flag exits on it", arg) } } - // The value it would have carried is the one such an ateom already assumes, + // The value it would have carried is the one such a worker already assumes, // so omitting it changes nothing for either binary. var clientIdentity string for _, arg := range c.Args { @@ -1026,11 +1028,11 @@ func TestBuildDeploymentOmitsBrokerIdentityForCanonicalInstall(t *testing.T) { } } -// TestBuildDeploymentApplyConfigWorkerPoolEnv asserts the ateom reads its +// TestBuildDeploymentApplyConfigWorkerPoolEnv asserts the worker reads its // pool from the label the controller puts on the pod, so the usage records // name the pool the pod belongs to. func TestBuildDeploymentApplyConfigWorkerPoolEnv(t *testing.T) { - d := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), ateomOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) + d := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), workerOTelSettings{}, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel) c := d.Spec.Template.Spec.Containers[0] var path string for _, e := range c.Env { @@ -1047,17 +1049,17 @@ func TestBuildDeploymentApplyConfigWorkerPoolEnv(t *testing.T) { } // TestBuildDeploymentApplyConfigLogsExporter asserts OTEL_LOGS_EXPORTER -// reaches the ateom container only alongside an endpoint. +// reaches the worker container only alongside an endpoint. func TestBuildDeploymentApplyConfigLogsExporter(t *testing.T) { const endpoint = "http://collector.otel-system.svc:4317" for _, tt := range []struct { name string - otel ateomOTelSettings + otel workerOTelSettings want string // "" means must not be set }{ - {"set with endpoint", ateomOTelSettings{Endpoint: endpoint, LogsExporter: "otlp"}, "otlp"}, - {"unset keeps binary default", ateomOTelSettings{Endpoint: endpoint}, ""}, - {"ignored without endpoint", ateomOTelSettings{LogsExporter: "otlp"}, ""}, + {"set with endpoint", workerOTelSettings{Endpoint: endpoint, LogsExporter: "otlp"}, "otlp"}, + {"unset keeps binary default", workerOTelSettings{Endpoint: endpoint}, ""}, + {"ignored without endpoint", workerOTelSettings{LogsExporter: "otlp"}, ""}, } { t.Run(tt.name, func(t *testing.T) { c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), tt.otel, installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). @@ -1069,3 +1071,32 @@ func TestBuildDeploymentApplyConfigLogsExporter(t *testing.T) { }) } } + +// worker gets preview features when the controller has them. +func TestBuildDeploymentPreview(t *testing.T) { + for _, tc := range []struct { + name string + gates []string + want []string + }{ + {name: "disabled", want: []string{"--preview="}}, + {name: "named", gates: []string{"Fake"}, want: []string{"--preview=Fake"}}, + {name: "ALL", gates: []string{"ALL"}, want: []string{"--preview=Fake,Phony"}}, + } { + t.Run(tc.name, func(t *testing.T) { + preview.InitForTestFake(t, []preview.Gate{"Fake", "Phony"}, tc.gates...) + c := buildDeploymentApplyConfig(testWorkerPoolApplyConfig(nil), workerOTelSettings{}, + installdefaults.SystemNamespace, installdefaults.AteletServiceAccount, installdefaults.RouterServiceAccount, egresspolicy.DNSEgressPolicyEnforceInAtunnel). + Spec.Template.Spec.Containers[0] + var got []string + for _, arg := range c.Args { + if strings.HasPrefix(arg, "--preview") { + got = append(got, arg) + } + } + if diff := cmp.Diff(tc.want, got); diff != "" { + t.Errorf("--preview args mismatch (-want +got):\n%s", diff) + } + }) + } +} diff --git a/cmd/atecontroller/internal/controllers/workerpool_controller.go b/cmd/atecontroller/internal/controllers/workerpool_controller.go index f306704e92..873f5b5126 100644 --- a/cmd/atecontroller/internal/controllers/workerpool_controller.go +++ b/cmd/atecontroller/internal/controllers/workerpool_controller.go @@ -56,19 +56,19 @@ type WorkerPoolReconciler struct { Scheme *runtime.Scheme OTelEndpoint string // OTelMetricExportInterval is the OTEL_METRIC_EXPORT_INTERVAL propagated to - // ateom pods. Empty keeps the SDK's default. + // worker pods. Empty keeps the SDK's default. OTelMetricExportInterval string // OTelMetricExportTimeout is the OTEL_METRIC_EXPORT_TIMEOUT propagated to - // ateom pods. Empty keeps the SDK's default. + // worker pods. Empty keeps the SDK's default. OTelMetricExportTimeout string - // OTelTracesSampler is the OTEL_TRACES_SAMPLER propagated to ateom pods. - // Empty keeps the ateom binary's default. + // OTelTracesSampler is the OTEL_TRACES_SAMPLER propagated to worker pods. + // Empty keeps the worker binary's default. OTelTracesSampler string - // OTelTracesSamplerArg is the OTEL_TRACES_SAMPLER_ARG propagated to ateom + // OTelTracesSamplerArg is the OTEL_TRACES_SAMPLER_ARG propagated to worker // pods. Ignored unless OTelTracesSampler is set. OTelTracesSamplerArg string - // OTelLogsExporter is the OTEL_LOGS_EXPORTER propagated to ateom pods. - // Empty keeps the ateom binary's default. + // OTelLogsExporter is the OTEL_LOGS_EXPORTER propagated to worker pods. + // Empty keeps the worker binary's default. OTelLogsExporter string // SystemNamespace is the namespace substrate's control plane runs in, and // AteletServiceAccount / RouterServiceAccount are the ServiceAccounts those @@ -79,7 +79,7 @@ type WorkerPoolReconciler struct { SystemNamespace string AteletServiceAccount string RouterServiceAccount string - // DNSEgressPolicy is the --dns-egress-policy mode propagated to ateom pods. + // DNSEgressPolicy is the --dns-egress-policy mode propagated to worker pods. DNSEgressPolicy string // Recorder emits a Normal event each time a pool's SandboxConfig // changes, and a Warning event when a class stops resolving. @@ -317,7 +317,7 @@ func (r *WorkerPoolReconciler) recordSandboxConfigEvents(wp *atev1alpha1.WorkerP } func (r *WorkerPoolReconciler) applyDeployment(ctx context.Context, wp *atev1alpha1.WorkerPool) error { - depAC := buildDeploymentApplyConfig(wp, ateomOTelSettings{ + depAC := buildDeploymentApplyConfig(wp, workerOTelSettings{ Endpoint: r.OTelEndpoint, MetricExportInterval: r.OTelMetricExportInterval, MetricExportTimeout: r.OTelMetricExportTimeout, diff --git a/cmd/atecontroller/internal/controllers/workerpool_controller_test.go b/cmd/atecontroller/internal/controllers/workerpool_controller_test.go index 44b8d51795..f7e97579c1 100644 --- a/cmd/atecontroller/internal/controllers/workerpool_controller_test.go +++ b/cmd/atecontroller/internal/controllers/workerpool_controller_test.go @@ -130,7 +130,7 @@ func TestMain(m *testing.M) { func TestWorkerPoolCreatesDeployment(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-create", "default", 3, "ateom:v1") + wp := makeWorkerPool("test-create", "default", 3, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } @@ -148,7 +148,7 @@ func TestWorkerPoolCreatesDeployment(t *testing.T) { return false, nil } container := dep.Spec.Template.Spec.Containers[0] - if container.Image != "ateom:v1" || container.Name != "ateom" { + if container.Image != "worker:v1" || container.Name != "worker" { return false, nil } if dep.Spec.Template.Labels["ate.dev/worker-pool"] != wp.Name { @@ -158,8 +158,8 @@ func TestWorkerPoolCreatesDeployment(t *testing.T) { return false, nil } return len(dep.Spec.Template.Spec.Volumes) == 4 && - dep.Spec.Template.Spec.Volumes[0].Name == ateomCapacityVolume && - dep.Spec.Template.Spec.Volumes[1].Name == "run-ateom" && + dep.Spec.Template.Spec.Volumes[0].Name == workerCapacityVolume && + dep.Spec.Template.Spec.Volumes[1].Name == "ate-base" && dep.Spec.Template.Spec.Volumes[2].Name == atunnelIdentityVolume && dep.Spec.Template.Spec.Volumes[3].Name == atunnelEgressTrustVolume, nil }) @@ -170,7 +170,7 @@ func TestWorkerPoolCreatesDeployment(t *testing.T) { func TestWorkerPoolReplicasUpdate(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-replicas", "default", 2, "ateom:v1") + wp := makeWorkerPool("test-replicas", "default", 2, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } @@ -199,7 +199,7 @@ func TestWorkerPoolReplicasUpdate(t *testing.T) { func TestWorkerPoolImageUpdate(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-image", "default", 1, "ateom:v1") + wp := makeWorkerPool("test-image", "default", 1, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } @@ -211,7 +211,7 @@ func TestWorkerPoolImageUpdate(t *testing.T) { }) updateWorkerPoolSpec(t, ctx, wp, "update WorkerPool image", func(current *atev1alpha1.WorkerPool) { - current.Spec.WorkerImage = "ateom:v2" + current.Spec.WorkerImage = "worker:v2" }) eventually(t, func(ctx context.Context) (bool, error) { @@ -219,7 +219,7 @@ func TestWorkerPoolImageUpdate(t *testing.T) { if err != nil || len(dep.Spec.Template.Spec.Containers) == 0 { return false, nil } - return dep.Spec.Template.Spec.Containers[0].Image == "ateom:v2", nil + return dep.Spec.Template.Spec.Containers[0].Image == "worker:v2", nil }) } @@ -228,7 +228,7 @@ func TestWorkerPoolImageUpdate(t *testing.T) { func TestSSAPreservesUnownedFields(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-ssa-unowned", "default", 2, "ateom:v1") + wp := makeWorkerPool("test-ssa-unowned", "default", 2, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } @@ -264,7 +264,7 @@ func TestSSAPreservesUnownedFields(t *testing.T) { func TestSSARevertsOwnedFields(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-ssa-owned", "default", 2, "ateom:v1") + wp := makeWorkerPool("test-ssa-owned", "default", 2, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } @@ -295,7 +295,7 @@ func TestSSARevertsOwnedFields(t *testing.T) { func TestDeletedDeploymentRecreated(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-recreate", "default", 2, "ateom:v1") + wp := makeWorkerPool("test-recreate", "default", 2, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } @@ -326,7 +326,7 @@ func TestDeletedDeploymentRecreated(t *testing.T) { func TestStatusReplicasPropagation(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-status", "default", 3, "ateom:v1") + wp := makeWorkerPool("test-status", "default", 3, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } @@ -395,7 +395,7 @@ func sampleWorkerPoolPodTemplate() *atev1alpha1.WorkerPoolPodTemplate { func TestWorkerPoolPodTemplatePropagation(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-template-propagate", "default", 1, "ateom:v1") + wp := makeWorkerPool("test-template-propagate", "default", 1, "worker:v1") wp.Spec.Template = sampleWorkerPoolPodTemplate() if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) @@ -445,7 +445,7 @@ func TestWorkerPoolPodTemplatePropagation(t *testing.T) { func TestWorkerPoolPodTemplateUpdate(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-template-update", "default", 1, "ateom:v1") + wp := makeWorkerPool("test-template-update", "default", 1, "worker:v1") wp.Spec.Template = sampleWorkerPoolPodTemplate() if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) @@ -477,7 +477,7 @@ func TestWorkerPoolPodTemplateUpdate(t *testing.T) { func TestWorkerPoolPodTemplateClear(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-template-clear", "default", 1, "ateom:v1") + wp := makeWorkerPool("test-template-clear", "default", 1, "worker:v1") wp.Spec.Template = sampleWorkerPoolPodTemplate() if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) @@ -507,7 +507,7 @@ func TestWorkerPoolPodTemplateClear(t *testing.T) { func TestWorkerPoolPodTemplateClearAll(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-template-clear-all", "default", 1, "ateom:v1") + wp := makeWorkerPool("test-template-clear-all", "default", 1, "worker:v1") wp.Spec.Template = sampleWorkerPoolPodTemplate() if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) @@ -560,7 +560,7 @@ func TestWorkerPoolPodTemplateClearAll(t *testing.T) { func TestSSARevertsOwnedPodTemplateFields(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-ssa-template", "default", 1, "ateom:v1") + wp := makeWorkerPool("test-ssa-template", "default", 1, "worker:v1") wp.Spec.Template = sampleWorkerPoolPodTemplate() if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) @@ -590,7 +590,7 @@ func TestSSARevertsOwnedPodTemplateFields(t *testing.T) { func TestReplicasValidationRejectsNegative(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-neg-replicas", "default", -1, "ateom:v1") + wp := makeWorkerPool("test-neg-replicas", "default", -1, "worker:v1") err := k8sClient.Create(ctx, wp) if err == nil { deleteOnCleanup(t, wp) @@ -699,7 +699,7 @@ func eventually(t *testing.T, condition func(ctx context.Context) (bool, error)) func TestSyncStatus_ReadyReplicas(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool("test-sync-ready", "default", 3, "ateom:v1") + wp := makeWorkerPool("test-sync-ready", "default", 3, "worker:v1") if err := k8sClient.Create(ctx, wp); err != nil { t.Fatalf("create WorkerPool: %v", err) } @@ -732,7 +732,7 @@ func TestWorkerPoolMetrics(t *testing.T) { if err := atev1alpha1.AddToScheme(scheme); err != nil { t.Fatalf("add scheme: %v", err) } - wp := makeWorkerPool("test-metrics", "default", 4, "ateom:v1") + wp := makeWorkerPool("test-metrics", "default", 4, "worker:v1") wp.Status = atev1alpha1.WorkerPoolStatus{Replicas: 4, ReadyReplicas: 2} reader := sdkmetric.NewManualReader() r := &WorkerPoolReconciler{ diff --git a/cmd/atecontroller/internal/controllers/workerpool_sandboxconfig_test.go b/cmd/atecontroller/internal/controllers/workerpool_sandboxconfig_test.go index 46e54e02f9..a1ac3b49f4 100644 --- a/cmd/atecontroller/internal/controllers/workerpool_sandboxconfig_test.go +++ b/cmd/atecontroller/internal/controllers/workerpool_sandboxconfig_test.go @@ -355,7 +355,7 @@ func TestReconcileProtectsBeforeRecording(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") if tc.ref != "" { wp.Spec.SandboxClasses[0].ConfigRef = &atev1alpha1.SandboxConfigReference{Name: tc.ref} } @@ -514,7 +514,7 @@ func TestReconcileConfigDeletedBeforeClaim(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") wp.Status.SandboxClasses = tc.status if tc.pinned { wp.Spec.SandboxClasses[0].ConfigRef = &atev1alpha1.SandboxConfigReference{Name: tc.newDefault.Name} @@ -613,7 +613,7 @@ func TestSandboxConfigReleasedWhenUnused(t *testing.T) { for _, name := range tc.configs { createSandboxConfig(t, makeSandboxConfig(name, atev1alpha1.SandboxClassGvisor, false)) } - wp := makeWorkerPool(tc.pool, "default", 1, "ateom:v1") + wp := makeWorkerPool(tc.pool, "default", 1, "worker:v1") wp.Spec.SandboxClasses[0].ConfigRef = &atev1alpha1.SandboxConfigReference{Name: tc.ref} createWorkerPool(t, wp) waitResolvedAndDeployed(t, wp, tc.ref) @@ -701,7 +701,7 @@ func TestWorkerPoolConfigRef(t *testing.T) { } } } - wp := makeWorkerPool("test-configref-"+tc.ref, "default", 1, "ateom:v1") + wp := makeWorkerPool("test-configref-"+tc.ref, "default", 1, "worker:v1") wp.Spec.SandboxClasses[0].ConfigRef = &atev1alpha1.SandboxConfigReference{Name: tc.ref} createWorkerPool(t, wp) @@ -790,7 +790,7 @@ func TestWorkerPoolUnresolvedKeepsConfigInUse(t *testing.T) { t.Run(tc.name, func(t *testing.T) { t.Parallel() ctx := t.Context() - wp := makeWorkerPool(tc.pool, "default", 1, "ateom:v1") + wp := makeWorkerPool(tc.pool, "default", 1, "worker:v1") inUse := testGvisorDefault if tc.config != nil { controllerutil.AddFinalizer(tc.config, testHoldFinalizer) @@ -875,7 +875,7 @@ func TestSandboxConfigInUseProtection(t *testing.T) { sentinel := makeSandboxConfig(tc.sentinel, atev1alpha1.SandboxClassGvisor, false) createSandboxConfig(t, sentinel) newPool := func(name, config string) *atev1alpha1.WorkerPool { - wp := makeWorkerPool(name, "default", 1, "ateom:v1") + wp := makeWorkerPool(name, "default", 1, "worker:v1") wp.Spec.SandboxClasses[0].ConfigRef = &atev1alpha1.SandboxConfigReference{Name: config} createWorkerPool(t, wp) waitResolved(t, wp, config) @@ -933,7 +933,7 @@ func TestWorkerPoolDefaultNewestWins(t *testing.T) { t.Fatalf("create SandboxConfig %s: %v", sc.Name, err) } } - wp := makeWorkerPool("test-default-newest", "default", 1, "ateom:v1") + wp := makeWorkerPool("test-default-newest", "default", 1, "worker:v1") wp.Spec.SandboxClasses[0].Name = atev1alpha1.SandboxClassMicroVM createWorkerPool(t, wp) @@ -1291,7 +1291,7 @@ func TestRecordSandboxConfigEvents(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") rec := events.NewFakeRecorder(2 * len(tc.classes)) r := &WorkerPoolReconciler{Recorder: rec} r.recordSandboxConfigEvents(wp, tc.prev, tc.classes) @@ -1453,7 +1453,7 @@ func TestResolveSandboxClasses(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") wp.Generation = generation wp.Spec.SandboxClasses[0].ConfigRef = ref(tc.specRef) wp.Status.SandboxClasses = tc.prev @@ -1555,7 +1555,7 @@ func TestClaimSandboxConfigs(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") wp.Status.SandboxClasses = tc.prev if got := claimSandboxConfigs(wp, tc.classes); !equality.Semantic.DeepEqual(got, tc.want) { t.Errorf("claimSandboxConfigs = %+v, want %+v", got, tc.want) @@ -1646,7 +1646,7 @@ func TestConfirmSandboxConfigSwitch(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") wp.Generation = generation wp.Status.SandboxClasses = tc.prev got := confirmSandboxConfigSwitch(wp, *class.DeepCopy(), tc.live, configs) @@ -1760,7 +1760,7 @@ func TestReconcileConfirmsSwitch(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") wp.Spec.SandboxClasses[0].ConfigRef = &atev1alpha1.SandboxConfigReference{Name: "x"} objs := []client.Object{wp, x(), protected(makeSandboxConfig("y", atev1alpha1.SandboxClassGvisor, false))} if tc.inUse != "" { @@ -1945,7 +1945,7 @@ func TestSwitchInterleavings(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") wp.Spec.SandboxClasses[0].ConfigRef = &atev1alpha1.SandboxConfigReference{Name: "x"} wp.Status.SandboxClasses = []atev1alpha1.WorkerPoolSandboxClassStatus{{ Name: atev1alpha1.SandboxClassGvisor, @@ -2227,7 +2227,7 @@ func TestReconcileAppliesExistingDeploymentFirst(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 3, "ateom:v1") + wp := makeWorkerPool("pool", "default", 3, "worker:v1") objs := []client.Object{wp, makeSandboxConfig("def", atev1alpha1.SandboxClassGvisor, true)} if tc.hasDeployment { objs = append(objs, &appsv1.Deployment{ @@ -2327,7 +2327,7 @@ func TestSyncStatus(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { t.Parallel() - wp := makeWorkerPool("pool", "default", 1, "ateom:v1") + wp := makeWorkerPool("pool", "default", 1, "worker:v1") wp.Status = *before.DeepCopy() c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(wp).WithStatusSubresource(wp).Build() stored := &atev1alpha1.WorkerPool{} @@ -2537,7 +2537,7 @@ func removeFinalizer(t *testing.T, name, finalizer string) { // makeWorkerPoolClasses returns a pool in ns with the given spec and status // sandbox classes. func makeWorkerPoolClasses(ns, name string, spec []atev1alpha1.WorkerPoolSandboxClass, status []atev1alpha1.WorkerPoolSandboxClassStatus) *atev1alpha1.WorkerPool { - wp := makeWorkerPool(name, ns, 1, "ateom:v1") + wp := makeWorkerPool(name, ns, 1, "worker:v1") wp.Spec.SandboxClasses = spec wp.Status.SandboxClasses = status return wp diff --git a/cmd/atecontroller/internal/workersync/syncer.go b/cmd/atecontroller/internal/workersync/syncer.go index 48587e094b..09eb6a3d3c 100644 --- a/cmd/atecontroller/internal/workersync/syncer.go +++ b/cmd/atecontroller/internal/workersync/syncer.go @@ -266,7 +266,7 @@ func (s *WorkerPoolSyncer) reconcile(ctx context.Context, key workerKey) error { if pod.DeletionTimestamp != nil { // The pod has entered Terminating: mark the worker DRAINING so the // scheduler stops routing new actors to it. We deliberately do NOT touch - // the bound actor here — inside the pod ateom has received SIGTERM and is + // the bound actor here — inside the pod worker has received SIGTERM and is // gracefully shutting the actor down. Actor cleanup happens on the Pod // Deleted event. return s.markWorkerDraining(ctx, key) @@ -280,14 +280,14 @@ func (s *WorkerPoolSyncer) reconcile(ctx context.Context, key workerKey) error { return s.createOrUpdateWorker(ctx, key, pod) } -// ateomContainer is the name of the worker pod's ateom container. -const ateomContainer = "ateom" +// workerContainer is the name of the worker pod's worker container. +const workerContainer = "worker" -// podEpoch counts the runs of the pod's ateom container: 1 for its first run +// podEpoch counts the runs of the pod's worker container: 1 for its first run // and one more for each restart. 0 until kubelet reports the container. func podEpoch(pod *corev1.Pod) int64 { for _, cs := range pod.Status.ContainerStatuses { - if cs.Name == ateomContainer { + if cs.Name == workerContainer { return int64(cs.RestartCount) + 1 } } @@ -297,7 +297,7 @@ func podEpoch(pod *corev1.Pod) int64 { // createOrUpdateWorker registers the pod's Worker, or brings the registered // one in line with the pod. A Worker is registered UNAVAILABLE and marked // available only while its pod is Ready, and only once its epoch matches the -// ateom running, so nothing is placed on it under the epoch of an ateom that +// worker running, so nothing is placed on it under the epoch of a worker that // has since restarted. func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerKey, pod *corev1.Pod) error { poolName := pod.Labels[workerPodLabel] @@ -337,7 +337,7 @@ func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerK Labels: pool.GetLabels(), Epoch: podEpoch(pod), // Capacity is the Worker's to report, not the syncer's to infer - // from the pod: it is what the ateom can actually supply. Until + // from the pod: it is what the worker can actually supply. Until // that report lands, CreateWorker's reified ceiling holds the // Worker to a single Actor. } @@ -374,7 +374,7 @@ func (s *WorkerPoolSyncer) createOrUpdateWorker(ctx context.Context, key workerK // new pod, which arrives under a new key. var changed bool if epoch := podEpoch(pod); epoch > w.GetEpoch() { - slog.InfoContext(ctx, "Syncer: updating worker (ateom restarted)", + slog.InfoContext(ctx, "Syncer: updating worker (worker restarted)", append(key.logAttrs(), slog.Int64("epoch", epoch))...) w.Epoch = epoch changed = true diff --git a/cmd/atecontroller/internal/workersync/syncer_restart_test.go b/cmd/atecontroller/internal/workersync/syncer_restart_test.go index 19045786cf..dd049d018f 100644 --- a/cmd/atecontroller/internal/workersync/syncer_restart_test.go +++ b/cmd/atecontroller/internal/workersync/syncer_restart_test.go @@ -22,12 +22,12 @@ import ( corev1 "k8s.io/api/core/v1" ) -// withAteomRestarts reports the pod's ateom container as having restarted n +// withWorkerRestarts reports the pod's worker container as having restarted n // times. -func withAteomRestarts(pod *corev1.Pod, n int32) *corev1.Pod { +func withWorkerRestarts(pod *corev1.Pod, n int32) *corev1.Pod { pod.Status.ContainerStatuses = []corev1.ContainerStatus{ {Name: "sidecar", RestartCount: 7}, - {Name: ateomContainer, RestartCount: n}, + {Name: workerContainer, RestartCount: n}, } return pod } @@ -35,12 +35,12 @@ func withAteomRestarts(pod *corev1.Pod, n int32) *corev1.Pod { func TestPodEpoch(t *testing.T) { pod := workerPod("ns", "worker", "pool", testPodUID, "10.0.0.1") if got := podEpoch(pod); got != 0 { - t.Errorf("podEpoch with no ateom status = %d, want 0", got) + t.Errorf("podEpoch with no worker status = %d, want 0", got) } - if got := podEpoch(withAteomRestarts(pod, 0)); got != 1 { + if got := podEpoch(withWorkerRestarts(pod, 0)); got != 1 { t.Errorf("podEpoch on the first run = %d, want 1", got) } - if got := podEpoch(withAteomRestarts(pod, 4)); got != 5 { + if got := podEpoch(withWorkerRestarts(pod, 4)); got != 5 { t.Errorf("podEpoch after 4 restarts = %d, want 5", got) } } @@ -51,7 +51,7 @@ func TestSyncer_RegistersEpoch(t *testing.T) { api := newFakeControl() s, pods, _ := setupReconcileTest(t, api, workerPool(ns, poolName, "gvisor", nil)) - key := seedPod(t, pods, withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 2)) + key := seedPod(t, pods, withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 2)) mustReconcile(t, ctx, s, key) if got := api.get(testPodUID).GetEpoch(); got != 3 { @@ -59,7 +59,7 @@ func TestSyncer_RegistersEpoch(t *testing.T) { } } -func TestSyncer_RaisesEpochOnAteomRestart(t *testing.T) { +func TestSyncer_RaisesEpochOnWorkerRestart(t *testing.T) { ctx := context.Background() ns, podName, poolName := "ns-syncer-restart", "worker-restart-1", "pool1" @@ -68,20 +68,20 @@ func TestSyncer_RaisesEpochOnAteomRestart(t *testing.T) { ready bool }{ {name: "Ready", ready: true}, - // A restarting ateom is not Ready, but its Actors are already lost. + // A restarting worker is not Ready, but its Actors are already lost. {name: "not Ready", ready: false}, } { t.Run(tc.name, func(t *testing.T) { api := newFakeControl() s, pods, _ := setupReconcileTest(t, api, workerPool(ns, poolName, "gvisor", nil)) - pod := withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0) + pod := withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0) key := seedPod(t, pods, pod) mustReconcile(t, ctx, s, key) if got := api.get(testPodUID).GetEpoch(); got != 1 { t.Fatalf("registered epoch = %d, want 1", got) } - restarted := withAteomRestarts(pod.DeepCopy(), 1) + restarted := withWorkerRestarts(pod.DeepCopy(), 1) if !tc.ready { restarted.Status.Conditions = []corev1.PodCondition{{Type: corev1.PodReady, Status: corev1.ConditionFalse}} } @@ -104,9 +104,9 @@ func TestSyncer_RaisesEpochOnAteomRestart(t *testing.T) { } } -// TestSyncer_NoPlacementUnderAnEarlierEpoch walks an ateom restart through +// TestSyncer_NoPlacementUnderAnEarlierEpoch walks a worker restart through // the pod states kubelet reports, and pins that no write leaves the Worker -// ACTIVE while its epoch trails the ateom that is running. An Actor placed +// ACTIVE while its epoch trails the worker that is running. An Actor placed // then would be stamped with the earlier epoch and crashed with the Actors the // restart lost. func TestSyncer_NoPlacementUnderAnEarlierEpoch(t *testing.T) { @@ -125,11 +125,11 @@ func TestSyncer_NoPlacementUnderAnEarlierEpoch(t *testing.T) { { name: "every transition observed", steps: []*corev1.Pod{ - // ateom exited; kubelet has not started it again. - notReady(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)), - // ateom running again, readiness probe not yet passed. - notReady(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1)), - withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), + // worker exited; kubelet has not started it again. + notReady(withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)), + // worker running again, readiness probe not yet passed. + notReady(withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1)), + withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), }, want: []ateapipb.WorkerState{ ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, @@ -142,7 +142,7 @@ func TestSyncer_NoPlacementUnderAnEarlierEpoch(t *testing.T) { // pod Ready again after the restart. name: "not Ready never observed", steps: []*corev1.Pod{ - withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), + withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), }, want: []ateapipb.WorkerState{ateapipb.WorkerState_WORKER_STATE_ACTIVE}, }, @@ -150,8 +150,8 @@ func TestSyncer_NoPlacementUnderAnEarlierEpoch(t *testing.T) { // The epoch has to be raised before the Worker is marked available. name: "Ready again under a new epoch", steps: []*corev1.Pod{ - notReady(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)), - withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), + notReady(withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)), + withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), }, want: []ateapipb.WorkerState{ ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, @@ -159,11 +159,11 @@ func TestSyncer_NoPlacementUnderAnEarlierEpoch(t *testing.T) { }, }, { - // The restart is first seen with ateom running again but not Ready. + // The restart is first seen with worker running again but not Ready. name: "not Ready under a new epoch", steps: []*corev1.Pod{ - notReady(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1)), - withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), + notReady(withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1)), + withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 1), }, want: []ateapipb.WorkerState{ ateapipb.WorkerState_WORKER_STATE_UNAVAILABLE, @@ -174,7 +174,7 @@ func TestSyncer_NoPlacementUnderAnEarlierEpoch(t *testing.T) { t.Run(tc.name, func(t *testing.T) { api := newFakeControl() s, pods, _ := setupReconcileTest(t, api, workerPool(ns, poolName, "gvisor", nil)) - key := seedPod(t, pods, withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)) + key := seedPod(t, pods, withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)) mustReconcile(t, ctx, s, key) if state := api.get(testPodUID).GetStatus().GetState(); state != ateapipb.WorkerState_WORKER_STATE_ACTIVE { t.Fatalf("state after registering = %v, want %v", state, ateapipb.WorkerState_WORKER_STATE_ACTIVE) @@ -202,15 +202,15 @@ func TestSyncer_NoPlacementUnderAnEarlierEpoch(t *testing.T) { } } -// A pod first seen not Ready after ateom restarts registers unavailable at -// the epoch of the ateom it is running. +// A pod first seen not Ready after worker restarts registers unavailable at +// the epoch of the worker it is running. func TestSyncer_NotReadyPodRegistersAtItsEpoch(t *testing.T) { ctx := context.Background() ns, poolName := "ns-syncer-notready-restart", "pool1" api := newFakeControl() s, pods, _ := setupReconcileTest(t, api, workerPool(ns, poolName, "gvisor", nil)) - pod := withAteomRestarts(workerPod(ns, "worker-1", poolName, testPodUID, "10.0.0.1"), 3) + pod := withWorkerRestarts(workerPod(ns, "worker-1", poolName, testPodUID, "10.0.0.1"), 3) pod.Status.Conditions = nil key := seedPod(t, pods, pod) @@ -234,7 +234,7 @@ func TestSyncer_EpochNeverLowered(t *testing.T) { registered.SandboxClass = "gvisor" registered.Epoch = 5 api.put(registered) - key := seedPod(t, pods, withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)) + key := seedPod(t, pods, withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.1"), 0)) mustReconcile(t, ctx, s, key) got := api.get(testPodUID) diff --git a/cmd/atecontroller/internal/workersync/syncer_test.go b/cmd/atecontroller/internal/workersync/syncer_test.go index 0bec94613f..8d8b6d4494 100644 --- a/cmd/atecontroller/internal/workersync/syncer_test.go +++ b/cmd/atecontroller/internal/workersync/syncer_test.go @@ -70,7 +70,7 @@ func workerPod(ns, name, poolName, uid, ip string) *corev1.Pod { PodIP: ip, PodIPs: []corev1.PodIP{{IP: ip}}, // Availability requires Ready in addition to an IP: readiness is - // what says ateom is actually serving, not just that the sandbox + // what says worker is actually serving, not just that the sandbox // got an address. Conditions: []corev1.PodCondition{{ Type: corev1.PodReady, @@ -262,9 +262,9 @@ func TestSyncer_OmittedFields(t *testing.T) { } // Capacity belongs to the Worker, which sets it over WorkerService. The -// syncer must not infer one from the pod, even when the ateom container spells -// its limits out: the pod's limits bound the sandbox, but what the ateom can -// actually supply to Actors is the ateom's to say. +// syncer must not infer one from the pod, even when the worker container spells +// its limits out: the pod's limits bound the sandbox, but what the worker can +// actually supply to Actors is the worker's to say. func TestSyncer_DoesNotInferCapacityFromThePod(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() @@ -276,8 +276,8 @@ func TestSyncer_DoesNotInferCapacityFromThePod(t *testing.T) { pod := workerPod(ns, podName, poolName, testPodUID, "127.0.0.1") pod.Spec.Containers = append(pod.Spec.Containers, corev1.Container{ - Name: "ateom", - Image: "ateom", + Name: "worker", + Image: "worker", Resources: corev1.ResourceRequirements{Limits: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("4"), corev1.ResourceMemory: resource.MustParse("8Gi"), @@ -289,7 +289,7 @@ func TestSyncer_DoesNotInferCapacityFromThePod(t *testing.T) { // The registry here is a fake, so what it holds is exactly what the syncer // sent. The syncer writes no capacity: it comes from the Worker's own - // report, which is the ateom's to make. + // report, which is the worker's to make. got := waitForWorker(t, ctx, api, testPodUID, func(w *ateapipb.Worker) bool { return w != nil }) if got.GetStatus().GetCapacity() != nil { t.Errorf("worker capacity = %v, want none", got.GetStatus().GetCapacity()) @@ -918,7 +918,7 @@ func TestSyncer_DeletePodWithoutIP(t *testing.T) { // TestSyncer_PodWithIPButNotReadyIsUnavailable pins that an IP alone registers // a worker that takes no Actors, because the sandbox having an address says -// nothing about ateom serving yet. Both not-Ready shapes are covered — +// nothing about worker serving yet. Both not-Ready shapes are covered — // condition absent (kubelet hasn't probed yet) and condition explicitly False // (probe failing). func TestSyncer_PodWithIPButNotReadyIsUnavailable(t *testing.T) { @@ -988,10 +988,10 @@ func TestSyncer_RegisteredPodWithoutIPIsUnavailable(t *testing.T) { api := newFakeControl() s, pods, _ := setupReconcileTest(t, api, workerPool(ns, poolName, "gvisor", nil)) - key := seedPod(t, pods, withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.9"), 0)) + key := seedPod(t, pods, withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, "10.0.0.9"), 0)) mustReconcile(t, ctx, s, key) - if err := pods.Update(withAteomRestarts(workerPod(ns, podName, poolName, testPodUID, ""), 1)); err != nil { + if err := pods.Update(withWorkerRestarts(workerPod(ns, podName, poolName, testPodUID, ""), 1)); err != nil { t.Fatalf("updating pod: %v", err) } mustReconcile(t, ctx, s, key) diff --git a/cmd/atecontroller/main.go b/cmd/atecontroller/main.go index 174ad5c2c5..7053be27c4 100644 --- a/cmd/atecontroller/main.go +++ b/cmd/atecontroller/main.go @@ -24,6 +24,7 @@ import ( "github.com/agent-substrate/substrate/internal/ateapiauth" "github.com/agent-substrate/substrate/internal/egresspolicy" "github.com/agent-substrate/substrate/internal/installdefaults" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/version" clientv1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" @@ -60,24 +61,25 @@ var ( showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") + previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.") otelEndpoint = pflag.String("otel-exporter-otlp-endpoint", os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"), - "OTLP endpoint set on ateom worker pods so they push telemetry. Defaults to the controller's own OTEL_EXPORTER_OTLP_ENDPOINT.") + "OTLP endpoint set on worker pods so they push telemetry. Defaults to the controller's own OTEL_EXPORTER_OTLP_ENDPOINT.") otelMetricExportInterval = pflag.String("otel-metric-export-interval", os.Getenv("OTEL_METRIC_EXPORT_INTERVAL"), - "Metric export interval in milliseconds set on ateom worker pods. Empty keeps the OTel SDK's 60s default. Defaults to the controller's own OTEL_METRIC_EXPORT_INTERVAL.") + "Metric export interval in milliseconds set on worker pods. Empty keeps the OTel SDK's 60s default. Defaults to the controller's own OTEL_METRIC_EXPORT_INTERVAL.") otelMetricExportTimeout = pflag.String("otel-metric-export-timeout", os.Getenv("OTEL_METRIC_EXPORT_TIMEOUT"), - "Per-export timeout in milliseconds set on ateom worker pods. Empty keeps the OTel SDK's 30s default. Defaults to the controller's own OTEL_METRIC_EXPORT_TIMEOUT.") + "Per-export timeout in milliseconds set on worker pods. Empty keeps the OTel SDK's 30s default. Defaults to the controller's own OTEL_METRIC_EXPORT_TIMEOUT.") otelTracesSampler = pflag.String("otel-traces-sampler", os.Getenv("OTEL_TRACES_SAMPLER"), - "Trace sampler set on ateom worker pods. Empty keeps the ateom binary's default. Defaults to the controller's own OTEL_TRACES_SAMPLER.") + "Trace sampler set on worker pods. Empty keeps the ateworker binary's default. Defaults to the controller's own OTEL_TRACES_SAMPLER.") otelTracesSamplerArg = pflag.String("otel-traces-sampler-arg", os.Getenv("OTEL_TRACES_SAMPLER_ARG"), - "Trace sampler argument set on ateom worker pods, ignored unless --otel-traces-sampler is set. Defaults to the controller's own OTEL_TRACES_SAMPLER_ARG.") + "Trace sampler argument set on worker pods, ignored unless --otel-traces-sampler is set. Defaults to the controller's own OTEL_TRACES_SAMPLER_ARG.") otelLogsExporter = pflag.String("otel-logs-exporter", os.Getenv("OTEL_LOGS_EXPORTER"), - "Logs exporter set on ateom worker pods. Empty keeps the ateom binary's default, none. Defaults to the controller's own OTEL_LOGS_EXPORTER.") + "Logs exporter set on worker pods. Empty keeps the ateworker binary's default, none. Defaults to the controller's own OTEL_LOGS_EXPORTER.") ateletServiceAccount = pflag.String("atelet-service-account", installdefaults.AteletServiceAccount, "ServiceAccount atelet runs as. It is the service-account segment of the SPIFFE ID each worker's atunnel expects on the credential broker, so it has to match what the deployment actually creates.") routerServiceAccount = pflag.String("router-service-account", installdefaults.RouterServiceAccount, "ServiceAccount atenet-router runs as. It is the service-account segment of the SPIFFE ID each worker's atunnel accepts on actor ingress, so it has to match what the deployment actually creates.") @@ -88,7 +90,7 @@ var ( workerSyncConcurrency = pflag.Int("worker-sync-concurrency", 2, "Number of concurrent worker goroutines reconciling worker pods into the Worker registry. Defaults to 2.") - dnsEgressPolicy = pflag.String("dns-egress-policy", "", "DNS egress policy enforcement mode propagated to ateom worker pods (must be "+egresspolicy.DNSEgressPolicyEnforceInAtunnel+").") + dnsEgressPolicy = pflag.String("dns-egress-policy", "", "DNS egress policy enforcement mode propagated to worker pods (must be "+egresspolicy.DNSEgressPolicyEnforceInAtunnel+").") ) func init() { @@ -150,6 +152,9 @@ func main() { if err := serverboot.SetLogLevel(*logLevelFlag); err != nil { serverboot.Fatal(ctx, "Invalid --log-level", err) } + if err := preview.Init(*previewFlags...); err != nil { + serverboot.Fatal(ctx, "Invalid --preview", err) + } slog.InfoContext(ctx, "atecontroller starting", slog.String("version", version.Version)) ctrl.SetLogger(newControllerRuntimeLogger(slog.Default().Handler())) @@ -208,6 +213,9 @@ func main() { os.Exit(1) } dialOpts = append(dialOpts, grpc.WithStatsHandler(otelgrpc.NewClientHandler())) + // atecontroller acts for the system, not a user, so it always opts in to + // the preview features the server enables. + dialOpts = append(dialOpts, preview.DialOptionsWithOptIn()...) ateapiConn, err := grpc.NewClient(*ateAPIConnSpec, dialOpts...) if err != nil { diff --git a/cmd/atelet/imagegc.go b/cmd/atelet/imagegc.go index fe70b46213..af81d261d4 100644 --- a/cmd/atelet/imagegc.go +++ b/cmd/atelet/imagegc.go @@ -77,7 +77,7 @@ func validateImageCacheGCFlags() error { return fmt.Errorf("--image-cache-min-age %v must be >= 0", *imageCacheMinAge) } if imageCacheDirOutsideBasePath(*imageCacheDir) { - slog.Warn("Image cache dir is outside the ateom base path; its volume watermarks are measured separately from actor state", + slog.Warn("Image cache dir is outside the worker base path; its volume watermarks are measured separately from actor state", slog.String("image_cache_dir", *imageCacheDir), slog.String("actors_dir", nodepath.ActorsDir)) } @@ -85,7 +85,7 @@ func validateImageCacheGCFlags() error { } // imageCacheDirOutsideBasePath reports whether the cache dir is outside -// the ateom base path — the watermarks then measure a different volume +// the worker base path — the watermarks then measure a different volume // than actor state. Warn-worthy, not an error: a separate cache volume is // legitimate (recommended for IOPS). func imageCacheDirOutsideBasePath(dir string) bool { diff --git a/cmd/atelet/imagevolume_test.go b/cmd/atelet/imagevolume_test.go index e22c60cb86..d39c42332c 100644 --- a/cmd/atelet/imagevolume_test.go +++ b/cmd/atelet/imagevolume_test.go @@ -103,7 +103,7 @@ func newImageVolumeStore(t *testing.T) *imagecache.Store { return s } -// A mounted image volume records its layers for ateom to compose, and its +// A mounted image volume records its layers for worker to compose, and its // digest so the cache GC can protect them. func TestResolveImageVolumes_RecordsLayersAndDigest(t *testing.T) { host := imageVolumeTestRegistry(t) diff --git a/cmd/atelet/internal/apivalidation/validation_test.go b/cmd/atelet/internal/apivalidation/validation_test.go index 6e020b666a..d84212e8cb 100644 --- a/cmd/atelet/internal/apivalidation/validation_test.go +++ b/cmd/atelet/internal/apivalidation/validation_test.go @@ -276,7 +276,7 @@ func TestValidateRunRequest(t *testing.T) { want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { name: "invalid worker_pod_uid: not a pod UID", - obj: valid(func(r *ateletpb.RunRequest) { r.WorkerPodUid = "ateom-1" }), + obj: valid(func(r *ateletpb.RunRequest) { r.WorkerPodUid = "worker-1" }), want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { name: "missing atespace", @@ -508,6 +508,7 @@ func TestValidateCheckpointRequest(t *testing.T) { Spec: &ateletpb.WorkloadSpec{}, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: "snap-1", + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, Object: &ateletpb.ObjectStorage{SnapshotUri: snapshotURI}, }, @@ -556,7 +557,7 @@ func TestValidateCheckpointRequest(t *testing.T) { want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { name: "invalid worker_pod_uid: not a pod UID", - obj: valid(func(r *ateletpb.CheckpointRequest) { r.WorkerPodUid = "ateom-1" }), + obj: valid(func(r *ateletpb.CheckpointRequest) { r.WorkerPodUid = "worker-1" }), want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { name: "invalid atespace: path escape", @@ -620,6 +621,14 @@ func TestValidateCheckpointRequest(t *testing.T) { name: "invalid snapshot_uuid: path escape", obj: valid(deferred("../escape")), want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "snapshot_uuid"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing generation", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Snapshot.Generation = 0 }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "generation"), "")}, + }, { + name: "negative generation", + obj: valid(func(r *ateletpb.CheckpointRequest) { r.Snapshot.Generation = -1 }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "generation"), nil, "").WithOrigin("minimum")}, }, { name: "missing snapshot_uri", obj: valid(func(r *ateletpb.CheckpointRequest) { r.Snapshot.Object.SnapshotUri = "" }), @@ -673,6 +682,7 @@ func TestValidateRestoreRequest(t *testing.T) { Spec: &ateletpb.WorkloadSpec{}, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: "snap-1", + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, Object: &ateletpb.ObjectStorage{SnapshotUri: snapshotURI}, }, @@ -722,7 +732,7 @@ func TestValidateRestoreRequest(t *testing.T) { want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { name: "invalid worker_pod_uid: not a pod UID", - obj: valid(func(r *ateletpb.RestoreRequest) { r.WorkerPodUid = "ateom-1" }), + obj: valid(func(r *ateletpb.RestoreRequest) { r.WorkerPodUid = "worker-1" }), want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { name: "invalid atespace: path escape", @@ -768,6 +778,14 @@ func TestValidateRestoreRequest(t *testing.T) { name: "invalid snapshot_uuid: path escape", obj: valid(local("../escape")), want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "snapshot_uuid"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing generation", + obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.Generation = 0 }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "generation"), "")}, + }, { + name: "negative generation", + obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.Generation = -1 }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "generation"), nil, "").WithOrigin("minimum")}, }, { name: "missing snapshot_uri", obj: valid(func(r *ateletpb.RestoreRequest) { r.Snapshot.Object.SnapshotUri = "" }), @@ -835,6 +853,7 @@ func TestValidatePromoteSnapshotRequest(t *testing.T) { ActorTemplateName: "tmpl-1", Snapshot: &ateletpb.Snapshot{ SnapshotUuid: "pause-snap-1", + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, Object: &ateletpb.ObjectStorage{ SnapshotUri: "gs://bucket/root/atespaces/team-a/actors/01234567-89ab-cdef-0123-456789abcdef/snapshots/snap-1", @@ -905,6 +924,14 @@ func TestValidatePromoteSnapshotRequest(t *testing.T) { name: "invalid snapshot_uuid: path escape", obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.SnapshotUuid = "../escape" }), want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "snapshot_uuid"), nil, "").WithOrigin("format=k8s-short-name")}, + }, { + name: "missing generation", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.Generation = 0 }), + want: field.ErrorList{field.Required(field.NewPath("snapshot", "generation"), "")}, + }, { + name: "negative generation", + obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.Generation = -1 }), + want: field.ErrorList{field.Invalid(field.NewPath("snapshot", "generation"), nil, "").WithOrigin("minimum")}, }, { name: "missing snapshot.object", obj: valid(func(r *ateletpb.PromoteSnapshotRequest) { r.Snapshot.Object = nil }), @@ -984,7 +1011,7 @@ func TestValidateTerminateRequest(t *testing.T) { want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { name: "invalid worker_pod_uid: not a pod UID", - obj: valid(func(r *ateletpb.TerminateRequest) { r.WorkerPodUid = "ateom-1" }), + obj: valid(func(r *ateletpb.TerminateRequest) { r.WorkerPodUid = "worker-1" }), want: field.ErrorList{field.Invalid(field.NewPath("worker_pod_uid"), nil, "").WithOrigin("format=k8s-uuid")}, }, { name: "missing atespace", diff --git a/cmd/atelet/internal/apivalidation/zz_generated.validation.go b/cmd/atelet/internal/apivalidation/zz_generated.validation.go index fb86928436..8cbdbae51c 100644 --- a/cmd/atelet/internal/apivalidation/zz_generated.validation.go +++ b/cmd/atelet/internal/apivalidation/zz_generated.validation.go @@ -3423,12 +3423,16 @@ func Validate_Snapshot( } // call field-attached validations earlyReturn := false - if e := validate.OptionalValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + if e := validate.RequiredValue(ctx, op, fldPath, obj, oldObj).MarkShortCircuit(); len(e) != 0 { + errs = append(errs, e...) earlyReturn = true } if earlyReturn { return // do not proceed } + if e := validate.Minimum(ctx, op, fldPath, obj, oldObj, 1); len(e) != 0 { + errs = append(errs, e...) + } return } oldVal := safe.Field(oldObj, diff --git a/cmd/atelet/internal/ateletpath/ateletpath.go b/cmd/atelet/internal/ateletpath/ateletpath.go index 9a5ff54907..fdae2ec031 100644 --- a/cmd/atelet/internal/ateletpath/ateletpath.go +++ b/cmd/atelet/internal/ateletpath/ateletpath.go @@ -13,21 +13,21 @@ // limitations under the License. // Package ateletpath is atelet's on-node layout: the per-actor directories -// it passes to ateom as ActorDirs, and the directories only atelet uses. +// it passes to worker as ActorDirs, and the directories only atelet uses. package ateletpath import ( "path/filepath" "github.com/agent-substrate/substrate/internal/nodepath" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) var ( // ImageCacheDir is the node-local OCI image layer cache (see // internal/imagecache). It lives under BasePath so the cached layer // directories are visible at the same path in atelet (which writes them) - // and in every ateom pod (which mounts them as overlay lowerdirs). + // and in every worker pod (which mounts them as overlay lowerdirs). ImageCacheDir = filepath.Join(nodepath.BasePath, "image-cache") // SnapshotScratchDir holds short-lived directories for snapshot manifests @@ -70,6 +70,17 @@ func ActorSandboxAssetsFile(actorUID string) string { ) } +// ActorSnapshotStateFile is the per-actor file where atelet keeps what it +// tracks of the actor's snapshots: its latest local and durable snapshots, so +// a restarted atelet still knows them. Like ActorSandboxAssetsFile it lives +// directly under ActorPath, outside the subdirs resetActorDirs wipes. +func ActorSnapshotStateFile(actorUID string) string { + return filepath.Join( + ActorPath(actorUID), + "snapshot-state.json", + ) +} + func OCIBundleDir(actorUID string) string { return filepath.Join( ActorPath(actorUID), @@ -127,7 +138,7 @@ func DurableDirVolumeMountPoint(actorUID, volumeName string) string { // Run/Restore; each sandbox class excludes them differently: // // - micro-VM captures by location: its checkpoint tars all of -// DurableDirVolumeMountsDir (see ateom-microvm's tarDurableVolumes), so +// DurableDirVolumeMountsDir (see ateworker-microvm's tarDurableVolumes), so // system-info roots are excluded by living in this separate directory. // - gVisor captures by declaration: durable mounts are registered with // the sandbox (mount-hint annotations for MEMORY checkpoints, the @@ -184,10 +195,10 @@ func VolumeHostPath(actorUID, volumeName string) string { ) } -// ActorDirs is the directory set atelet passes to ateom for an actor. ateom +// ActorDirs is the directory set atelet passes to worker for an actor. worker // takes these from the request rather than deriving them from the actor UID. -func ActorDirs(actorUID string) *ateompb.ActorDirs { - return &ateompb.ActorDirs{ +func ActorDirs(actorUID string) *ateworkerpb.ActorDirs { + return &ateworkerpb.ActorDirs{ RootDir: ActorPath(actorUID), OciBundleDir: OCIBundleDir(actorUID), CheckpointDir: CheckpointStateDir(actorUID), diff --git a/cmd/atelet/internal/atomicfile/atomicfile.go b/cmd/atelet/internal/atomicfile/atomicfile.go new file mode 100644 index 0000000000..d9c71c70cd --- /dev/null +++ b/cmd/atelet/internal/atomicfile/atomicfile.go @@ -0,0 +1,49 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package atomicfile writes files that readers never observe partly written. +package atomicfile + +import ( + "os" + "path/filepath" +) + +// Write writes data to path by writing a temp file in the same directory and +// renaming it over the target so readers never observe a truncated or +// partially written value. +// +// TODO: Write does not fsync, so after a node crash path can be empty or still +// hold its old contents. Investigate an opt-in sync for callers whose files +// must survive a crash; other callers don't need it. +func Write(path string, data []byte, perm os.FileMode) error { + f, err := os.CreateTemp(filepath.Dir(path), "."+filepath.Base(path)+".tmp-*") + if err != nil { + return err + } + defer os.Remove(f.Name()) // no-op once the rename succeeds + + if _, err := f.Write(data); err != nil { + f.Close() + return err + } + if err := f.Chmod(perm); err != nil { + f.Close() + return err + } + if err := f.Close(); err != nil { + return err + } + return os.Rename(f.Name(), path) +} diff --git a/cmd/atelet/internal/atomicfile/atomicfile_test.go b/cmd/atelet/internal/atomicfile/atomicfile_test.go new file mode 100644 index 0000000000..94d2576352 --- /dev/null +++ b/cmd/atelet/internal/atomicfile/atomicfile_test.go @@ -0,0 +1,69 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package atomicfile + +import ( + "os" + "path/filepath" + "testing" +) + +func TestWrite(t *testing.T) { + dir := t.TempDir() + target := filepath.Join(dir, "state.json") + + // One shared write over an existing file; each subtest checks one + // postcondition. + if err := os.WriteFile(target, []byte("old"), 0o600); err != nil { + t.Fatalf("seeding target: %v", err) + } + if err := Write(target, []byte("new"), 0o644); err != nil { + t.Fatalf("Write: %v", err) + } + + t.Run("replaces content", func(t *testing.T) { + got, err := os.ReadFile(target) + if err != nil { + t.Fatalf("reading target: %v", err) + } + if string(got) != "new" { + t.Errorf("content = %q, want %q", got, "new") + } + }) + + t.Run("sets permissions", func(t *testing.T) { + info, err := os.Stat(target) + if err != nil { + t.Fatalf("stat target: %v", err) + } + if perm := info.Mode().Perm(); perm != 0o644 { + t.Errorf("perm = %o, want 644", perm) + } + }) + + t.Run("leaves no temp files", func(t *testing.T) { + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatalf("reading dir: %v", err) + } + if len(entries) != 1 { + names := make([]string, 0, len(entries)) + for _, e := range entries { + names = append(names, e.Name()) + } + t.Errorf("files in the target's directory = %v, want only the target", names) + } + }) +} diff --git a/cmd/atelet/internal/filecache/getfilecopyto_test.go b/cmd/atelet/internal/filecache/getfilecopyto_test.go index 3a9dcf038b..dc468d530f 100644 --- a/cmd/atelet/internal/filecache/getfilecopyto_test.go +++ b/cmd/atelet/internal/filecache/getfilecopyto_test.go @@ -73,7 +73,7 @@ func TestGetFileCopyToMutationIsolation(t *testing.T) { t.Fatal(err) } // The consumer rewrites its staged file in place — the whole reason copy - // mode exists (ateom-microvm does this to config.json). + // mode exists (ateworker-microvm does this to config.json). if err := os.WriteFile(first, []byte("mutated by consumer"), 0o600); err != nil { t.Fatalf("consumer write to its copy: %v", err) } diff --git a/cmd/atelet/internal/snapshottracker/actor.go b/cmd/atelet/internal/snapshottracker/actor.go new file mode 100644 index 0000000000..ec4ecbb724 --- /dev/null +++ b/cmd/atelet/internal/snapshottracker/actor.go @@ -0,0 +1,117 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package snapshottracker + +import ( + "encoding/json" + "time" + + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "google.golang.org/protobuf/encoding/protojson" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/known/timestamppb" +) + +// SnapshotRef is one snapshot of an actor that atelet holds or has uploaded. +type SnapshotRef struct { + // Snapshot is the snapshot as the control plane named it. + Snapshot *ateletpb.Snapshot + // CommitTime is when the checkpoint that produced the snapshot committed + // on this node. Zero when this atelet did not take it. + CommitTime time.Time + // UploadTime is when this atelet committed the snapshot's manifest to + // object storage. Zero unless it did. + UploadTime time.Time +} + +// snapshotRefJSON is a SnapshotRef as the snapshot state file stores it. +// Snapshot is protojson, so the file keeps every field of ateletpb.Snapshot. +type snapshotRefJSON struct { + Snapshot json.RawMessage `json:"snapshot"` + CommitTime time.Time `json:"commitTime,omitzero"` + UploadTime time.Time `json:"uploadTime,omitzero"` +} + +// MarshalJSON encodes r as a snapshotRefJSON. +func (r SnapshotRef) MarshalJSON() ([]byte, error) { + snap, err := protojson.Marshal(r.Snapshot) + if err != nil { + return nil, err + } + return json.Marshal(snapshotRefJSON{Snapshot: snap, CommitTime: r.CommitTime, UploadTime: r.UploadTime}) +} + +// UnmarshalJSON decodes a snapshotRefJSON into r. +func (r *SnapshotRef) UnmarshalJSON(data []byte) error { + var rec snapshotRefJSON + if err := json.Unmarshal(data, &rec); err != nil { + return err + } + snap := &ateletpb.Snapshot{} + if err := protojson.Unmarshal(rec.Snapshot, snap); err != nil { + return err + } + *r = SnapshotRef{Snapshot: snap, CommitTime: rec.CommitTime, UploadTime: rec.UploadTime} + return nil +} + +// clone returns a copy of r that shares no memory with it. +func (r SnapshotRef) clone() *SnapshotRef { + r.Snapshot = proto.CloneOf(r.Snapshot) + return &r +} + +// report converts the SnapshotRef for a field of ActorSnapshotState. +func (r *SnapshotRef) report() *ateletpb.SnapshotReport { + if r == nil { + return nil + } + rep := &ateletpb.SnapshotReport{Snapshot: proto.CloneOf(r.Snapshot)} + if !r.CommitTime.IsZero() { + rep.LocalCommitTime = timestamppb.New(r.CommitTime) + } + if !r.UploadTime.IsZero() { + rep.DurableCompleteTime = timestamppb.New(r.UploadTime) + } + return rep +} + +// actorSnapshotsRecord is the part of actorSnapshots saved to the actor's +// snapshot state file. +type actorSnapshotsRecord struct { + // LatestLocal is the newest snapshot currently stored in the actor's + // node-local checkpoint directory. Nil after an UPLOAD_MODE_IMMEDIATE + // checkpoint. + LatestLocal *SnapshotRef `json:"latestLocal,omitempty"` + // LatestDurable is the newest snapshot confirmed in object storage. + LatestDurable *SnapshotRef `json:"latestDurable,omitempty"` +} + +// actorSnapshots is what atelet tracks of one actor's snapshots on this node. +// Its fields are guarded by the actor's lock in Tracker.locks. +type actorSnapshots struct { + actorUID string + + // actorSnapshotsRecord holds what the snapshot state file keeps. + actorSnapshotsRecord + + // op is the actor's background operation in progress, or nil. An actor + // has at most one. The entry stays in memory while op is set. + op *actorOp + // unsaved is set while the snapshot state file lacks a change to the + // entry because writing the file failed. The entry stays in memory while + // unsaved is set. + unsaved bool +} diff --git a/cmd/atelet/internal/snapshottracker/background.go b/cmd/atelet/internal/snapshottracker/background.go new file mode 100644 index 0000000000..303dfeb5b2 --- /dev/null +++ b/cmd/atelet/internal/snapshottracker/background.go @@ -0,0 +1,180 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package snapshottracker + +import ( + "context" + "log/slog" + "time" +) + +// Fixed until they become flags or per-request settings on UploadConfig. +const ( + // snapshotRetryDelay is how long atelet holds an actor's background work + // back after a failure. Each further consecutive failure doubles it, up to + // maxSnapshotRetryDelay. + snapshotRetryDelay = time.Minute + maxSnapshotRetryDelay = 10 * time.Minute + // snapshotWorkers is how many background workers run at once on a node. + snapshotWorkers = 4 +) + +// actorOp is an actor's background operation in progress. An actor has at +// most one. The operation holds the actor's lock only while it begins and +// ends, not while it runs, so Checkpoint and Terminate never wait for it. +type actorOp struct { + // target is the snapshot the operation acts on. + target SnapshotRef + // cancel ends the context the operation runs under. + cancel context.CancelFunc +} + +// Start queues every actor with a snapshot state file on this node, so its +// background work resumes after an atelet restart, and starts the workers +// that run actors' background work. A worker reads each actor's file and +// queues the actor for when its next background work is due. +func (t *Tracker) Start() { + actorUIDs, err := savedActorUIDs() + if err != nil { + // Best-effort: each actor's next commit queues it again. + slog.Warn("Failed to find the actors with snapshot state to queue", slog.Any("err", err)) + } + for _, actorUID := range actorUIDs { + t.queue.Add(actorUID) + } + for range t.workers { + t.workersDone.Go(t.runWorker) + } +} + +// Wait blocks until the workers have stopped, which they do once the tracker's +// context ends, or until ctx ends. It returns ctx's error if ctx ended first. +func (t *Tracker) Wait(ctx context.Context) error { + stopped := make(chan struct{}) + go func() { + t.workersDone.Wait() + close(stopped) + }() + select { + case <-stopped: + return nil + case <-ctx.Done(): + return ctx.Err() + } +} + +// runWorker runs the background work of the actors it takes from the queue, +// until the queue shuts down. +func (t *Tracker) runWorker() { + for { + actorUID, shutdown := t.queue.Get() + if shutdown { + return + } + t.syncActor(actorUID) + t.queue.Done(actorUID) + } +} + +// syncActor runs the actor's due background work, and otherwise queues the +// actor for when its next background work is due. +func (t *Tracker) syncActor(actorUID string) { + if !t.locks.Lock(t.ctx, actorUID) { + // atelet is shutting down. + return + } + defer t.unlock(actorUID) + if t.ctx.Err() != nil { + // atelet is shutting down. + return + } + a := t.findLocked(actorUID) + if a == nil || a.op != nil { + // Forgotten, or an operation is in progress, whose caller queues the + // actor again when it ends. + return + } + // TODO: Run the actor's due background work here, such as uploading its + // latest local snapshot. Begin it with beginOpLocked and release the + // actor's lock while it runs, so Checkpoint and Terminate never wait for + // it. Then retake the lock and end it with endOpLocked. + t.requeueLocked(a) +} + +// requeueLocked queues the actor for when its next background work is due, +// then drops the actor's entry from memory unless the entry has an operation +// in progress or a change that failed to save. An actor whose directory is +// gone has nothing to save, so its entry is dropped. requeueLocked runs after +// a change to the actor is saved, so it must not change what the snapshot +// state file keeps. No background work is defined, so it queues nothing. +func (t *Tracker) requeueLocked(a *actorSnapshots) { + if a.op == nil && !a.unsaved { + t.dropLocked(a.actorUID) + } +} + +// beginOpLocked records an operation on target as the actor's background +// operation in progress, which keeps the actor's entry in memory until the +// operation ends, and returns it with the context it runs under. The +// operation and its context are nil when the actor already has an operation +// in progress. The caller releases the actor's lock while the operation runs. +func (t *Tracker) beginOpLocked(a *actorSnapshots, target SnapshotRef) (context.Context, *actorOp) { + if a.op != nil { + return nil, nil + } + ctx, cancel := context.WithCancel(t.ctx) + a.op = &actorOp{target: *target.clone(), cancel: cancel} + t.addLocked(a) + return ctx, a.op +} + +// endOpLocked ends op and returns the actor's entry, or nil once Forget has +// abandoned op: the caller must then leave the actor's state and local +// snapshots alone. Otherwise the caller records the operation's outcome on +// the entry, saves the entry if that changed it, and calls requeueLocked. +func (t *Tracker) endOpLocked(actorUID string, op *actorOp) *actorSnapshots { + op.cancel() + a := t.findLocked(actorUID) + if a == nil || a.op != op { + return nil + } + a.op = nil + return a +} + +// abandonOpLocked cancels the actor's background operation in progress, if +// any, without waiting for it to end. +func (t *Tracker) abandonOpLocked(actorUID string) { + // An entry with an operation in progress is always in memory. + t.mu.Lock() + a := t.actors[actorUID] + t.mu.Unlock() + if a != nil && a.op != nil { + a.op.cancel() + a.op = nil + } +} + +// backOffLocked records a failure of the actor's background work, and returns +// when the actor may try again: later with each consecutive failure. +func (t *Tracker) backOffLocked(actorUID string) time.Time { + return t.clock.Now().Add(t.retries.When(actorUID)) +} + +// resetBackoffLocked forgets the actor's consecutive failures, so its next +// failure holds it back for snapshotRetryDelay again. +func (t *Tracker) resetBackoffLocked(actorUID string) { + t.retries.Forget(actorUID) +} diff --git a/cmd/atelet/internal/snapshottracker/background_test.go b/cmd/atelet/internal/snapshottracker/background_test.go new file mode 100644 index 0000000000..4c4f40c005 --- /dev/null +++ b/cmd/atelet/internal/snapshottracker/background_test.go @@ -0,0 +1,325 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package snapshottracker + +import ( + "context" + "errors" + "os" + "path/filepath" + "slices" + "testing" + "time" + + "github.com/agent-substrate/substrate/cmd/atelet/internal/ateletpath" + "github.com/agent-substrate/substrate/internal/nodepath" + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "google.golang.org/protobuf/proto" + clocktesting "k8s.io/utils/clock/testing" +) + +// useActorDir roots atelet's on-node state in temp directories and creates the +// tracked actor's directory, so the tracker writes its snapshot state file. +func useActorDir(t *testing.T) { + t.Helper() + useTempNodeDirs(t) + if err := os.MkdirAll(ateletpath.ActorPath(trackedActorUID), 0o700); err != nil { + t.Fatal(err) + } +} + +// returnsPromptly fails the test unless f returns within a minute. +func returnsPromptly(t *testing.T, what string, f func()) { + t.Helper() + done := make(chan struct{}) + go func() { + defer close(done) + f() + }() + select { + case <-done: + case <-time.After(time.Minute): + t.Fatalf("%s did not return within a minute", what) + } +} + +// beginOp begins a background operation on target for the tracked actor, as a +// worker does, and returns it with the context it runs under. +func beginOp(t *testing.T, tr *Tracker, target SnapshotRef) (context.Context, *actorOp) { + t.Helper() + tr.lock(trackedActorUID) + defer tr.unlock(trackedActorUID) + ctx, op := tr.beginOpLocked(tr.findOrAddLocked(trackedActorUID), target) + if op == nil { + t.Fatal("beginOpLocked: the actor already has an operation in progress") + } + return ctx, op +} + +// endOp ends op, as a worker does, and reports whether it was still the +// tracked actor's operation in progress. It then finishes like a worker +// whose operation changed nothing, calling requeueLocked. +func endOp(tr *Tracker, op *actorOp) bool { + tr.lock(trackedActorUID) + defer tr.unlock(trackedActorUID) + a := tr.endOpLocked(trackedActorUID, op) + if a == nil { + return false + } + tr.requeueLocked(a) + return true +} + +// inMemory reports whether the tracker holds the tracked actor's entry in +// memory. +func inMemory(tr *Tracker) bool { + tr.mu.Lock() + defer tr.mu.Unlock() + _, ok := tr.actors[trackedActorUID] + return ok +} + +// Start runs the workers until the tracker's context ends. Wait then returns +// once they have stopped, even when a worker has taken an actor whose lock an +// RPC still holds. +func TestWorkersStopWhenContextEnds(t *testing.T) { + useTempNodeDirs(t) + ctx, cancel := context.WithCancel(t.Context()) + tr := New(ctx, clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0))) + tr.Start() + tr.lock(trackedActorUID) + defer tr.unlock(trackedActorUID) + tr.queue.Add(trackedActorUID) + cancel() + returnsPromptly(t, "Wait", func() { + if err := tr.Wait(t.Context()); err != nil { + t.Errorf("Wait: %v", err) + } + }) +} + +// Wait gives up once its own context ends, while the workers still run. +func TestWaitGivesUpWhenItsContextEnds(t *testing.T) { + useTempNodeDirs(t) + tr := newTracker(t) + tr.Start() + ctx, cancel := context.WithTimeout(t.Context(), 10*time.Millisecond) + defer cancel() + if err := tr.Wait(ctx); !errors.Is(err, context.DeadlineExceeded) { + t.Errorf("Wait with the workers running = %v, want %v", err, context.DeadlineExceeded) + } +} + +// An actor's entry stays in memory while it has an operation in progress, +// even as commits save it, and requeueLocked drops it once the operation ends. +func TestEntryStaysInMemoryDuringOp(t *testing.T) { + useActorDir(t) + tr := newTracker(t) + s1 := testRef("s1", 1, time.Now()) + tr.CommitLocal(trackedActorUID, s1) + if inMemory(tr) { + t.Error("entry in memory after a commit with no operation in progress, want it dropped") + } + + _, op := beginOp(t, tr, s1) + tr.CommitLocal(trackedActorUID, testRef("s2", 2, time.Now())) + if !inMemory(tr) { + t.Error("entry dropped from memory during an operation, want it kept") + } + + if !endOp(tr, op) { + t.Fatal("endOpLocked = nil, want the operation still the actor's") + } + if inMemory(tr) { + t.Error("entry in memory after the operation ended, want it dropped") + } + if got := uuidOf(newTracker(t).State(trackedActorUID).GetLatestLocal()); got != "s2" { + t.Errorf("latest_local from the state file = %q, want %q", got, "s2") + } +} + +// Start queues each actor whose snapshot state file survived an atelet +// restart, and skips actor directories without one. A node with no actor +// directories yet queues nothing. +func TestStartQueuesActorsWithSnapshotState(t *testing.T) { + useTempNodeDirs(t) + // queued starts a tracker, as a restarted atelet does, and returns the + // actors it queued. + queued := func() []string { + tr := newTracker(t) + tr.workers = 0 // Leave the queue for the test to read. + tr.Start() + var actorUIDs []string + for tr.queue.Len() > 0 { + actorUID, _ := tr.queue.Get() + actorUIDs = append(actorUIDs, actorUID) + tr.queue.Done(actorUID) + } + slices.Sort(actorUIDs) + return actorUIDs + } + if got := queued(); len(got) != 0 { + t.Errorf("queued on a node with no actor directories = %v, want none", got) + } + + for _, actorUID := range []string{"actor-a", "actor-b", "actor-without-state"} { + if err := os.MkdirAll(ateletpath.ActorPath(actorUID), 0o700); err != nil { + t.Fatal(err) + } + } + if err := os.WriteFile(filepath.Join(nodepath.ActorsDir, "not-an-actor"), nil, 0o600); err != nil { + t.Fatal(err) + } + before := newTracker(t) + before.CommitLocal("actor-a", testRef("s1", 1, time.Now())) + before.CommitDurable("actor-b", testRef("s2", 2, time.Now())) + if got, want := queued(), []string{"actor-a", "actor-b"}; !slices.Equal(got, want) { + t.Errorf("queued after a restart = %v, want %v", got, want) + } +} + +// Forget does not wait for the actor's background operation in progress. It +// cancels the operation, which finds itself abandoned when it ends, so it +// leaves alone both the forgotten actor's state and an operation that began +// for the actor after Forget. +func TestForgetAbandonsInFlightOp(t *testing.T) { + useActorDir(t) + tr := newTracker(t) + s1 := testRef("s1", 1, time.Now()) + tr.CommitLocal(trackedActorUID, s1) + opCtx, op := beginOp(t, tr, s1) + + returnsPromptly(t, "Forget during a background operation", func() { + if err := tr.Forget(trackedActorUID); err != nil { + t.Errorf("Forget: %v", err) + } + }) + if opCtx.Err() == nil { + t.Error("the operation's context is live after Forget, want it cancelled") + } + _, next := beginOp(t, tr, s1) + if endOp(tr, op) { + t.Error("endOpLocked = true after Forget, want the operation abandoned") + } + if got := tr.State(trackedActorUID); !proto.Equal(got, &ateletpb.ActorSnapshotState{}) { + t.Errorf("state after forget = %v, want empty", got) + } + if !endOp(tr, next) { + t.Error("endOpLocked = false for the operation begun after Forget, want the abandoned one's end to leave it alone") + } +} + +// A Checkpoint's commits, PromoteSnapshot's, and State neither wait for the +// actor's background operation in progress nor cancel it. +func TestCommitsDoNotWaitForInFlightOp(t *testing.T) { + useActorDir(t) + tr := newTracker(t) + t0 := time.Now() + s1 := testRef("s1", 1, t0) + tr.CommitLocal(trackedActorUID, s1) + opCtx, op := beginOp(t, tr, s1) + + for _, c := range []struct { + name string + call func() + }{ + {"CommitLocal", func() { tr.CommitLocal(trackedActorUID, testRef("s2", 2, t0)) }}, + {"ForgetLocal", func() { tr.ForgetLocal(trackedActorUID) }}, + {"CommitDurable", func() { tr.CommitDurable(trackedActorUID, testRef("s3", 3, t0)) }}, + {"CommitPromoted", func() { tr.CommitPromoted(trackedActorUID, testRef("s4", 4, t0)) }}, + {"State", func() { tr.State(trackedActorUID) }}, + } { + returnsPromptly(t, c.name+" during a background operation", c.call) + } + if err := opCtx.Err(); err != nil { + t.Errorf("the operation's context ended with %v, want it still running", err) + } + if !endOp(tr, op) { + t.Error("endOpLocked = false, want the operation still the actor's") + } +} + +// An actor has at most one background operation in progress, and another +// actor's operation does not stand in its way. The operation keeps its own +// copy of its target. +func TestBeginOpLockedAllowsOneOpPerActor(t *testing.T) { + tr := newTracker(t) + s1 := testRef("s1", 1, time.Now()) + _, op := beginOp(t, tr, s1) + + tr.lock(trackedActorUID) + _, second := tr.beginOpLocked(tr.findOrAddLocked(trackedActorUID), s1) + tr.unlock(trackedActorUID) + if second != nil { + t.Error("beginOpLocked began a second operation for the actor, want none") + } + + const otherActorUID = "other-actor-uid" + tr.lock(otherActorUID) + _, other := tr.beginOpLocked(tr.findOrAddLocked(otherActorUID), s1) + tr.unlock(otherActorUID) + if other == nil { + t.Error("beginOpLocked for another actor = nil, want an operation") + } + + if !endOp(tr, op) { + t.Fatal("endOpLocked = false, want the operation still the actor's") + } + _, op = beginOp(t, tr, s1) + s1.Snapshot.SnapshotUuid = "changed" + if got := op.target.Snapshot.GetSnapshotUuid(); got != "s1" { + t.Errorf("operation target = %q after the caller changed its snapshot, want %q", got, "s1") + } +} + +// A failure holds the actor's background work back for snapshotRetryDelay, +// doubling with each consecutive failure up to maxSnapshotRetryDelay. Making +// the actor's latest snapshot durable, or forgetting the actor, starts the +// doubling over. +func TestTrackerBackoff(t *testing.T) { + useActorDir(t) + clk := clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0)) + tr := New(t.Context(), clk) + fail := func() time.Duration { + tr.lock(trackedActorUID) + defer tr.unlock(trackedActorUID) + return tr.backOffLocked(trackedActorUID).Sub(clk.Now()) + } + for i, want := range []time.Duration{time.Minute, 2 * time.Minute, 4 * time.Minute, 8 * time.Minute, 10 * time.Minute, 10 * time.Minute} { + if got := fail(); got != want { + t.Errorf("failure %d held the actor back %v, want %v", i+1, got, want) + } + } + + for _, lift := range []struct { + name string + call func() + }{ + {"CommitDurable", func() { tr.CommitDurable(trackedActorUID, testRef("s1", 1, clk.Now())) }}, + {"CommitPromoted", func() { tr.CommitPromoted(trackedActorUID, testRef("s2", 2, clk.Now())) }}, + {"Forget", func() { + if err := tr.Forget(trackedActorUID); err != nil { + t.Fatalf("Forget: %v", err) + } + }}, + } { + fail() + fail() + lift.call() + if got := fail(); got != snapshotRetryDelay { + t.Errorf("the first failure after %s held the actor back %v, want %v", lift.name, got, snapshotRetryDelay) + } + } +} diff --git a/cmd/atelet/internal/snapshottracker/lifecycle.go b/cmd/atelet/internal/snapshottracker/lifecycle.go new file mode 100644 index 0000000000..fde83307b7 --- /dev/null +++ b/cmd/atelet/internal/snapshottracker/lifecycle.go @@ -0,0 +1,108 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package snapshottracker + +import ( + "github.com/agent-substrate/substrate/internal/proto/ateletpb" +) + +// CommitLocal records snap, just stored locally under UPLOAD_MODE_DEFERRED, as +// the actor's latest local snapshot, and returns the actor's state. +func (t *Tracker) CommitLocal(actorUID string, snap SnapshotRef) *ateletpb.ActorSnapshotState { + t.lock(actorUID) + defer t.unlock(actorUID) + a := t.findOrAddLocked(actorUID) + a.LatestLocal = snap.clone() + // TODO: Use succeededLocked here once UPLOAD_MODE_DEFERRED uploads are + // implemented: a deferred checkpoint is then a successful end state, which + // starts the actor's backoff over. + t.saveLocked(a) + t.requeueLocked(a) + return t.stateLocked(a) +} + +// CommitDurable records snap, uploaded during Checkpoint under +// UPLOAD_MODE_IMMEDIATE, as the actor's latest snapshot. The actor keeps no +// local snapshot. It returns the actor's state. +func (t *Tracker) CommitDurable(actorUID string, snap SnapshotRef) *ateletpb.ActorSnapshotState { + t.lock(actorUID) + defer t.unlock(actorUID) + a := t.findOrAddLocked(actorUID) + a.LatestDurable = snap.clone() + // TODO: change this when we add detach. + a.LatestLocal = nil + t.succeededLocked(a) + return t.stateLocked(a) +} + +// ForgetLocal drops the actor's latest local snapshot. Call it before deleting +// the actor's local checkpoints, so the state never names a deleted snapshot, +// even if the caller then fails. +func (t *Tracker) ForgetLocal(actorUID string) { + t.lock(actorUID) + defer t.unlock(actorUID) + a := t.findLocked(actorUID) + if a == nil || a.LatestLocal == nil { + return + } + a.LatestLocal = nil + t.addLocked(a) + t.saveLocked(a) + t.requeueLocked(a) +} + +// CommitPromoted records snap, the local snapshot PromoteSnapshot just +// uploaded, as the actor's latest snapshot, keeping the time its checkpoint +// committed. PromoteSnapshot then deletes the local copy, so the actor keeps +// no local snapshot. A retried promote of a snapshot already recorded keeps +// what was recorded. It returns the actor's state. +func (t *Tracker) CommitPromoted(actorUID string, snap SnapshotRef) *ateletpb.ActorSnapshotState { + t.lock(actorUID) + defer t.unlock(actorUID) + a := t.findOrAddLocked(actorUID) + uuid := snap.Snapshot.GetSnapshotUuid() + if d := a.LatestDurable; d == nil || d.Snapshot.GetSnapshotUuid() != uuid { + if l := a.LatestLocal; l != nil && l.Snapshot.GetSnapshotUuid() == uuid { + snap.CommitTime = l.CommitTime + } + a.LatestDurable = snap.clone() + } + // TODO: change this when PromoteSnapshot keeps the local copy. + a.LatestLocal = nil + t.succeededLocked(a) + return t.stateLocked(a) +} + +// Forget drops the actor's state, in memory and on disk. It does not wait for +// the actor's background operation in progress: it cancels it, and the +// operation finds itself abandoned when it ends. +func (t *Tracker) Forget(actorUID string) error { + t.lock(actorUID) + defer t.unlock(actorUID) + t.abandonOpLocked(actorUID) + t.resetBackoffLocked(actorUID) + t.dropLocked(actorUID) + return removeSnapshotState(actorUID) +} + +// succeededLocked finishes a change that leaves the actor in a successful end +// state, such as its latest snapshot being durable: it starts the actor's +// backoff over, saves the actor's state, and queues the actor for its next +// background work. +func (t *Tracker) succeededLocked(a *actorSnapshots) { + t.resetBackoffLocked(a.actorUID) + t.saveLocked(a) + t.requeueLocked(a) +} diff --git a/cmd/atelet/internal/snapshottracker/state.go b/cmd/atelet/internal/snapshottracker/state.go new file mode 100644 index 0000000000..1d9eef3d92 --- /dev/null +++ b/cmd/atelet/internal/snapshottracker/state.go @@ -0,0 +1,99 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package snapshottracker + +import ( + "encoding/json" + "errors" + "fmt" + "log/slog" + "os" + + "github.com/agent-substrate/substrate/cmd/atelet/internal/ateletpath" + "github.com/agent-substrate/substrate/cmd/atelet/internal/atomicfile" + "github.com/agent-substrate/substrate/internal/nodepath" +) + +// saveLocked writes what atelet tracks of the actor to its snapshot state +// file. An actor whose directory is gone was terminated on this node, so it is +// not written, and requeueLocked drops its entry as it does once the file is +// written. Best-effort: if writing fails, the entry stays in memory and atelet +// carries on from it, and a restart loses what changed. +func (t *Tracker) saveLocked(a *actorSnapshots) { + if _, err := os.Stat(ateletpath.ActorPath(a.actorUID)); errors.Is(err, os.ErrNotExist) { + slog.Warn("Dropping the snapshot state of an actor whose directory is gone", slog.String("actorUID", a.actorUID)) + a.unsaved = false + return + } + data, err := json.Marshal(a.actorSnapshotsRecord) + if err == nil { + err = atomicfile.Write(ateletpath.ActorSnapshotStateFile(a.actorUID), data, 0o600) + } + a.unsaved = err != nil + if err != nil { + slog.Warn("Failed to save the actor's snapshot state", slog.String("actorUID", a.actorUID), slog.Any("err", err)) + } +} + +// readSnapshotState reads what atelet tracked of the actor from its snapshot +// state file, or returns nil when the file does not exist or cannot be read. +func readSnapshotState(actorUID string) *actorSnapshots { + data, err := os.ReadFile(ateletpath.ActorSnapshotStateFile(actorUID)) + if errors.Is(err, os.ErrNotExist) { + return nil + } + var rec actorSnapshotsRecord + if err == nil { + err = json.Unmarshal(data, &rec) + } + if err != nil { + slog.Warn("Skipping an unreadable snapshot state file", slog.String("actorUID", actorUID), slog.Any("err", err)) + return nil + } + return &actorSnapshots{actorUID: actorUID, actorSnapshotsRecord: rec} +} + +// removeSnapshotState deletes the actor's snapshot state file. A missing file +// is not an error. +func removeSnapshotState(actorUID string) error { + if err := os.Remove(ateletpath.ActorSnapshotStateFile(actorUID)); err != nil && !errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("while deleting the actor's snapshot state: %w", err) + } + return nil +} + +// savedActorUIDs returns the UIDs of the actors with a snapshot state file on +// this node. A node with no actor directories yet has none. An actor whose +// file cannot be checked is included, so reading the file reports why. +func savedActorUIDs() ([]string, error) { + entries, err := os.ReadDir(nodepath.ActorsDir) + if errors.Is(err, os.ErrNotExist) { + return nil, nil + } + if err != nil { + return nil, fmt.Errorf("while listing actor directories: %w", err) + } + var actorUIDs []string + for _, entry := range entries { + if !entry.IsDir() { + continue + } + if _, err := os.Stat(ateletpath.ActorSnapshotStateFile(entry.Name())); errors.Is(err, os.ErrNotExist) { + continue + } + actorUIDs = append(actorUIDs, entry.Name()) + } + return actorUIDs, nil +} diff --git a/cmd/atelet/internal/snapshottracker/tracker.go b/cmd/atelet/internal/snapshottracker/tracker.go new file mode 100644 index 0000000000..50a4045296 --- /dev/null +++ b/cmd/atelet/internal/snapshottracker/tracker.go @@ -0,0 +1,150 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package snapshottracker tracks each actor's snapshots on an atelet node, and +// runs background work on them. +package snapshottracker + +import ( + "context" + "sync" + + "github.com/agent-substrate/substrate/internal/actorlock" + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "k8s.io/client-go/util/workqueue" + "k8s.io/utils/clock" +) + +// Tracker tracks each actor's snapshots on this node: the latest one stored +// locally, and the latest one confirmed in object storage. State reports them. +// +// Background work runs off a delaying workqueue of actor UIDs drained by a +// fixed pool of workers. The queue hands an actor to one worker at a time, and +// the number of workers bounds how much background work runs at once. +// +// TODO: Commits record the snapshot they are given without comparing +// Snapshot.generation. Checkpoint and PromoteSnapshot should reject a stale +// generation before doing any work, as Snapshot.generation describes, so that +// no commit replaces a snapshot with an older one. +type Tracker struct { + // ctx bounds background work, and shuts the queue down when it ends. + ctx context.Context + // clock is what due times are measured on, and the queue waits on. + clock clock.WithTicker + // retries says how long an actor's background work is held back after a + // failure: longer with each consecutive failure for the actor. + retries workqueue.TypedRateLimiter[string] + // workers is how many background workers run at once. + workers int + // queue holds the UIDs of actors with background work scheduled. + queue workqueue.TypedDelayingInterface[string] + // workersDone counts the workers still running. + workersDone sync.WaitGroup + + // locks holds each actor's lock. Work on an actor, including its snapshot + // state file I/O, holds that actor's lock, so it never holds up another + // actor. + locks *actorlock.Locks + + // mu guards the actors map, and is never held during file I/O. + mu sync.Mutex + // actors maps actorUID to the entry of an actor with a background + // operation in progress, or whose state its snapshot state file lacks + // because the save failed. requeueLocked drops any other entry, and + // findLocked then reads the state from the file. Read and change it + // through findLocked, findOrAddLocked, addLocked, and dropLocked. + actors map[string]*actorSnapshots +} + +// New returns a tracker that tracks no actor yet and measures time on clk. +// Its workers run once Start is called, until ctx ends. +func New(ctx context.Context, clk clock.WithTicker) *Tracker { + t := &Tracker{ + ctx: ctx, + clock: clk, + retries: workqueue.NewTypedItemExponentialFailureRateLimiter[string](snapshotRetryDelay, maxSnapshotRetryDelay), + workers: snapshotWorkers, + queue: workqueue.NewTypedDelayingQueueWithConfig(workqueue.TypedDelayingQueueConfig[string]{Clock: clk}), + locks: actorlock.New(), + actors: map[string]*actorSnapshots{}, + } + context.AfterFunc(ctx, t.queue.ShutDown) + return t +} + +// lock takes the actor's lock, which the methods named *Locked require. +func (t *Tracker) lock(actorUID string) { + // Lock gives up only when its context is done, which Background never is. + t.locks.Lock(context.Background(), actorUID) +} + +// unlock releases the actor's lock. +func (t *Tracker) unlock(actorUID string) { + t.locks.Unlock(actorUID) +} + +// findLocked returns the actor's entry from memory, or reads it from disk when +// not in memory. It returns nil when atelet tracks nothing of the actor. +func (t *Tracker) findLocked(actorUID string) *actorSnapshots { + t.mu.Lock() + a, ok := t.actors[actorUID] + t.mu.Unlock() + if ok { + return a + } + return readSnapshotState(actorUID) +} + +// findOrAddLocked returns the actor's entry, adding an empty one when there is +// none, so it never returns nil. +func (t *Tracker) findOrAddLocked(actorUID string) *actorSnapshots { + a := t.findLocked(actorUID) + if a == nil { + a = &actorSnapshots{actorUID: actorUID} + } + t.addLocked(a) + return a +} + +// addLocked makes a its actor's entry. +func (t *Tracker) addLocked(a *actorSnapshots) { + t.mu.Lock() + defer t.mu.Unlock() + t.actors[a.actorUID] = a +} + +// dropLocked removes the actor's entry. +func (t *Tracker) dropLocked(actorUID string) { + t.mu.Lock() + defer t.mu.Unlock() + delete(t.actors, actorUID) +} + +// stateLocked returns what atelet holds for the actor. +func (t *Tracker) stateLocked(a *actorSnapshots) *ateletpb.ActorSnapshotState { + return &ateletpb.ActorSnapshotState{ + LatestLocal: a.LatestLocal.report(), + LatestDurable: a.LatestDurable.report(), + } +} + +// State returns what atelet holds for the actor. +func (t *Tracker) State(actorUID string) *ateletpb.ActorSnapshotState { + t.lock(actorUID) + defer t.unlock(actorUID) + if a := t.findLocked(actorUID); a != nil { + return t.stateLocked(a) + } + return &ateletpb.ActorSnapshotState{} +} diff --git a/cmd/atelet/internal/snapshottracker/tracker_test.go b/cmd/atelet/internal/snapshottracker/tracker_test.go new file mode 100644 index 0000000000..c8e2ec72d0 --- /dev/null +++ b/cmd/atelet/internal/snapshottracker/tracker_test.go @@ -0,0 +1,310 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package snapshottracker + +import ( + "fmt" + "os" + "path/filepath" + "sync" + "testing" + "time" + + "github.com/agent-substrate/substrate/cmd/atelet/internal/ateletpath" + "github.com/agent-substrate/substrate/internal/nodepath" + "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/known/timestamppb" + clocktesting "k8s.io/utils/clock/testing" +) + +const trackedActorUID = "actor-uid-1" + +// useTempNodeDirs roots atelet's on-node state in temp directories so a test +// can drive the real filesystem layout. Not parallel-safe: the paths are +// process-global. +func useTempNodeDirs(t *testing.T) { + t.Helper() + root := t.TempDir() + origActors, origStatic := nodepath.ActorsDir, nodepath.StaticFilesDir + nodepath.ActorsDir = filepath.Join(root, "actors") + nodepath.StaticFilesDir = filepath.Join(root, "static-files") + t.Cleanup(func() { + nodepath.ActorsDir, nodepath.StaticFilesDir = origActors, origStatic + }) +} + +// newTracker returns a tracker on a fake clock whose context ends with the +// test. Its workers are not started. +func newTracker(t *testing.T) *Tracker { + return New(t.Context(), clocktesting.NewFakeClock(time.Unix(1_700_000_000, 0))) +} + +// testRef is a MEMORY snapshot of the tracked actor. +func testRef(uuid string, generation int32, commitTime time.Time) SnapshotRef { + return SnapshotRef{ + Snapshot: &ateletpb.Snapshot{ + SnapshotUuid: uuid, + Generation: generation, + Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + Object: &ateletpb.ObjectStorage{SnapshotUri: "gs://test-bucket/atespaces/ate-demo/actors/" + trackedActorUID + "/snapshots/" + uuid}, + }, + CommitTime: commitTime, + } +} + +func timeOf(ts *timestamppb.Timestamp) time.Time { + if ts == nil { + return time.Time{} + } + return ts.AsTime() +} + +func uuidOf(r *ateletpb.SnapshotReport) string { + return r.GetSnapshot().GetSnapshotUuid() +} + +// State reports the latest local snapshot a deferred checkpoint leaves, and +// the latest durable one an immediate checkpoint leaves without a local copy. +// Each update is written to the actor's snapshot state file and evicted from +// memory; Forget deletes the file and drops any in-memory state. +func TestTrackerState(t *testing.T) { + useTempNodeDirs(t) + if err := os.MkdirAll(ateletpath.ActorPath(trackedActorUID), 0o700); err != nil { + t.Fatal(err) + } + tr := newTracker(t) + t0 := time.Now() + st := tr.CommitLocal(trackedActorUID, testRef("s1", 1, t0)) + if got := st.GetLatestLocal(); uuidOf(got) != "s1" || !timeOf(got.GetLocalCommitTime()).Equal(t0) { + t.Errorf("latest_local = %v, want s1 committed at %v", got, t0) + } + if st.GetLatestDurable() != nil { + t.Errorf("latest_durable = %v, want unset", st.GetLatestDurable()) + } + if len(tr.actors) != 0 { + t.Errorf("in-memory actors after CommitLocal = %d, want 0 after writing to file", len(tr.actors)) + } + + s2 := testRef("s2", 2, t0.Add(time.Minute)) + s2.UploadTime = s2.CommitTime.Add(time.Second) + st = tr.CommitDurable(trackedActorUID, s2) + if st.GetLatestLocal() != nil { + t.Errorf("latest_local = %v after an immediate checkpoint, want unset", st.GetLatestLocal()) + } + if got := st.GetLatestDurable(); uuidOf(got) != "s2" || !timeOf(got.GetDurableCompleteTime()).Equal(s2.UploadTime) { + t.Errorf("latest_durable = %v, want s2 uploaded at %v", got, s2.UploadTime) + } + if len(tr.actors) != 0 { + t.Errorf("in-memory actors after CommitDurable = %d, want 0 after writing to file", len(tr.actors)) + } + + st = tr.CommitLocal(trackedActorUID, testRef("s3", 3, t0.Add(2*time.Minute))) + if uuidOf(st.GetLatestLocal()) != "s3" || uuidOf(st.GetLatestDurable()) != "s2" { + t.Errorf("state = %v, want s3 local and s2 still durable", st) + } + if got := tr.State(trackedActorUID); !proto.Equal(got, st) { + t.Errorf("State() = %v, want %v, as CommitLocal returned", got, st) + } + if got := newTracker(t).State(trackedActorUID); !proto.Equal(got, st) { + t.Errorf("State() on a fresh tracker = %v, want %v from the state file", got, st) + } + + if err := tr.Forget(trackedActorUID); err != nil { + t.Fatalf("forget: %v", err) + } + if _, err := os.Stat(ateletpath.ActorSnapshotStateFile(trackedActorUID)); !os.IsNotExist(err) { + t.Errorf("state file after forget: stat = %v, want it deleted", err) + } + if got := tr.State(trackedActorUID); !proto.Equal(got, &ateletpb.ActorSnapshotState{}) { + t.Errorf("state after forget = %v, want empty", got) + } +} + +// ForgetLocal drops the local snapshot and keeps the durable one. With no +// local snapshot to drop, it writes nothing. +func TestTrackerForgetLocal(t *testing.T) { + useTempNodeDirs(t) + if err := os.MkdirAll(ateletpath.ActorPath(trackedActorUID), 0o700); err != nil { + t.Fatal(err) + } + tr := newTracker(t) + tr.ForgetLocal(trackedActorUID) + if _, err := os.Stat(ateletpath.ActorSnapshotStateFile(trackedActorUID)); !os.IsNotExist(err) { + t.Errorf("state file after forgetting nothing: stat = %v, want none written", err) + } + + t0 := time.Now() + tr.CommitDurable(trackedActorUID, testRef("s1", 1, t0)) + tr.CommitLocal(trackedActorUID, testRef("s2", 2, t0.Add(time.Minute))) + tr.ForgetLocal(trackedActorUID) + if st := newTracker(t).State(trackedActorUID); st.GetLatestLocal() != nil || uuidOf(st.GetLatestDurable()) != "s1" { + t.Errorf("state from the file = %v, want no local snapshot and s1 still durable", st) + } + if len(tr.actors) != 0 { + t.Errorf("in-memory actors = %d, want 0 after writing to file", len(tr.actors)) + } +} + +// CommitPromoted makes the promoted local snapshot the latest durable one, +// keeping the commit time recorded for its local copy, and drops the local +// one. A retried promote of the same snapshot keeps what was recorded. +func TestTrackerCommitPromoted(t *testing.T) { + useTempNodeDirs(t) + if err := os.MkdirAll(ateletpath.ActorPath(trackedActorUID), 0o700); err != nil { + t.Fatal(err) + } + tr := newTracker(t) + t0 := time.Now() + tr.CommitLocal(trackedActorUID, testRef("s1", 1, t0)) + + promoted := testRef("s1", 1, time.Time{}) + promoted.UploadTime = t0.Add(time.Minute) + st := tr.CommitPromoted(trackedActorUID, promoted) + if st.GetLatestLocal() != nil { + t.Errorf("latest_local = %v after the promote, want unset", st.GetLatestLocal()) + } + if got := st.GetLatestDurable(); uuidOf(got) != "s1" || !timeOf(got.GetLocalCommitTime()).Equal(t0) || !timeOf(got.GetDurableCompleteTime()).Equal(promoted.UploadTime) { + t.Errorf("latest_durable = %v, want s1 committed at %v and uploaded at %v", got, t0, promoted.UploadTime) + } + if got := newTracker(t).State(trackedActorUID); !proto.Equal(got, st) { + t.Errorf("State() on a fresh tracker = %v, want %v from the state file", got, st) + } + + retry := testRef("s1", 1, time.Time{}) + retry.UploadTime = t0.Add(time.Hour) + if got := tr.CommitPromoted(trackedActorUID, retry); !proto.Equal(got, st) { + t.Errorf("state after a retried promote = %v, want %v", got, st) + } + + // Promoting a snapshot other than the local one carries no commit time. + tr.CommitLocal(trackedActorUID, testRef("s2", 2, t0.Add(2*time.Hour))) + other := testRef("s3", 3, time.Time{}) + other.UploadTime = t0.Add(3 * time.Hour) + st = tr.CommitPromoted(trackedActorUID, other) + if got := st.GetLatestDurable(); uuidOf(got) != "s3" || got.GetLocalCommitTime() != nil || st.GetLatestLocal() != nil { + t.Errorf("state = %v, want s3 durable without a commit time, and no local snapshot", st) + } +} + +// When a save fails, the entry stays in memory and is reported instead of the +// state file, until a later save succeeds. +func TestTrackerKeepsUnsavedStateInMemory(t *testing.T) { + useTempNodeDirs(t) + stateFile := ateletpath.ActorSnapshotStateFile(trackedActorUID) + // A directory where the state file goes makes the write's rename fail. + if err := os.MkdirAll(stateFile, 0o700); err != nil { + t.Fatal(err) + } + tr := newTracker(t) + t0 := time.Now() + st := tr.CommitLocal(trackedActorUID, testRef("s1", 1, t0)) + if len(tr.actors) != 1 { + t.Fatalf("in-memory actors = %d after a failed save, want 1", len(tr.actors)) + } + if got := tr.State(trackedActorUID); !proto.Equal(got, st) { + t.Errorf("State() = %v, want %v from memory", got, st) + } + + if err := os.Remove(stateFile); err != nil { + t.Fatal(err) + } + st = tr.CommitLocal(trackedActorUID, testRef("s2", 2, t0.Add(time.Minute))) + if len(tr.actors) != 0 { + t.Errorf("in-memory actors = %d after a successful save, want 0", len(tr.actors)) + } + if got := newTracker(t).State(trackedActorUID); !proto.Equal(got, st) { + t.Errorf("State() on a fresh tracker = %v, want %v from the state file", got, st) + } +} + +// A save that finds the actor's directory gone, as after the actor was +// terminated on this node, writes nothing and drops the entry. +func TestTrackerDropsActorWithoutDirectory(t *testing.T) { + useTempNodeDirs(t) + tr := newTracker(t) + tr.CommitLocal(trackedActorUID, testRef("s1", 1, time.Now())) + if len(tr.actors) != 0 { + t.Errorf("in-memory actors = %d, want 0 for an actor whose directory is gone", len(tr.actors)) + } + if _, err := os.Stat(ateletpath.ActorSnapshotStateFile(trackedActorUID)); !os.IsNotExist(err) { + t.Errorf("state file: stat = %v, want none written", err) + } + if got := tr.State(trackedActorUID); !proto.Equal(got, &ateletpb.ActorSnapshotState{}) { + t.Errorf("State() = %v, want empty", got) + } +} + +// Commits for one actor take turns, so a racing deferred checkpoint never +// loses an immediate checkpoint's durable snapshot. The control plane never +// sends one actor's checkpoints concurrently: the race only exercises the +// lock, so either commit may land last, even leaving a local snapshot older +// than the durable one. +func TestTrackerSerializesEachActor(t *testing.T) { + useTempNodeDirs(t) + if err := os.MkdirAll(ateletpath.ActorPath(trackedActorUID), 0o700); err != nil { + t.Fatal(err) + } + tr := newTracker(t) + t0 := time.Now() + for i := range 20 { + local := testRef(fmt.Sprintf("local-%d", i), int32(2*i+1), t0) + durable := testRef(fmt.Sprintf("durable-%d", i), int32(2*i+2), t0) + var wg sync.WaitGroup + wg.Go(func() { tr.CommitLocal(trackedActorUID, local) }) + wg.Go(func() { tr.CommitDurable(trackedActorUID, durable) }) + wg.Wait() + if got, want := uuidOf(tr.State(trackedActorUID).GetLatestDurable()), durable.Snapshot.GetSnapshotUuid(); got != want { + t.Fatalf("round %d: latest_durable = %q, want %q", i, got, want) + } + } + if len(tr.actors) != 0 { + t.Errorf("in-memory actors = %d, want 0 once every commit is written", len(tr.actors)) + } +} + +// A caller holding one actor's lock does not hold up another actor's commit. +func TestTrackerLocksEachActorSeparately(t *testing.T) { + useTempNodeDirs(t) + tr := newTracker(t) + tr.lock("other-actor-uid") + defer tr.unlock("other-actor-uid") + + done := make(chan struct{}) + go func() { + defer close(done) + tr.CommitLocal(trackedActorUID, testRef("s1", 1, time.Now())) + }() + select { + case <-done: + case <-time.After(time.Minute): + t.Fatal("CommitLocal waited for another actor's lock") + } +} + +// An unreadable state file on disk is skipped and treated as empty state. +func TestTrackerSkipsUnreadableSnapshotState(t *testing.T) { + useTempNodeDirs(t) + if err := os.MkdirAll(ateletpath.ActorPath(trackedActorUID), 0o700); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(ateletpath.ActorSnapshotStateFile(trackedActorUID), []byte("{"), 0o600); err != nil { + t.Fatal(err) + } + tr := newTracker(t) + if got := tr.State(trackedActorUID); !proto.Equal(got, &ateletpb.ActorSnapshotState{}) { + t.Errorf("state with a broken file = %v, want empty", got) + } +} diff --git a/cmd/atelet/lifecycle_test.go b/cmd/atelet/lifecycle_test.go index 1d12c0cf57..12ce8befb0 100644 --- a/cmd/atelet/lifecycle_test.go +++ b/cmd/atelet/lifecycle_test.go @@ -23,19 +23,22 @@ import ( "os" "path/filepath" "runtime" + "slices" "strings" "testing" "github.com/agent-substrate/substrate/cmd/atelet/internal/ateletpath" + "github.com/agent-substrate/substrate/cmd/atelet/internal/snapshottracker" "github.com/agent-substrate/substrate/cmd/atelet/internal/trustbundle" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/proto/ateletpb" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "google.golang.org/grpc" "google.golang.org/grpc/codes" "google.golang.org/protobuf/proto" + "k8s.io/utils/clock" ) // useTempNodeDirs roots atelet's on-node state in temp directories so a test @@ -52,12 +55,12 @@ func useTempNodeDirs(t *testing.T) { }) } -// fakeAteom is a fake ateom in a worker pod. It writes the files a +// fakeWorker is a fake worker in a worker pod. It writes the files a // real checkpoint would leave in the checkpoint dir, and reads back what a -// restore was handed. Like a real ateom it takes every actor directory from +// restore was handed. Like a real worker it takes every actor directory from // the request, never derived from the actor UID. -type fakeAteom struct { - ateompb.UnimplementedAteomServer +type fakeWorker struct { + ateworkerpb.UnimplementedWorkerServer // snapshotFiles are written at checkpoint and reported back to atelet as // the exact set the snapshot consists of. snapshotFiles map[string]string @@ -65,35 +68,35 @@ type fakeAteom struct { // most recent RestoreWorkload. restored map[string]string // actorDirs records the ActorDirs each RPC arrived with, by RPC name. - actorDirs map[string]*ateompb.ActorDirs + actorDirs map[string]*ateworkerpb.ActorDirs // preserveRestoreDir records the PreserveRestoreDir flag from the most // recent RestoreWorkload request. preserveRestoreDir bool // fidelities records the fidelity each checkpoint or restore arrived // with, by RPC name. - fidelities map[string]ateompb.SnapshotFidelity + fidelities map[string]ateworkerpb.SnapshotFidelity } -func (f *fakeAteom) recordFidelity(rpc string, fidelity ateompb.SnapshotFidelity) { +func (f *fakeWorker) recordFidelity(rpc string, fidelity ateworkerpb.SnapshotFidelity) { if f.fidelities == nil { - f.fidelities = map[string]ateompb.SnapshotFidelity{} + f.fidelities = map[string]ateworkerpb.SnapshotFidelity{} } f.fidelities[rpc] = fidelity } -func (f *fakeAteom) recordActorDirs(rpc string, actorDirs *ateompb.ActorDirs) { +func (f *fakeWorker) recordActorDirs(rpc string, actorDirs *ateworkerpb.ActorDirs) { if f.actorDirs == nil { - f.actorDirs = map[string]*ateompb.ActorDirs{} + f.actorDirs = map[string]*ateworkerpb.ActorDirs{} } f.actorDirs[rpc] = actorDirs } -func (f *fakeAteom) RunWorkload(_ context.Context, req *ateompb.RunWorkloadRequest) (*ateompb.RunWorkloadResponse, error) { +func (f *fakeWorker) RunWorkload(_ context.Context, req *ateworkerpb.RunWorkloadRequest) (*ateworkerpb.RunWorkloadResponse, error) { f.recordActorDirs("RunWorkload", req.GetActorDirs()) - return &ateompb.RunWorkloadResponse{}, nil + return &ateworkerpb.RunWorkloadResponse{}, nil } -func (f *fakeAteom) CheckpointWorkload(_ context.Context, req *ateompb.CheckpointWorkloadRequest) (*ateompb.CheckpointWorkloadResponse, error) { +func (f *fakeWorker) CheckpointWorkload(_ context.Context, req *ateworkerpb.CheckpointWorkloadRequest) (*ateworkerpb.CheckpointWorkloadResponse, error) { f.recordActorDirs("CheckpointWorkload", req.GetActorDirs()) f.recordFidelity("CheckpointWorkload", req.GetFidelity()) dir := req.GetActorDirs().GetCheckpointDir() @@ -104,10 +107,10 @@ func (f *fakeAteom) CheckpointWorkload(_ context.Context, req *ateompb.Checkpoin } names = append(names, name) } - return &ateompb.CheckpointWorkloadResponse{SnapshotFiles: names}, nil + return &ateworkerpb.CheckpointWorkloadResponse{SnapshotFiles: names}, nil } -func (f *fakeAteom) RestoreWorkload(_ context.Context, req *ateompb.RestoreWorkloadRequest) (*ateompb.RestoreWorkloadResponse, error) { +func (f *fakeWorker) RestoreWorkload(_ context.Context, req *ateworkerpb.RestoreWorkloadRequest) (*ateworkerpb.RestoreWorkloadResponse, error) { f.recordActorDirs("RestoreWorkload", req.GetActorDirs()) f.recordFidelity("RestoreWorkload", req.GetFidelity()) f.preserveRestoreDir = req.GetPreserveRestoreDir() @@ -120,44 +123,56 @@ func (f *fakeAteom) RestoreWorkload(_ context.Context, req *ateompb.RestoreWorkl } f.restored[name] = string(body) } - return &ateompb.RestoreWorkloadResponse{}, nil + return &ateworkerpb.RestoreWorkloadResponse{}, nil } -func (f *fakeAteom) TerminateWorkload(_ context.Context, req *ateompb.TerminateWorkloadRequest) (*ateompb.TerminateWorkloadResponse, error) { +func (f *fakeWorker) TerminateWorkload(_ context.Context, req *ateworkerpb.TerminateWorkloadRequest) (*ateworkerpb.TerminateWorkloadResponse, error) { f.recordActorDirs("TerminateWorkload", req.GetActorDirs()) - return &ateompb.TerminateWorkloadResponse{}, nil + return &ateworkerpb.TerminateWorkloadResponse{}, nil } -// serveFakeAteom serves ateom on a unix socket and points atelet's dialer at +// serveFakeWorker serves worker on a unix socket and points atelet's dialer at // it. The socket lives in its own short temp dir. -func serveFakeAteom(t *testing.T, f *fakeAteom) { +func serveFakeWorker(t *testing.T, f *fakeWorker) { t.Helper() - dir, err := os.MkdirTemp("", "ateom-") + dir, err := os.MkdirTemp("", "worker-") if err != nil { t.Fatalf("creating socket dir: %v", err) } t.Cleanup(func() { os.RemoveAll(dir) }) - sock := filepath.Join(dir, "ateom.sock") + sock := filepath.Join(dir, "worker.sock") lis, err := net.Listen("unix", sock) if err != nil { t.Fatalf("listening on %q: %v", sock, err) } srv := grpc.NewServer() - ateompb.RegisterAteomServer(srv, f) + ateworkerpb.RegisterWorkerServer(srv, f) go func() { _ = srv.Serve(lis) }() t.Cleanup(srv.Stop) - orig := ateomSocketPath - ateomSocketPath = func(string) string { return sock } - t.Cleanup(func() { ateomSocketPath = orig }) + orig := workerSocketPath + workerSocketPath = func(string) string { return sock } + t.Cleanup(func() { workerSocketPath = orig }) +} + +// lifecycleSnapshot is the Snapshot the control plane names for an actor's +// checkpoint at generation. +func lifecycleSnapshot(actorUID, name string, generation int32, fidelity ateletpb.SnapshotFidelity) *ateletpb.Snapshot { + return &ateletpb.Snapshot{ + SnapshotUuid: name, + Fidelity: fidelity, + Object: &ateletpb.ObjectStorage{SnapshotUri: "gs://test-bucket/atespaces/ate-demo/actors/" + actorUID + "/snapshots/" + name}, + Generation: generation, + } } // TestLocalSnapshotGC walks an actor through -// run -> pause -> resume -> terminate over atelet's RPC surface and ensures that -// the local snapshot is garbage collected after the actor is terminated. It -// runs at every fidelity atelet forwards, pinning that the pause and the -// resume hand ateom the fidelity the control plane asked for, unchanged. +// run -> pause -> resume -> pause -> terminate over atelet's RPC surface and +// ensures that superseded local snapshots are pruned by the next checkpoint +// and all remaining state is garbage collected after the actor is terminated. +// It runs at every fidelity atelet forwards, pinning that the pause and the +// resume hand worker the fidelity the control plane asked for, unchanged. func TestLocalSnapshotGC(t *testing.T) { for _, fidelity := range []ateletpb.SnapshotFidelity{ ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, @@ -173,15 +188,16 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { ctx := t.Context() const ( - atespace = "ate-demo" - actorName = "counter" - actorUID = "01234567-89ab-cdef-0123-456789abcdef" - workerPodUID = "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f" - snapshotName = "pause-snap-1" + atespace = "ate-demo" + actorName = "counter" + actorUID = "01234567-89ab-cdef-0123-456789abcdef" + workerPodUID = "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f" + snapshotName = "pause-snap-1" + snapshotName2 = "pause-snap-2" ) - ateom := &fakeAteom{snapshotFiles: map[string]string{"checkpoint.img": "guest-memory"}} - serveFakeAteom(t, ateom) + worker := &fakeWorker{snapshotFiles: map[string]string{"checkpoint.img": "guest-memory"}} + serveFakeWorker(t, worker) host := imageVolumeTestRegistry(t) image := host + "/actor:v1" @@ -190,11 +206,12 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { // A single "runsc" asset served from a fake bucket: enough to exercise the // content-addressed asset fetch without a gVisor release tarball. runsc := []byte("runsc binary") - s := &AteomHerder{ - ateomDialer: newAteomDialer(1), + s := &Atelet{ + workerDialer: newWorkerDialer(1), imageCache: newImageVolumeStore(t), anonGCSClient: fakeObjectStorage{data: runsc}, systemInfoVolumes: newSystemInfoVolumeRefresher(nil, nil), + snapshots: snapshottracker.New(t.Context(), clock.RealClock{}), } sandboxAssets := &ateletpb.SandboxAssets{ SandboxClass: "gvisor", @@ -225,8 +242,8 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { t.Fatalf("Run: %v", err) } - // Pause: a local checkpoint, which leaves the snapshot on this node. - if _, err := s.Checkpoint(ctx, &ateletpb.CheckpointRequest{ + // Pause: a deferred checkpoint, which leaves the snapshot on this node. + resp, err := s.Checkpoint(ctx, &ateletpb.CheckpointRequest{ Atespace: atespace, ActorName: actorName, ActorUid: actorUID, @@ -234,18 +251,23 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { ActorTemplateName: "counter", WorkerPodUid: workerPodUID, Spec: spec, - Snapshot: &ateletpb.Snapshot{ - SnapshotUuid: snapshotName, - Fidelity: fidelity, - }, - UploadConfig: &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED}, - }); err != nil { + Snapshot: lifecycleSnapshot(actorUID, snapshotName, 1, fidelity), + UploadConfig: deferredUploadConfig(), + }) + if err != nil { t.Fatalf("Checkpoint: %v", err) } snapshotFile := filepath.Join(ateletpath.LocalSnapshotDir(actorUID, snapshotName), "checkpoint.img") if _, err := os.Stat(snapshotFile); err != nil { t.Fatalf("pause did not write the local snapshot: %v", err) } + st := resp.GetState() + if got := st.GetLatestLocal(); got.GetSnapshot().GetSnapshotUuid() != snapshotName || got.GetLocalCommitTime() == nil { + t.Errorf("latest_local = %v, want the pause snapshot with its commit time", got) + } + if st.GetLatestDurable() != nil { + t.Errorf("state = %v, want nothing durable", st) + } // Resume: no Snapshot.Object; atelet must find that local snapshot. if _, err := s.Restore(ctx, &ateletpb.RestoreRequest{ @@ -259,27 +281,50 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { Spec: spec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: snapshotName, + Generation: 1, Fidelity: fidelity, }, }); err != nil { t.Fatalf("Restore: %v", err) } - if got := ateom.restored["checkpoint.img"]; got != "guest-memory" { - t.Fatalf("restore staged %q for ateom, want the pause snapshot's %q", got, "guest-memory") + if got := worker.restored["checkpoint.img"]; got != "guest-memory" { + t.Fatalf("restore staged %q for worker, want the pause snapshot's %q", got, "guest-memory") } for _, rpc := range []string{"CheckpointWorkload", "RestoreWorkload"} { - if got := ateom.fidelities[rpc]; got != toAteomFidelity(fidelity) { - t.Errorf("%s carried fidelity %v, want %v", rpc, got, toAteomFidelity(fidelity)) + if got := worker.fidelities[rpc]; got != toWorkerFidelity(fidelity) { + t.Errorf("%s carried fidelity %v, want %v", rpc, got, toWorkerFidelity(fidelity)) } } - if !ateom.preserveRestoreDir { + if !worker.preserveRestoreDir { t.Errorf("RestoreWorkload preserve_restore_dir = false, want true for pure-local restore") } if entries, err := os.ReadDir(ateletpath.RestoreStateDir(actorUID)); err != nil || len(entries) != 0 { t.Fatalf("expected RestoreStateDir to remain empty on local pause restore, got entries=%v err=%v", entries, err) } - // Terminate: the actor is gone, and so should its snapshot be. + // Second pause: must replace, not accumulate alongside, the first snapshot. + if _, err := s.Checkpoint(ctx, &ateletpb.CheckpointRequest{ + Atespace: atespace, + ActorName: actorName, + ActorUid: actorUID, + ActorTemplateAtespace: "default", + ActorTemplateName: "counter", + WorkerPodUid: workerPodUID, + Spec: spec, + Snapshot: lifecycleSnapshot(actorUID, snapshotName2, 2, fidelity), + UploadConfig: deferredUploadConfig(), + }); err != nil { + t.Fatalf("Checkpoint 2: %v", err) + } + snapshotFile2 := filepath.Join(ateletpath.LocalSnapshotDir(actorUID, snapshotName2), "checkpoint.img") + if _, err := os.Stat(ateletpath.LocalSnapshotDir(actorUID, snapshotName)); !os.IsNotExist(err) { + t.Fatalf("first local snapshot survived second checkpoint: %v", err) + } + if _, err := os.Stat(snapshotFile2); err != nil { + t.Fatalf("second pause did not write the local snapshot: %v", err) + } + + // Terminate: the actor is gone, and so should its remaining snapshot be. if _, err := s.Terminate(ctx, &ateletpb.TerminateRequest{ Atespace: atespace, ActorName: actorName, @@ -292,7 +337,7 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { t.Fatalf("Terminate: %v", err) } - // Every RPC hands ateom the same directory set, except that a local + // Every RPC hands worker the same directory set, except that a local // restore points restore_dir at the pause snapshot; the fake already // relied on checkpoint_dir and restore_dir above to place and find it. for _, rpc := range []string{"RunWorkload", "CheckpointWorkload", "RestoreWorkload", "TerminateWorkload"} { @@ -300,7 +345,7 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { if rpc == "RestoreWorkload" { want.RestoreDir = ateletpath.LocalSnapshotDir(actorUID, snapshotName) } - if got := ateom.actorDirs[rpc]; !proto.Equal(got, want) { + if got := worker.actorDirs[rpc]; !proto.Equal(got, want) { t.Errorf("%s carried actor actorDirs %v, want %v", rpc, got, want) } } @@ -310,6 +355,9 @@ func runLocalSnapshotGC(t *testing.T, fidelity ateletpb.SnapshotFidelity) { leaked, _ := filepath.Glob(filepath.Join(localDir, "*", "*")) t.Errorf("local checkpoint dir survived terminate (stat err = %v), leaked files: %v", err, leaked) } + if got := s.snapshots.State(actorUID); !proto.Equal(got, &ateletpb.ActorSnapshotState{}) { + t.Errorf("snapshot tracker state after terminate = %v, want empty", got) + } // Terminate is the only chance to reclaim the actor's directory: nothing // else on the node deletes it. @@ -340,8 +388,8 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { snapshotName = "pause-snap-1" ) - ateom := &fakeAteom{snapshotFiles: map[string]string{"checkpoint.img": "guest-memory"}} - serveFakeAteom(t, ateom) + worker := &fakeWorker{snapshotFiles: map[string]string{"checkpoint.img": "guest-memory"}} + serveFakeWorker(t, worker) host := imageVolumeTestRegistry(t) image := host + "/actor:v1" @@ -350,11 +398,12 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { restorePause := pushTestImage(t, host+"/pause:v2", singleFileLayer(t, "pause", "pause-v2")) runsc := []byte("runsc binary") - s := &AteomHerder{ - ateomDialer: newAteomDialer(1), + s := &Atelet{ + workerDialer: newWorkerDialer(1), imageCache: newImageVolumeStore(t), anonGCSClient: fakeObjectStorage{data: runsc}, systemInfoVolumes: newSystemInfoVolumeRefresher(nil, nil), + snapshots: snapshottracker.New(t.Context(), clock.RealClock{}), } assetsWithPause := func(pause string) *ateletpb.SandboxAssets { return &ateletpb.SandboxAssets{ @@ -397,6 +446,7 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { Spec: spec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: snapshotName, + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, UploadConfig: &ateletpb.UploadConfig{Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED}, @@ -427,6 +477,7 @@ func TestRestoreUsesRequestSandboxAssets(t *testing.T) { Spec: spec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: snapshotName, + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }); err != nil { @@ -509,7 +560,7 @@ func TestActivationFailureBeforeRegistration(t *testing.T) { if tc.failure == "asset" { storage.err = fmt.Errorf("asset unavailable") } - s := &AteomHerder{anonGCSClient: storage, systemInfoVolumes: refresher} + s := &Atelet{anonGCSClient: storage, systemInfoVolumes: refresher, snapshots: snapshottracker.New(t.Context(), clock.RealClock{})} assets := &ateletpb.SandboxAssets{ SandboxClass: "gvisor", PauseImage: testPauseImage, Assets: map[string]*ateletpb.ArchAssets{runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{ @@ -522,6 +573,7 @@ func TestActivationFailureBeforeRegistration(t *testing.T) { SandboxAssets: assets, Spec: spec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: snapshotName, + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }) @@ -579,10 +631,11 @@ func TestRunFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { content := []byte("runsc binary") assetHash := fmt.Sprintf("%x", sha256.Sum256(content)) const actorUID = "0f6c2b0e-1b7e-4b3f-9a6a-1d2c3e4f5a62" - s := &AteomHerder{ + s := &Atelet{ anonGCSClient: fakeObjectStorage{data: content}, imageCache: newImageVolumeStore(t), systemInfoVolumes: refresher, + snapshots: snapshottracker.New(t.Context(), clock.RealClock{}), } spec := &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{{ @@ -646,10 +699,11 @@ func TestRestoreFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { Name: "trust", SystemInfo: trustVolumeSpec("ca.pem"), }}} - s := &AteomHerder{ + s := &Atelet{ anonGCSClient: fakeObjectStorage{data: content}, imageCache: newImageVolumeStore(t), systemInfoVolumes: refresher, + snapshots: snapshottracker.New(t.Context(), clock.RealClock{}), } _, err := s.Restore(t.Context(), &ateletpb.RestoreRequest{ Atespace: "team-a", @@ -666,6 +720,7 @@ func TestRestoreFailureAfterRegistrationRemovesOwnRegistration(t *testing.T) { Spec: spec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: snapshotName, + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, }, }) @@ -694,8 +749,8 @@ func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) { snapshotName = "pause-snap-1" ) - ateom := &fakeAteom{snapshotFiles: map[string]string{"checkpoint.img": ""}} - serveFakeAteom(t, ateom) + worker := &fakeWorker{snapshotFiles: map[string]string{"checkpoint.img": ""}} + serveFakeWorker(t, worker) host := imageVolumeTestRegistry(t) pinnedImage := pushTestImage(t, host+"/actor:v1", singleFileLayer(t, "bin/app", "app")) @@ -728,7 +783,7 @@ func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) { // with snapshot.object pointing at it. objStore := &recordingObjectStorage{} s := newPluginHerder(t, objStore) - s.ateomDialer = newAteomDialer(1) + s.workerDialer = newWorkerDialer(1) s.imageCache = newImageVolumeStore(t) s.anonGCSClient = fakeObjectStorage{data: runsc} s.systemInfoVolumes = newSystemInfoVolumeRefresher(nil, nil) @@ -804,6 +859,7 @@ func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) { Spec: spec, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: snapshotName, + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, Object: &ateletpb.ObjectStorage{SnapshotUri: pausedSnapshotURI}, }, @@ -817,13 +873,13 @@ func TestRestorePrefersLocalAndFallsBackToObjectStorage(t *testing.T) { wantContent = "local-memory" wantRestoreDir = localDir } - if got := ateom.restored["checkpoint.img"]; got != wantContent { + if got := worker.restored["checkpoint.img"]; got != wantContent { t.Errorf("restored checkpoint.img = %q, want %q", got, wantContent) } - if ateom.preserveRestoreDir != tc.wantLocal { - t.Errorf("RestoreWorkload preserve_restore_dir = %v, want %v", ateom.preserveRestoreDir, tc.wantLocal) + if worker.preserveRestoreDir != tc.wantLocal { + t.Errorf("RestoreWorkload preserve_restore_dir = %v, want %v", worker.preserveRestoreDir, tc.wantLocal) } - if got := ateom.actorDirs["RestoreWorkload"].GetRestoreDir(); got != wantRestoreDir { + if got := worker.actorDirs["RestoreWorkload"].GetRestoreDir(); got != wantRestoreDir { t.Errorf("RestoreWorkload restore_dir = %q, want %q", got, wantRestoreDir) } if tc.wantLocal { @@ -891,6 +947,7 @@ func TestRestoreWithoutUsableLocalCopy(t *testing.T) { Spec: &ateletpb.WorkloadSpec{}, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: snapshotName, + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, Object: tc.object, }, @@ -910,7 +967,7 @@ func TestRestoreWithoutUsableLocalCopy(t *testing.T) { // TestTerminateWithoutWorkerPodUID verifies that Terminate with an empty // WorkerPodUid (used when cleaning up the assigned node for a paused or -// crashed actor that no longer has a worker pod) skips dialing ateom while +// crashed actor that no longer has a worker pod) skips dialing worker while // still pruning local checkpoints and removing actor directories on the node. func TestTerminateWithoutWorkerPodUID(t *testing.T) { useTempNodeDirs(t) @@ -935,12 +992,13 @@ func TestTerminateWithoutWorkerPodUID(t *testing.T) { t.Fatalf("writing local snapshot file: %v", err) } - ateom := &fakeAteom{} - serveFakeAteom(t, ateom) + worker := &fakeWorker{} + serveFakeWorker(t, worker) - s := &AteomHerder{ - ateomDialer: newAteomDialer(1), + s := &Atelet{ + workerDialer: newWorkerDialer(1), systemInfoVolumes: newSystemInfoVolumeRefresher(nil, nil), + snapshots: snapshottracker.New(t.Context(), clock.RealClock{}), } spec := &ateletpb.WorkloadSpec{ Containers: []*ateletpb.Container{{Name: "app", Image: "example.com/app@sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}, @@ -958,8 +1016,8 @@ func TestTerminateWithoutWorkerPodUID(t *testing.T) { t.Fatalf("Terminate: %v", err) } - if got := ateom.actorDirs["TerminateWorkload"]; got != nil { - t.Errorf("TerminateWorkload was called on ateom (%v), want skipped when WorkerPodUid is empty", got) + if got := worker.actorDirs["TerminateWorkload"]; got != nil { + t.Errorf("TerminateWorkload was called on worker (%v), want skipped when WorkerPodUid is empty", got) } if _, err := os.Stat(ateletpath.LocalCheckpointsDir(actorUID)); !os.IsNotExist(err) { t.Errorf("local checkpoint dir survived terminate: %v", err) @@ -968,3 +1026,210 @@ func TestTerminateWithoutWorkerPodUID(t *testing.T) { t.Errorf("actor dir survived terminate: %v", err) } } + +// pluginActorUID is the actor runPluginActor runs. +const ( + pluginActorUID = "01234567-89ab-cdef-0123-456789abcdef" + pluginWorkerPodUID = "5f1c0d2e-3a4b-4c5d-8e9f-0a1b2c3d4e5f" +) + +// pluginActor is an actor running on a fake worker, on a herder whose snapshot +// plugin stores into store. +type pluginActor struct { + s *Atelet + worker *fakeWorker + store *recordingObjectStorage + assets *ateletpb.SandboxAssets + spec *ateletpb.WorkloadSpec +} + +func runPluginActor(t *testing.T) *pluginActor { + t.Helper() + useTempNodeDirs(t) + + a := &pluginActor{ + worker: &fakeWorker{snapshotFiles: map[string]string{"checkpoint.img": "guest-memory"}}, + store: &recordingObjectStorage{}, + } + serveFakeWorker(t, a.worker) + + host := imageVolumeTestRegistry(t) + image := host + "/actor:v1" + pinnedImage := pushTestImage(t, image, singleFileLayer(t, "bin/app", "app")) + + runsc := []byte("runsc binary") + a.s = newPluginHerder(t, a.store) + a.s.workerDialer = newWorkerDialer(1) + a.s.imageCache = newImageVolumeStore(t) + a.s.anonGCSClient = fakeObjectStorage{data: runsc} + a.s.systemInfoVolumes = newSystemInfoVolumeRefresher(nil, nil) + a.assets = &ateletpb.SandboxAssets{ + SandboxClass: "gvisor", + PauseImage: pinnedImage, + Assets: map[string]*ateletpb.ArchAssets{ + runtime.GOARCH: {Files: map[string]*ateletpb.AssetFile{ + runscAssetName: { + Url: "gs://test-bucket/runsc", + Sha256: fmt.Sprintf("%x", sha256.Sum256(runsc)), + }, + }}, + }, + } + a.spec = &ateletpb.WorkloadSpec{ + Containers: []*ateletpb.Container{{Name: "app", Image: pinnedImage, Command: []string{"/bin/app"}}}, + } + + if _, err := a.s.Run(t.Context(), &ateletpb.RunRequest{ + Atespace: "ate-demo", + ActorName: "counter", + ActorUid: pluginActorUID, + ActorTemplateAtespace: "default", + ActorTemplateName: "counter", + WorkerPodUid: pluginWorkerPodUID, + SandboxAssets: a.assets, + Spec: a.spec, + }); err != nil { + t.Fatalf("Run: %v", err) + } + return a +} + +// snapshot is the Snapshot the control plane names for the actor's checkpoint +// at generation. +func (a *pluginActor) snapshot(name string, generation int32) *ateletpb.Snapshot { + return lifecycleSnapshot(pluginActorUID, name, generation, ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY) +} + +func (a *pluginActor) checkpoint(t *testing.T, snap *ateletpb.Snapshot, mode ateletpb.UploadMode) (*ateletpb.CheckpointResponse, error) { + t.Helper() + return a.s.Checkpoint(t.Context(), &ateletpb.CheckpointRequest{ + Atespace: "ate-demo", + ActorName: "counter", + ActorUid: pluginActorUID, + ActorTemplateAtespace: "default", + ActorTemplateName: "counter", + WorkerPodUid: pluginWorkerPodUID, + Spec: a.spec, + Snapshot: snap, + UploadConfig: &ateletpb.UploadConfig{Mode: mode}, + }) +} + +// restoreLocal resumes the actor from its local snapshot named name, taken at +// generation. +func (a *pluginActor) restoreLocal(t *testing.T, name string, generation int32) { + t.Helper() + if _, err := a.s.Restore(t.Context(), &ateletpb.RestoreRequest{ + Atespace: "ate-demo", + ActorName: "counter", + ActorUid: pluginActorUID, + ActorTemplateAtespace: "default", + ActorTemplateName: "counter", + WorkerPodUid: pluginWorkerPodUID, + SandboxAssets: a.assets, + Spec: a.spec, + Snapshot: &ateletpb.Snapshot{SnapshotUuid: name, Generation: generation, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, + }); err != nil { + t.Fatalf("Restore: %v", err) + } +} + +// promote promotes snap and returns the actor's state the response reports. +func (a *pluginActor) promote(t *testing.T, snap *ateletpb.Snapshot) *ateletpb.ActorSnapshotState { + t.Helper() + resp, err := a.s.PromoteSnapshot(t.Context(), &ateletpb.PromoteSnapshotRequest{ + Atespace: "ate-demo", + ActorName: "counter", + ActorUid: pluginActorUID, + ActorTemplateAtespace: "default", + ActorTemplateName: "counter", + Snapshot: snap, + }) + if err != nil { + t.Fatalf("PromoteSnapshot: %v", err) + } + return resp.GetState() +} + +// manifestKey is where the object store keeps the manifest of the actor's +// snapshot named name. +func manifestKey(name string) string { + return "test-bucket/atespaces/ate-demo/actors/" + pluginActorUID + "/snapshots/" + name + "/" + sandboxManifestName +} + +// TestImmediateCheckpointKeepsNoLocalCopy suspends a running actor: the +// checkpoint uploads the snapshot before it returns and leaves none on the +// node. +func TestImmediateCheckpointKeepsNoLocalCopy(t *testing.T) { + a := runPluginActor(t) + snap := a.snapshot("suspend-snap-1", 1) + + resp, err := a.checkpoint(t, snap, ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE) + if err != nil { + t.Fatalf("Checkpoint: %v", err) + } + if !slices.Contains(a.store.keys(), manifestKey("suspend-snap-1")) { + t.Errorf("objects = %v, want the snapshot uploaded", a.store.keys()) + } + if _, err := os.Stat(ateletpath.LocalCheckpointsDir(pluginActorUID)); !os.IsNotExist(err) { + t.Errorf("local checkpoints after an immediate checkpoint: stat = %v, want none", err) + } + st := resp.GetState() + if st.GetLatestLocal() != nil { + t.Errorf("state = %v, want no local snapshot", st) + } + if got := st.GetLatestDurable(); !proto.Equal(got.GetSnapshot(), snap) || got.GetDurableCompleteTime() == nil { + t.Errorf("latest_durable = %v, want %v with its upload time", got, snap) + } +} + +// TestFailedCheckpointForgetsLocalSnapshot suspends a resumed actor whose +// upload fails after the checkpoint has deleted the pause snapshot: the +// tracker no longer reports the deleted snapshot. +func TestFailedCheckpointForgetsLocalSnapshot(t *testing.T) { + a := runPluginActor(t) + if _, err := a.checkpoint(t, a.snapshot("pause-snap-1", 1), ateletpb.UploadMode_UPLOAD_MODE_DEFERRED); err != nil { + t.Fatalf("Checkpoint (pause): %v", err) + } + a.restoreLocal(t, "pause-snap-1", 1) + + a.store.putErr = errors.New("object store unavailable") + if _, err := a.checkpoint(t, a.snapshot("suspend-snap-2", 2), ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE); err == nil { + t.Fatal("Checkpoint (suspend) succeeded, want the upload to fail") + } + if _, err := os.Stat(ateletpath.LocalSnapshotDir(pluginActorUID, "pause-snap-1")); !os.IsNotExist(err) { + t.Fatalf("pause snapshot after the failed suspend: stat = %v, want it deleted", err) + } + if st := a.s.snapshots.State(pluginActorUID); st.GetLatestLocal() != nil || st.GetLatestDurable() != nil { + t.Errorf("state = %v, want neither a local nor a durable snapshot", st) + } +} + +// TestPromoteSnapshotRecordsUpload suspends a paused actor: the promote +// records the uploaded snapshot as the latest durable one, with the pause's +// commit time, and the deleted local copy is no longer reported. A retry, +// which finds the snapshot already uploaded, reports the same state. +func TestPromoteSnapshotRecordsUpload(t *testing.T) { + a := runPluginActor(t) + snap := a.snapshot("pause-snap-1", 1) + resp, err := a.checkpoint(t, snap, ateletpb.UploadMode_UPLOAD_MODE_DEFERRED) + if err != nil { + t.Fatalf("Checkpoint: %v", err) + } + commitTime := resp.GetState().GetLatestLocal().GetLocalCommitTime() + + st := a.promote(t, snap) + if st.GetLatestLocal() != nil { + t.Errorf("latest_local = %v after the promote deleted it, want unset", st.GetLatestLocal()) + } + if got := st.GetLatestDurable(); !proto.Equal(got.GetSnapshot(), snap) || !proto.Equal(got.GetLocalCommitTime(), commitTime) || got.GetDurableCompleteTime() == nil { + t.Errorf("latest_durable = %v, want %v committed at %v, with its upload time", got, snap, commitTime) + } + if got := a.s.snapshots.State(pluginActorUID); !proto.Equal(got, st) { + t.Errorf("tracked state = %v, want %v, as PromoteSnapshot returned", got, st) + } + + if got := a.promote(t, snap); !proto.Equal(got, st) { + t.Errorf("state after a retried promote = %v, want %v", got, st) + } +} diff --git a/cmd/atelet/main.go b/cmd/atelet/main.go index c5e9e74670..489d952a9a 100644 --- a/cmd/atelet/main.go +++ b/cmd/atelet/main.go @@ -33,6 +33,7 @@ import ( "github.com/agent-substrate/substrate/cmd/atelet/internal/apivalidation" "github.com/agent-substrate/substrate/cmd/atelet/internal/ateletpath" "github.com/agent-substrate/substrate/cmd/atelet/internal/credentialprovider" + "github.com/agent-substrate/substrate/cmd/atelet/internal/snapshottracker" "github.com/agent-substrate/substrate/cmd/atelet/internal/sparsefile" "github.com/agent-substrate/substrate/cmd/atelet/internal/trustbundle" "github.com/agent-substrate/substrate/internal/apierror" @@ -47,8 +48,9 @@ import ( "github.com/agent-substrate/substrate/internal/ocispec" "github.com/agent-substrate/substrate/internal/otlprelay" "github.com/agent-substrate/substrate/internal/pluginsocket" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/proto/ateletpb" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/substratex509" @@ -82,6 +84,7 @@ import ( "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" "k8s.io/client-go/tools/cache" + "k8s.io/utils/clock" "k8s.io/utils/lru" ) @@ -105,12 +108,13 @@ var ( imageCredentialProviderBinDir = pflag.String("image-credential-provider-bin-dir", "", "Directory holding the credential provider executables named by --image-credential-provider-config. Required when that flag is set.") localhostRegistryReplacement = pflag.String("localhost-registry-replacement", "", "The replacement registry endpoint for localhost and/or loopback IP addresses, useful for local development. for example kind-registry:5000") - imageCacheDir = pflag.String("image-cache-dir", ateletpath.ImageCacheDir, "Directory for the node-local OCI image layer cache. Must be on the volume shared with the ateom pods (the cached layers are their overlay lowerdirs), and on a disk sized for both capacity and IOPS: unpack throughput is gated by the volume's IOPS.") + imageCacheDir = pflag.String("image-cache-dir", ateletpath.ImageCacheDir, "Directory for the node-local OCI image layer cache. Must be on the volume shared with the worker pods (the cached layers are their overlay lowerdirs), and on a disk sized for both capacity and IOPS: unpack throughput is gated by the volume's IOPS.") showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") + previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.") - otlpRelaySocket = pflag.String("otlp-relay-socket", nodepath.AteletOTLPSocketPath(), "Unix socket to serve the OTLP relay on, which forwards the node's ateom telemetry to OTEL_EXPORTER_OTLP_ENDPOINT so worker pods need no network path to the collector. Empty disables the relay.") + otlpRelaySocket = pflag.String("otlp-relay-socket", nodepath.AteletOTLPSocketPath(), "Unix socket to serve the OTLP relay on, which forwards the node's worker telemetry to OTEL_EXPORTER_OTLP_ENDPOINT so worker pods need no network path to the collector. Empty disables the relay.") actorStatsPollInterval = pflag.Duration("actor-stats-poll-interval", time.Minute, fmt.Sprintf("Actor resource utilization sampling frequency. 0 disables the sampling entirely; minimum accepted value is %v.", minActorStatsPollInterval)) @@ -133,6 +137,9 @@ func main() { if err := serverboot.SetLogLevel(*logLevelFlag); err != nil { serverboot.Fatal(ctx, "Invalid --log-level", err) } + if err := preview.Init(*previewFlags...); err != nil { + serverboot.Fatal(ctx, "Invalid --preview", err) + } slog.InfoContext(ctx, "atelet starting", slog.String("version", version.Version)) if err := rejectStorageEnv(); err != nil { serverboot.Fatal(ctx, "Invalid environment", err) @@ -174,22 +181,22 @@ func main() { go serverboot.StartMetricsServer(ctx, *metricsListenAddr) go serverboot.StartHealthServer(ctx, *healthListenAddr, readiness) - // The OTLP relay lets the ateom pods on this node export telemetry over a + // The OTLP relay lets the worker pods on this node export telemetry over a // unix socket instead of their own network (see internal/otlprelay). Started - // early: an ateom that finds no socket at startup falls back to exporting + // early: a worker that finds no socket at startup falls back to exporting // directly for its whole life, so the socket should exist before any worker // pod on this node boots. if relay, err := otlprelay.NewServer(ctx, *otlpRelaySocket); err != nil { - slog.ErrorContext(ctx, "Failed to create the OTLP relay; ateoms will export directly", slog.Any("err", err)) + slog.ErrorContext(ctx, "Failed to create the OTLP relay; workers will export directly", slog.Any("err", err)) } else if relay != nil { // Deferred rather than tied to the drain: the relay carries other // processes' telemetry, so it should outlive atelet's own RPC serving - // and stay up while the ateoms it serves are themselves shutting down. + // and stay up while the workers it serves are themselves shutting down. defer relay.Stop() go func() { if err := relay.Serve(ctx); err != nil { // Not fatal: atelet's actual job does not depend on the relay, - // and the ateoms fall back to exporting directly. + // and the workers fall back to exporting directly. slog.ErrorContext(ctx, "OTLP relay stopped", slog.Any("err", err)) } }() @@ -197,7 +204,7 @@ func main() { startDevicePlugins(ctx) - ateomDialer := newAteomDialer(256) + workerDialer := newWorkerDialer(256) // Without a credential provider config, pulls are anonymous: all an // unconfigured node (kind, say) can do and all a public registry needs. @@ -298,9 +305,14 @@ func main() { serverboot.Fatal(ctx, "Object-store plugin is not serving", err) } + // The snapshot tracker's workers get a context of their own, cancelled only + // once the gRPC server has drained, so background snapshot work keeps + // running while in-flight RPCs finish. + trackerCtx, stopTracker := context.WithCancel(ctx) + snapshots := snapshottracker.New(trackerCtx, clock.RealClock{}) wmService := NewService( ctx, - ateomDialer, + workerDialer, wrappedAnonGCS, objectstorev1.NewNodeProviderClient(objectStorePluginConn), objectstorev1.NewAssetProviderClient(objectStorePluginConn), @@ -310,7 +322,9 @@ func main() { volPlugins, csiDriverConfigGetter, systemInfoVolumes, + snapshots, ) + snapshots.Start() go systemInfoVolumes.run(ctx) lis, err := net.Listen("tcp", ":"+strconv.Itoa(*port)) @@ -318,7 +332,7 @@ func main() { serverboot.Fatal(ctx, "Failed to listen", err) } - serverOpts := ateomHerderServerOptions(insecure.NewCredentials()) + serverOpts := ateletServerOptions(insecure.NewCredentials()) nodeName := os.Getenv("NODE_NAME") if *grpcInsecure { slog.WarnContext(ctx, "Serving atelet gRPC without transport security") @@ -332,6 +346,8 @@ func main() { if err != nil { serverboot.Fatal(ctx, "Failed to build ateapi client credentials", err) } + // atelet acts for the system, so it opts in to enabled preview features. + dialOpts = append(dialOpts, preview.DialOptionsWithOptIn()...) ateapiConn, err := grpc.NewClient(*ateapiAddress, dialOpts...) if err != nil { serverboot.Fatal(ctx, "Failed to create ateapi client", err) @@ -355,30 +371,30 @@ func main() { } nodeName = ateletIdentity.NodeName - serverOpts = ateomHerderServerOptions(credentials.NewTLS(tlsCfg)) + serverOpts = ateletServerOptions(credentials.NewTLS(tlsCfg)) - ateomFacingTLS := tlsCfg.Clone() - ateomFacingTLS.VerifyConnection = verifyClientOnSameNode(ateletIdentity) - if err := os.Remove(nodepath.AteomSupportSocket); err != nil && !errors.Is(err, os.ErrNotExist) { + workerFacingTLS := tlsCfg.Clone() + workerFacingTLS.VerifyConnection = verifyClientOnSameNode(ateletIdentity) + if err := os.Remove(nodepath.WorkerSupportSocket); err != nil && !errors.Is(err, os.ErrNotExist) { serverboot.Fatal(ctx, "Failed to remove stale credential broker socket", err) } - ateomFacingLis, err := net.Listen("unix", nodepath.AteomSupportSocket) + workerFacingLis, err := net.Listen("unix", nodepath.WorkerSupportSocket) if err != nil { serverboot.Fatal(ctx, "Failed to listen for credential broker", err) } - defer ateomFacingLis.Close() - if err := os.Chmod(nodepath.AteomSupportSocket, 0o600); err != nil { + defer workerFacingLis.Close() + if err := os.Chmod(nodepath.WorkerSupportSocket, 0o600); err != nil { serverboot.Fatal(ctx, "Failed to restrict credential broker socket", err) } - ateomFacingSrv := grpc.NewServer(grpc.Creds(credentials.NewTLS(ateomFacingTLS))) + workerFacingSrv := grpc.NewServer(grpc.Creds(credentials.NewTLS(workerFacingTLS))) - ateletpb.RegisterAteomSupportServer(ateomFacingSrv, &ateomSupportServer{ + ateletpb.RegisterWorkerSupportServer(workerFacingSrv, &workerSupportServer{ workers: ateapipb.NewWorkerServiceClient(ateapiConn), control: ateapipb.NewControlClient(ateapiConn), }) go func() { - if err := ateomFacingSrv.Serve(ateomFacingLis); err != nil { + if err := workerFacingSrv.Serve(workerFacingLis); err != nil { serverboot.Fatal(ctx, "Failed to serve credential broker", err) } }() @@ -423,7 +439,7 @@ func main() { workerPodFactory.Core().V1().Pods(), ateFactory.Api().V1alpha1().WorkerPools(), ateFactory.Api().V1alpha1().SandboxConfigs(), - nodepath.AteomsDir()); err != nil { + nodepath.WorkersDir()); err != nil { slog.ErrorContext(ctx, "Worker sandbox versions syncer disabled", slog.Any("err", err)) } // Start after the informers are registered: a factory only runs @@ -432,7 +448,7 @@ func main() { workerPodFactory.Start(stopCh) svr := grpc.NewServer(serverOpts...) - ateletpb.RegisterAteomHerderServer(svr, wmService) + ateletpb.RegisterAteletServer(svr, wmService) reflection.Register(svr) slog.InfoContext(ctx, "WorkersManagerService listening", slog.Any("address", lis.Addr())) @@ -441,13 +457,21 @@ func main() { serverboot.Fatal(ctx, "Failed to serve", err) } <-drainDone + stopTracker() + // Workers stop once their operations see the cancellation. The bound keeps + // one that does not from holding atelet past its telemetry flushes. + waitCtx, cancelWait := context.WithTimeout(ctx, 30*time.Second) + if err := snapshots.Wait(waitCtx); err != nil { + slog.WarnContext(ctx, "Background snapshot work did not stop in time; exiting anyway", slog.Any("err", err)) + } + cancelWait() slog.InfoContext(ctx, "Shutdown complete") } -// ateomHerderServerOptions are the options of the gRPC server that serves -// AteomHerder to ate-api-server. Its interceptor logs every request; fields +// ateletServerOptions are the options of the gRPC server that serves +// Atelet to ate-api-server. Its interceptor logs every request; fields // marked debug_redact (env values, asset URLs) are masked in that log. -func ateomHerderServerOptions(creds credentials.TransportCredentials) []grpc.ServerOption { +func ateletServerOptions(creds credentials.TransportCredentials) []grpc.ServerOption { return []grpc.ServerOption{ grpc.Creds(creds), grpc.StatsHandler(otelgrpc.NewServerHandler()), @@ -507,13 +531,13 @@ func (g *directCSIDriverConfigGetter) Get(name string) (*atev1alpha1.CSIDriverCo return g.client.ApiV1alpha1().CSIDriverConfigs().Get(context.Background(), name, metav1.GetOptions{}) } -// AteomHerder is a service that allows controlling workloads on individual -// ateoms. -type AteomHerder struct { - ateletpb.UnimplementedAteomHerderServer +// Atelet is a service that allows controlling workloads on individual +// workers. +type Atelet struct { + ateletpb.UnimplementedAteletServer - ateomDialer *AteomDialer - imageCache *imagecache.Store + workerDialer *WorkerDialer + imageCache *imagecache.Store // anonGCSClient reads gs:// assets the plugin cannot serve, without credentials. anonGCSClient objectstorage.ObjectStorage // objectStorePlugin moves external snapshot files between the node and storage. @@ -531,14 +555,17 @@ type AteomHerder struct { volumePlugins map[string]volume.VolumePluginWorkerPlane csiDriverConfigGetter csi.CSIDriverConfigGetter systemInfoVolumes *systemInfoVolumeRefresher + + // snapshots tracks each actor's snapshots on this node. + snapshots *snapshottracker.Tracker } -var _ ateletpb.AteomHerderServer = (*AteomHerder)(nil) +var _ ateletpb.AteletServer = (*Atelet)(nil) // NewService creates a new WorkersManagerService. func NewService( ctx context.Context, - ateomDialer *AteomDialer, + workerDialer *WorkerDialer, anonGCSClient objectstorage.ObjectStorage, objectStorePlugin objectstorev1.NodeProviderClient, assetPlugin objectstorev1.AssetProviderClient, @@ -548,9 +575,10 @@ func NewService( volumePlugins map[string]volume.VolumePluginWorkerPlane, csiDriverConfigGetter csi.CSIDriverConfigGetter, systemInfoVolumes *systemInfoVolumeRefresher, -) *AteomHerder { - wms := &AteomHerder{ - ateomDialer: ateomDialer, + snapshots *snapshottracker.Tracker, +) *Atelet { + wms := &Atelet{ + workerDialer: workerDialer, imageCache: imageCache, anonGCSClient: anonGCSClient, objectStorePlugin: objectStorePlugin, @@ -561,11 +589,12 @@ func NewService( volumePlugins: volumePlugins, csiDriverConfigGetter: csiDriverConfigGetter, systemInfoVolumes: systemInfoVolumes, + snapshots: snapshots, } return wms } -func (s *AteomHerder) Run(ctx context.Context, req *ateletpb.RunRequest) (resp *ateletpb.RunResponse, err error) { +func (s *Atelet) Run(ctx context.Context, req *ateletpb.RunRequest) (resp *ateletpb.RunResponse, err error) { if errs := apivalidation.ValidateRunRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } @@ -614,19 +643,19 @@ func (s *AteomHerder) Run(ctx context.Context, req *ateletpb.RunRequest) (resp * return nil, err } - client, err := s.dialAteom(ctx, req.GetWorkerPodUid()) + client, err := s.dialWorker(ctx, req.GetWorkerPodUid()) if err != nil { return nil, err } - spec, err := buildAteomWorkloadSpec(req.GetSpec(), containerSpecs) + spec, err := buildWorkerWorkloadSpec(req.GetSpec(), containerSpecs) if err != nil { return nil, apierror.InvalidArgument("invalid workload spec: %v", err) } - // Tell ateom to start the workload. gVisor uses RunscPath; the micro-VM + // Tell worker to start the workload. gVisor uses RunscPath; the micro-VM // runtime uses the full RuntimeAssetPaths set. - if _, err := client.RunWorkload(ctx, &ateompb.RunWorkloadRequest{ + if _, err := client.RunWorkload(ctx, &ateworkerpb.RunWorkloadRequest{ Atespace: actorRef.Atespace, ActorName: actorRef.Name, ActorTemplateAtespace: req.GetActorTemplateAtespace(), @@ -636,11 +665,11 @@ func (s *AteomHerder) Run(ctx context.Context, req *ateletpb.RunRequest) (resp * Spec: spec, ActorUid: actorUID, ActorDirs: ateletpath.ActorDirs(actorUID), - EgressGateway: toAteomEgressGateway(req.GetEgressGateway()), + EgressGateway: toWorkerEgressGateway(req.GetEgressGateway()), CpuMilli: req.GetCpuMilli(), MemoryBytes: req.GetMemoryBytes(), }); err != nil { - return nil, fmt.Errorf("while calling ateom.RunWorkload: %w", err) + return nil, fmt.Errorf("while calling worker RunWorkload: %w", err) } return &ateletpb.RunResponse{}, nil @@ -686,7 +715,7 @@ func allocatedBytes(info os.FileInfo) int64 { return info.Size() } -func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRequest) (_ *ateletpb.CheckpointResponse, err error) { +func (s *Atelet) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRequest) (_ *ateletpb.CheckpointResponse, err error) { if errs := apivalidation.ValidateCheckpointRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } @@ -697,7 +726,7 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe // Per-phase timing, recorded on the way out so a failed checkpoint still // reports the phases it completed. Phases left at zero never ran. tStart := time.Now() - var dAssets, dAteom, dPersist time.Duration + var dAssets, dWorker, dPersist time.Duration op := snapshotOp{ templateNamespace: req.GetActorTemplateAtespace(), templateName: req.GetActorTemplateName(), @@ -716,7 +745,7 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe // excluded from metric labels because of cardinality. phases := []phase{ {ateattr.SnapshotPhaseSandboxAssets, dAssets}, - {ateattr.SnapshotPhaseAteomCheckpoint, dAteom}, + {ateattr.SnapshotPhaseWorkerCheckpoint, dWorker}, {ateattr.SnapshotPhasePersist, dPersist}, {ateattr.SnapshotPhaseTotal, time.Since(tStart)}, } @@ -727,7 +756,7 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe // Checkpoint requests no longer carry the sandbox config; recover the // version this actor was started with from the on-node record and re-fetch - // it (a cache hit) so ateom can drive runsc, and so we can pin it into the + // it (a cache hit) so worker can drive runsc, and so we can pin it into the // snapshot manifest below. sandboxRec, err := readSandboxRecord(actorUID) if err != nil { @@ -744,21 +773,21 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe checkpointDir := ateletpath.CheckpointStateDir(actorUID) - client, err := s.dialAteom(ctx, req.GetWorkerPodUid()) + client, err := s.dialWorker(ctx, req.GetWorkerPodUid()) if err != nil { return nil, err } - // Tell ateom to take the checkpoint and delete containers. ateom reports the + // Tell worker to take the checkpoint and delete containers. worker reports the // exact files it wrote so we ship precisely that set (gVisor's image files, // cloud-hypervisor's snapshot set, ...) rather than a hardcoded list. - spec, err := buildAteomWorkloadSpec(req.GetSpec(), nil) + spec, err := buildWorkerWorkloadSpec(req.GetSpec(), nil) if err != nil { return nil, apierror.InvalidArgument("invalid workload spec: %v", err) } - tAteom := time.Now() - resp, err := client.CheckpointWorkload(ctx, &ateompb.CheckpointWorkloadRequest{ + tWorker := time.Now() + resp, err := client.CheckpointWorkload(ctx, &ateworkerpb.CheckpointWorkloadRequest{ Atespace: actorRef.Atespace, ActorName: actorRef.Name, ActorTemplateAtespace: req.GetActorTemplateAtespace(), @@ -766,16 +795,19 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe RunscPath: runscPathFor(assetPaths), RuntimeAssetPaths: assetPaths, Spec: spec, - Fidelity: toAteomFidelity(req.GetSnapshot().GetFidelity()), + Fidelity: toWorkerFidelity(req.GetSnapshot().GetFidelity()), ActorUid: actorUID, ActorDirs: ateletpath.ActorDirs(actorUID), }) - dAteom = time.Since(tAteom) + dWorker = time.Since(tWorker) if err != nil { - // TODO: Ateom should classify checkpoint failures, and set "should-crash" + // TODO: Worker should classify checkpoint failures, and set "should-crash" // in the metadata if the error is not retriable. - return nil, fmt.Errorf("while calling ateom.CheckpointWorkload: %w", err) + return nil, fmt.Errorf("while calling worker CheckpointWorkload: %w", err) } + // The snapshot commits on this node once worker has written it, in either + // upload mode. + snap := snapshottracker.SnapshotRef{Snapshot: req.GetSnapshot(), CommitTime: time.Now()} s.systemInfoVolumes.Deregister(actorUID) @@ -790,6 +822,17 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe sandboxRec.ActorTemplateName = req.GetActorTemplateName() sandboxRec.Fidelity = ateattr.SnapshotFidelityValue(req.GetSnapshot().GetFidelity()) + // Forget the local snapshot before the prune below deletes it, so the + // tracker never names a deleted snapshot, even if this checkpoint fails. + // + // TODO: Revisit this once the durable case can keep its local snapshot too + // (see UPLOAD_MODE_IMMEDIATE). Today the prune deletes every local + // snapshot, so the tracker forgets the latest one unconditionally. The + // snapshot this checkpoint just took is safe: it stays in + // CheckpointStateDir until after the prune, and only moveLocalCheckpoint + // moves it into LocalCheckpointsDir, the directory the prune empties. + s.snapshots.ForgetLocal(actorUID) + // No earlier pause snapshot can ever be restored again, so remove them // all: the actor's current state was just captured by CheckpointWorkload, // and the control plane tracks only a single local snapshot, which this @@ -810,7 +853,14 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe // snapshots on both paths, so timing it as part of an external upload would // mix local disk deletion into the object-storage measurement. tPersist := time.Now() + var state *ateletpb.ActorSnapshotState switch mode := req.GetUploadConfig().GetMode(); mode { + case ateletpb.UploadMode_UPLOAD_MODE_DEFERRED: + if err := s.moveLocalCheckpoint(ctx, req, sandboxRec); err != nil { + dPersist = time.Since(tPersist) + return nil, fmt.Errorf("while moving to local snapshot: %w", err) + } + state = s.snapshots.CommitLocal(actorUID, snap) case ateletpb.UploadMode_UPLOAD_MODE_IMMEDIATE: // CheckpointWorkload has already deleted the workload, so a retried // Checkpoint cannot redo this upload. It waits for the object-store plugin @@ -821,11 +871,8 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe dPersist = time.Since(tPersist) return nil, fmt.Errorf("while uploading external snapshot: %w", err) } - case ateletpb.UploadMode_UPLOAD_MODE_DEFERRED: - if err := s.moveLocalCheckpoint(ctx, req, sandboxRec); err != nil { - dPersist = time.Since(tPersist) - return nil, fmt.Errorf("while moving to local snapshot: %w", err) - } + snap.UploadTime = time.Now() + state = s.snapshots.CommitDurable(actorUID, snap) default: return nil, fmt.Errorf("unexpected upload mode: %v", mode) } @@ -840,41 +887,41 @@ func (s *AteomHerder) Checkpoint(ctx context.Context, req *ateletpb.CheckpointRe return nil, fmt.Errorf("while resetting actor dirs: %w", err) } - return &ateletpb.CheckpointResponse{}, nil + return &ateletpb.CheckpointResponse{State: state}, nil } -func checkpointSnapshotFiles(resp *ateompb.CheckpointWorkloadResponse, required bool) (files, dataFiles []string, err error) { +func checkpointSnapshotFiles(resp *ateworkerpb.CheckpointWorkloadResponse, required bool) (files, dataFiles []string, err error) { files = resp.GetSnapshotFiles() if len(files) == 0 && required { - return nil, nil, errors.New("ateom reported no snapshot files for checkpoint") + return nil, nil, errors.New("worker reported no snapshot files for checkpoint") } if err := validateSnapshotFiles(files); err != nil { - return nil, nil, fmt.Errorf("ateom reported invalid snapshot files: %w", err) + return nil, nil, fmt.Errorf("worker reported invalid snapshot files: %w", err) } dataFiles = resp.GetDataSnapshotFiles() if err := validateDataSnapshotFiles(files, dataFiles); err != nil { - return nil, nil, fmt.Errorf("ateom reported invalid data snapshot files: %w", err) + return nil, nil, fmt.Errorf("worker reported invalid data snapshot files: %w", err) } return files, dataFiles, nil } -// toAteomFidelity maps the atelet enum onto the identical ateom one, level by +// toWorkerFidelity maps the atelet enum onto the identical worker one, level by // level; the request was validated already, so anything else is a bug and -// goes out as UNSPECIFIED for ateom to reject. -func toAteomFidelity(fidelity ateletpb.SnapshotFidelity) ateompb.SnapshotFidelity { +// goes out as UNSPECIFIED for worker to reject. +func toWorkerFidelity(fidelity ateletpb.SnapshotFidelity) ateworkerpb.SnapshotFidelity { switch fidelity { case ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: - return ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + return ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES case ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS: - return ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS + return ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS case ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: - return ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY + return ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY default: - return ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED + return ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED } } -func (s *AteomHerder) moveLocalCheckpoint(ctx context.Context, req *ateletpb.CheckpointRequest, rec *sandboxAssetsRecord) error { +func (s *Atelet) moveLocalCheckpoint(ctx context.Context, req *ateletpb.CheckpointRequest, rec *sandboxAssetsRecord) error { actorDir := ateletpath.ActorPath(req.GetActorUid()) root, err := os.OpenRoot(actorDir) if err != nil { @@ -895,7 +942,7 @@ func (s *AteomHerder) moveLocalCheckpoint(ctx context.Context, req *ateletpb.Che return fmt.Errorf("while creating local checkpoint directory: %w", err) } - // Move exactly the files ateom reported. + // Move exactly the files worker reported. for _, fileName := range rec.SnapshotFiles { src := filepath.Join(checkpointDir, fileName) dst := filepath.Join(localDir, fileName) @@ -946,7 +993,7 @@ func shouldHaveSnapshots(req *ateletpb.CheckpointRequest) bool { // took. Its plugin calls pass grpc.WaitForReady, so they wait out a plugin // outage for as long as ctx allows rather than only pluginsocket's ready // wait, and their errors are not reported as retryable: see Checkpoint. -func (s *AteomHerder) uploadExternalCheckpoint(ctx context.Context, req *ateletpb.CheckpointRequest, checkpointDir string, rec *sandboxAssetsRecord) error { +func (s *Atelet) uploadExternalCheckpoint(ctx context.Context, req *ateletpb.CheckpointRequest, checkpointDir string, rec *sandboxAssetsRecord) error { uri, err := resources.ParseSnapshotURI(req.GetSnapshot().GetObject().GetSnapshotUri()) if err != nil { return err @@ -961,7 +1008,7 @@ func (s *AteomHerder) uploadExternalCheckpoint(ctx context.Context, req *ateletp // files, never a manifest pointing at files that never landed; retries // overwrite the deterministic object names. opts are passed to every plugin // call, and plugin errors are returned as uploadSnapshotFiles returns them. -func (s *AteomHerder) uploadSnapshot(ctx context.Context, uri resources.SnapshotURI, srcDir string, rec *sandboxAssetsRecord, templateAtespace, templateName string, opts ...grpc.CallOption) error { +func (s *Atelet) uploadSnapshot(ctx context.Context, uri resources.SnapshotURI, srcDir string, rec *sandboxAssetsRecord, templateAtespace, templateName string, opts ...grpc.CallOption) error { root, err := os.OpenRoot(srcDir) if err != nil { return fmt.Errorf("while opening snapshot directory: %w", err) @@ -991,10 +1038,10 @@ func (s *AteomHerder) uploadSnapshot(ctx context.Context, uri resources.Snapshot // PromoteSnapshot uploads a paused actor's local checkpoint to object // storage so the actor can be suspended and is no longer tied to this node. It -// drives no ateom — the actor's sandbox is gone; the checkpoint files and their +// drives no worker — the actor's sandbox is gone; the checkpoint files and their // self-describing manifest already sit under the actor's local-checkpoints // directory, written by an earlier local Checkpoint (pause). -func (s *AteomHerder) PromoteSnapshot(ctx context.Context, req *ateletpb.PromoteSnapshotRequest) (_ *ateletpb.PromoteSnapshotResponse, err error) { +func (s *Atelet) PromoteSnapshot(ctx context.Context, req *ateletpb.PromoteSnapshotRequest) (_ *ateletpb.PromoteSnapshotResponse, err error) { if errs := apivalidation.ValidatePromoteSnapshotRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } @@ -1029,6 +1076,10 @@ func (s *AteomHerder) PromoteSnapshot(ctx context.Context, req *ateletpb.Promote return nil, err } + // Record the upload before the prune below deletes the local copy, so the + // tracker never names a deleted snapshot. + state := s.snapshots.CommitPromoted(req.GetActorUid(), snapshottracker.SnapshotRef{Snapshot: req.GetSnapshot(), UploadTime: time.Now()}) + // Best-effort: the snapshot is in object storage now and the actor will // not resume from this node's local copy. Kept outside the persist window // so local disk deletion does not mix into the upload measurement. @@ -1036,13 +1087,13 @@ func (s *AteomHerder) PromoteSnapshot(ctx context.Context, req *ateletpb.Promote slog.WarnContext(ctx, "failed to prune uploaded local checkpoints", slog.String("actorUID", req.GetActorUid()), slog.Any("err", err)) } - return &ateletpb.PromoteSnapshotResponse{}, nil + return &ateletpb.PromoteSnapshotResponse{State: state}, nil } // uploadLocalCheckpointDir uploads the local checkpoint in localDir to uri. It // returns the sandbox class recorded in the snapshot manifest (empty when the // manifest was not read). Parameterized by localDir for tests. -func (s *AteomHerder) uploadLocalCheckpointDir(ctx context.Context, req *ateletpb.PromoteSnapshotRequest, localDir string, uri resources.SnapshotURI) (string, error) { +func (s *Atelet) uploadLocalCheckpointDir(ctx context.Context, req *ateletpb.PromoteSnapshotRequest, localDir string, uri resources.SnapshotURI) (string, error) { localSnapshotName := req.GetSnapshot().GetSnapshotUuid() manifest, err := readSnapshotManifest(localDir) if errors.Is(err, os.ErrNotExist) { @@ -1124,7 +1175,7 @@ func readSnapshotManifest(dir string) ([]byte, error) { return root.ReadFile(sandboxManifestName) } -func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (resp *ateletpb.RestoreResponse, err error) { +func (s *Atelet) Restore(ctx context.Context, req *ateletpb.RestoreRequest) (resp *ateletpb.RestoreResponse, err error) { if errs := apivalidation.ValidateRestoreRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } @@ -1143,11 +1194,11 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) } // Per-step timing so we can attribute resume latency between the rustfs - // download/decompress, the OCI image unpack, and ateom's own work. Reported on + // download/decompress, the OCI image unpack, and worker's own work. Reported on // the way out, so a failed restore still accounts for the phases it completed. // Phases left at zero never ran. tStart := time.Now() - var dMount, dManifest, dAssets, dDownload, dBundles, dAteom time.Duration + var dMount, dManifest, dAssets, dDownload, dBundles, dWorker time.Duration op := snapshotOp{ templateNamespace: req.GetActorTemplateAtespace(), templateName: req.GetActorTemplateName(), @@ -1169,7 +1220,7 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) {ateattr.SnapshotPhaseSandboxAssets, dAssets}, {ateattr.SnapshotPhaseDownload, dDownload}, {ateattr.SnapshotPhaseOCIUnpack, dBundles}, - {ateattr.SnapshotPhaseAteomRestore, dAteom}, + {ateattr.SnapshotPhaseWorkerRestore, dWorker}, {ateattr.SnapshotPhaseTotal, time.Since(tStart)}, } s.instruments.recordRestore(ctx, op, phases...) @@ -1254,12 +1305,12 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) }() // Download the memory snapshot and prepare the sandbox assets + OCI bundle - // CONCURRENTLY. They are independent — only the final ateom.RestoreWorkload + // CONCURRENTLY. They are independent — only the final worker.RestoreWorkload // needs both — so overlapping the GCS download (~0.5s warm) with the asset // fetch + image unpack hides whichever leg is shorter, and on a cold node // (uncached assets + image, ~2.5s unpack) that overlap is large. var assetPaths map[string]string - var containerSpecs []*ateompb.ContainerSpec + var containerSpecs []*ateworkerpb.ContainerSpec // One per leg: a single field written from both goroutines would race. var downloadErr, prepErr error var prepFailedPhase string @@ -1311,14 +1362,14 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) return nil, err } - client, err := s.dialAteom(ctx, req.GetWorkerPodUid()) + client, err := s.dialWorker(ctx, req.GetWorkerPodUid()) if err != nil { return nil, err } - // Tell ateom to do runsc create + runsc restore for pause container and + // Tell worker to do runsc create + runsc restore for pause container and // all application containers. - spec, err := buildAteomWorkloadSpec(req.GetSpec(), containerSpecs) + spec, err := buildWorkerWorkloadSpec(req.GetSpec(), containerSpecs) if err != nil { return nil, apierror.InvalidArgument("invalid workload spec: %v", err) } @@ -1326,10 +1377,10 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) actorDirs := ateletpath.ActorDirs(actorUID) actorDirs.RestoreDir = checkpointDir - // The ateom_restore phase is opaque from here; ateom logs its own breakdown of + // The worker_restore phase is opaque from here; worker logs its own breakdown of // this call as "Actor restore phases". - tAteom := time.Now() - _, err = client.RestoreWorkload(ctx, &ateompb.RestoreWorkloadRequest{ + tWorker := time.Now() + _, err = client.RestoreWorkload(ctx, &ateworkerpb.RestoreWorkloadRequest{ Atespace: actorRef.Atespace, ActorName: actorRef.Name, ActorTemplateAtespace: req.GetActorTemplateAtespace(), @@ -1337,18 +1388,18 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) RunscPath: runscPathFor(assetPaths), RuntimeAssetPaths: assetPaths, Spec: spec, - Fidelity: toAteomFidelity(req.GetSnapshot().GetFidelity()), + Fidelity: toWorkerFidelity(req.GetSnapshot().GetFidelity()), ActorUid: req.GetActorUid(), ActorDirs: actorDirs, PreserveRestoreDir: directLocal, - EgressGateway: toAteomEgressGateway(req.GetEgressGateway()), + EgressGateway: toWorkerEgressGateway(req.GetEgressGateway()), CpuMilli: req.GetCpuMilli(), MemoryBytes: req.GetMemoryBytes(), }) - dAteom = time.Since(tAteom) + dWorker = time.Since(tWorker) if err != nil { - // TODO: classify the errors returned by Ateom and crash the actor if needed. - return nil, fmt.Errorf("while calling ateom.RestoreWorkload: %w", err) + // TODO: classify the errors returned by Worker and crash the actor if needed. + return nil, fmt.Errorf("while calling worker RestoreWorkload: %w", err) } // Record the sandbox binaries actually running the guest on-node so a @@ -1362,9 +1413,9 @@ func (s *AteomHerder) Restore(ctx context.Context, req *ateletpb.RestoreRequest) return &ateletpb.RestoreResponse{}, nil } -// Terminate terminates any running workload on ateom, unmounts external volumes, +// Terminate terminates any running workload on worker, unmounts external volumes, // and resets actor directories on the node. -func (s *AteomHerder) Terminate(ctx context.Context, req *ateletpb.TerminateRequest) (*ateletpb.TerminateResponse, error) { +func (s *Atelet) Terminate(ctx context.Context, req *ateletpb.TerminateRequest) (*ateletpb.TerminateResponse, error) { if errs := apivalidation.ValidateTerminateRequest(ctx, req); len(errs) > 0 { return nil, resources.ToAPIError(errs) } @@ -1384,16 +1435,16 @@ func (s *AteomHerder) Terminate(ctx context.Context, req *ateletpb.TerminateRequ } assetPaths = paths - client, err := s.dialAteom(ctx, req.GetWorkerPodUid()) + client, err := s.dialWorker(ctx, req.GetWorkerPodUid()) if err != nil { - return nil, fmt.Errorf("failed to dial ateom for terminate (actor: %s, actorUID: %s): %w", actorRef, actorUID, err) + return nil, fmt.Errorf("failed to dial worker for terminate (actor: %s, actorUID: %s): %w", actorRef, actorUID, err) } - spec, err := buildAteomWorkloadSpec(req.GetSpec(), nil) + spec, err := buildWorkerWorkloadSpec(req.GetSpec(), nil) if err != nil { return nil, apierror.InvalidArgument("invalid workload spec: %v", err) } - if _, err := client.TerminateWorkload(ctx, &ateompb.TerminateWorkloadRequest{ + if _, err := client.TerminateWorkload(ctx, &ateworkerpb.TerminateWorkloadRequest{ Atespace: req.GetAtespace(), ActorName: req.GetActorName(), ActorUid: req.GetActorUid(), @@ -1404,9 +1455,9 @@ func (s *AteomHerder) Terminate(ctx context.Context, req *ateletpb.TerminateRequ ActorDirs: ateletpath.ActorDirs(actorUID), }); err != nil { if status.Code(err) == codes.NotFound { - slog.InfoContext(ctx, "workload not found on ateom during terminate", slog.Any("actor", actorRef), slog.String("actorUID", actorUID)) + slog.InfoContext(ctx, "workload not found on worker during terminate", slog.Any("actor", actorRef), slog.String("actorUID", actorUID)) } else { - return nil, fmt.Errorf("failed calling ateom.TerminateWorkload (actor: %s, actorUID: %s): %w", actorRef, actorUID, err) + return nil, fmt.Errorf("failed calling worker TerminateWorkload (actor: %s, actorUID: %s): %w", actorRef, actorUID, err) } } } @@ -1424,6 +1475,9 @@ func (s *AteomHerder) Terminate(ctx context.Context, req *ateletpb.TerminateRequ // up the copies on any other NodeVmsWithLocalSnapshots. This is fine *as of // the day this was written* because today NodeVmsWithLocalSnapshots has at // most one item. + if err := s.snapshots.Forget(actorUID); err != nil { + return nil, fmt.Errorf("failed to delete snapshot state during terminate (actor: %s, actorUID: %s): %w", actorRef, actorUID, err) + } if err := pruneLocalCheckpoints(ctx, actorUID); err != nil { return nil, fmt.Errorf("failed to prune local checkpoints during terminate (actor: %s, actorUID: %s): %w", actorRef, actorUID, err) } @@ -1437,7 +1491,7 @@ func (s *AteomHerder) Terminate(ctx context.Context, req *ateletpb.TerminateRequ } // checkLocalSnapshotFiles verifies each snapshot file exists in dir as a -// regular file. Lstat via os.Root, so a symlink cannot point ateom outside the +// regular file. Lstat via os.Root, so a symlink cannot point worker outside the // snapshot. func checkLocalSnapshotFiles(dir string, files []string) error { root, err := os.OpenRoot(dir) @@ -1459,7 +1513,7 @@ func checkLocalSnapshotFiles(dir string, files []string) error { // copyLocalCheckpoint stages files from the local checkpoint snapshotName under // srcDir into dstDir. Both must be inside actorDir, which confines every access. -func (s *AteomHerder) copyLocalCheckpoint(ctx context.Context, actorDir, snapshotName, srcDir, dstDir string, files []string) error { +func (s *Atelet) copyLocalCheckpoint(ctx context.Context, actorDir, snapshotName, srcDir, dstDir string, files []string) error { root, err := os.OpenRoot(actorDir) if err != nil { return fmt.Errorf("while opening actor directory: %w", err) @@ -1545,7 +1599,7 @@ func copyRootFile(root *os.Root, src, dst string) (int64, error) { return nBytes, errors.Join(err, destination.Close()) } -func (s *AteomHerder) downloadExternalCheckpoint(ctx context.Context, snapshotURI string, dstDir string, files []string) error { +func (s *Atelet) downloadExternalCheckpoint(ctx context.Context, snapshotURI string, dstDir string, files []string) error { if err := s.fetchSnapshotFiles(ctx, snapshotURI, dstDir, files); err != nil { return fmt.Errorf("while downloading snapshot files: %w", err) } @@ -1559,13 +1613,13 @@ func (s *AteomHerder) downloadExternalCheckpoint(ctx context.Context, snapshotUR // with. It is empty for sandboxes without a pause container, which get no // pause bundle. It returns each application container's spec, indexed like // spec.GetContainers(). -func (s *AteomHerder) prepareOCIBundles( +func (s *Atelet) prepareOCIBundles( ctx context.Context, actorUID string, actorRef resources.ActorRef, spec *ateletpb.WorkloadSpec, pauseImage string, -) ([]*ateompb.ContainerSpec, error) { +) ([]*ateworkerpb.ContainerSpec, error) { // Prepare host folders for volume types that need them. for _, vol := range spec.GetVolumes() { if vol.GetDurableDir() != nil { @@ -1589,7 +1643,7 @@ func (s *AteomHerder) prepareOCIBundles( } // Application containers. - containerSpecs := make([]*ateompb.ContainerSpec, len(spec.GetContainers())) + containerSpecs := make([]*ateworkerpb.ContainerSpec, len(spec.GetContainers())) for i, ctr := range spec.GetContainers() { var envs []string for _, env := range ctr.GetEnv() { @@ -1604,11 +1658,11 @@ func (s *AteomHerder) prepareOCIBundles( if err != nil { return fmt.Errorf("while resolving process args for container %q: %w", ctr.GetName(), err) } - containerSpecs[i] = &ateompb.ContainerSpec{ + containerSpecs[i] = &ateworkerpb.ContainerSpec{ Args: args, Env: resolveActorEnv(imageConfig, envs), Capabilities: resolveCapabilities(ctr.GetSecurityContext().GetCapabilities()), - Resources: toAteomResourceLimits(ctr.GetResources()), + Resources: toWorkerResourceLimits(ctr.GetResources()), } return nil }) @@ -1619,21 +1673,21 @@ func (s *AteomHerder) prepareOCIBundles( return containerSpecs, nil } -// dialAteom opens (or reuses) the gRPC connection to the target ateom -// pod and returns an ateom client. -func (s *AteomHerder) dialAteom(ctx context.Context, workerPodUID string) (ateompb.AteomClient, error) { - conn, err := s.ateomDialer.DialAteomPod(ctx, workerPodUID) +// dialWorker opens (or reuses) the gRPC connection to the target worker +// pod and returns a worker client. +func (s *Atelet) dialWorker(ctx context.Context, workerPodUID string) (ateworkerpb.WorkerClient, error) { + conn, err := s.workerDialer.DialWorkerPod(ctx, workerPodUID) if err != nil { - return nil, fmt.Errorf("while getting ateom conn for %s: %w", workerPodUID, err) + return nil, fmt.Errorf("while getting worker conn for %s: %w", workerPodUID, err) } - return ateompb.NewAteomClient(conn), nil + return ateworkerpb.NewWorkerClient(conn), nil } -// buildAteomWorkloadSpec projects the atelet-facing workload spec onto -// the ateom-facing one. containerSpecs holds each container's spec from +// buildWorkerWorkloadSpec projects the atelet-facing workload spec onto +// the worker-facing one. containerSpecs holds each container's spec from // prepareOCIBundles, indexed like spec.GetContainers(); it is nil for RPCs that // start no process. -func buildAteomWorkloadSpec(spec *ateletpb.WorkloadSpec, containerSpecs []*ateompb.ContainerSpec) (*ateompb.WorkloadSpec, error) { +func buildWorkerWorkloadSpec(spec *ateletpb.WorkloadSpec, containerSpecs []*ateworkerpb.ContainerSpec) (*ateworkerpb.WorkloadSpec, error) { volumes := make(map[string]*ateletpb.Volume) for _, vol := range spec.GetVolumes() { name := vol.GetName() @@ -1643,12 +1697,12 @@ func buildAteomWorkloadSpec(spec *ateletpb.WorkloadSpec, containerSpecs []*ateom volumes[name] = vol } - out := &ateompb.WorkloadSpec{} + out := &ateworkerpb.WorkloadSpec{} for i, ctr := range spec.GetContainers() { - var ddMounts []*ateompb.DurableDirVolumeMount - var csiMounts []*ateompb.VolumeMount - var siMounts []*ateompb.SystemInfoVolumeMount - var imgMounts []*ateompb.ImageVolumeMount + var ddMounts []*ateworkerpb.DurableDirVolumeMount + var csiMounts []*ateworkerpb.VolumeMount + var siMounts []*ateworkerpb.SystemInfoVolumeMount + var imgMounts []*ateworkerpb.ImageVolumeMount for _, vm := range ctr.GetVolumeMounts() { volName := vm.GetName() vol, ok := volumes[volName] @@ -1658,22 +1712,22 @@ func buildAteomWorkloadSpec(spec *ateletpb.WorkloadSpec, containerSpecs []*ateom switch { case vol.GetDurableDir() != nil: - ddMounts = append(ddMounts, &ateompb.DurableDirVolumeMount{ + ddMounts = append(ddMounts, &ateworkerpb.DurableDirVolumeMount{ VolumeName: volName, MountPath: vm.GetMountPath(), }) case vol.GetExternal() != nil: - csiMounts = append(csiMounts, &ateompb.VolumeMount{ + csiMounts = append(csiMounts, &ateworkerpb.VolumeMount{ VolumeName: volName, MountPath: vm.GetMountPath(), }) case vol.GetSystemInfo() != nil: - siMounts = append(siMounts, &ateompb.SystemInfoVolumeMount{ + siMounts = append(siMounts, &ateworkerpb.SystemInfoVolumeMount{ VolumeName: volName, MountPath: vm.GetMountPath(), }) case vol.GetImage() != nil: - imgMounts = append(imgMounts, &ateompb.ImageVolumeMount{ + imgMounts = append(imgMounts, &ateworkerpb.ImageVolumeMount{ VolumeName: volName, MountPath: vm.GetMountPath(), }) @@ -1681,13 +1735,13 @@ func buildAteomWorkloadSpec(spec *ateletpb.WorkloadSpec, containerSpecs []*ateom return nil, fmt.Errorf("container %q mounts volume %q with no source set", ctr.GetName(), volName) } } - container := &ateompb.Container{ + container := &ateworkerpb.Container{ Name: ctr.GetName(), DurableDirVolumeMounts: ddMounts, CsiVolumeMounts: csiMounts, SystemInfoVolumeMounts: siMounts, ImageVolumeMounts: imgMounts, - WakeupProbe: toAteomWakeupProbe(ctr.GetWakeupProbe()), + WakeupProbe: toWorkerWakeupProbe(ctr.GetWakeupProbe()), } if containerSpecs != nil { container.ContainerSpec = containerSpecs[i] @@ -1697,30 +1751,30 @@ func buildAteomWorkloadSpec(spec *ateletpb.WorkloadSpec, containerSpecs []*ateom return out, nil } -func toAteomResourceLimits(limits *ateletpb.ResourceLimits) *ateompb.ResourceLimits { +func toWorkerResourceLimits(limits *ateletpb.ResourceLimits) *ateworkerpb.ResourceLimits { if limits == nil { return nil } - return &ateompb.ResourceLimits{MemoryBytes: limits.GetMemoryBytes(), CpuMillis: limits.GetCpuMillis()} + return &ateworkerpb.ResourceLimits{MemoryBytes: limits.GetMemoryBytes(), CpuMillis: limits.GetCpuMillis()} } -func toAteomEgressGateway(gateway *ateletpb.EgressGateway) *ateompb.EgressGateway { +func toWorkerEgressGateway(gateway *ateletpb.EgressGateway) *ateworkerpb.EgressGateway { if gateway == nil { return nil } - return &ateompb.EgressGateway{Address: gateway.GetAddress()} + return &ateworkerpb.EgressGateway{Address: gateway.GetAddress()} } -// toAteomWakeupProbe converts an ateletpb wakeup probe into the ateompb wire +// toWorkerWakeupProbe converts an ateletpb wakeup probe into the ateworkerpb wire // type. Returns nil when the source is nil so containers without a probe -// stay unchanged on the wire to ateom. -func toAteomWakeupProbe(in *ateletpb.WakeupProbe) *ateompb.WakeupProbe { +// stay unchanged on the wire to worker. +func toWorkerWakeupProbe(in *ateletpb.WakeupProbe) *ateworkerpb.WakeupProbe { if in == nil { return nil } - out := &ateompb.WakeupProbe{} + out := &ateworkerpb.WakeupProbe{} if hg := in.GetHttpGet(); hg != nil { - out.HttpGet = &ateompb.HttpGetAction{ + out.HttpGet = &ateworkerpb.HttpGetAction{ Path: hg.GetPath(), Port: hg.GetPort(), } @@ -1729,11 +1783,11 @@ func toAteomWakeupProbe(in *ateletpb.WakeupProbe) *ateompb.WakeupProbe { return out } -type AteomDialer struct { +type WorkerDialer struct { conns *lru.Cache } -// newAteomDialer builds a dialer whose cache closes the connections it +// newWorkerDialer builds a dialer whose cache closes the connections it // evicts. A conn pushed out of the LRU without Close is not reclaimed: grpc // keeps most of its goroutines and buffers alive for the life of the process // (the channel idle timeout parks only one of them). Closing on eviction can @@ -1745,20 +1799,20 @@ type AteomDialer struct { // (e.g. refcounted checkout/release) — rather than being closed out from // under a caller. Worth revisiting if the retryable eviction failures show // up in practice. -func newAteomDialer(size int) *AteomDialer { - return &AteomDialer{ +func newWorkerDialer(size int) *WorkerDialer { + return &WorkerDialer{ conns: lru.NewWithEvictionFunc(size, func(_ lru.Key, value interface{}) { value.(*grpc.ClientConn).Close() }), } } -// ateomSocketPath resolves a pod UID to the ateom socket atelet dials. A +// workerSocketPath resolves a pod UID to the worker socket atelet dials. A // variable because the real path is rooted at the node's BasePath, which a // test cannot serve on. -var ateomSocketPath = nodepath.AteomSocketPath +var workerSocketPath = nodepath.WorkerSocketPath -func (d *AteomDialer) DialAteomPod(ctx context.Context, podUID string) (*grpc.ClientConn, error) { +func (d *WorkerDialer) DialWorkerPod(ctx context.Context, podUID string) (*grpc.ClientConn, error) { key := podUID connAny, ok := d.conns.Get(key) @@ -1767,7 +1821,7 @@ func (d *AteomDialer) DialAteomPod(ctx context.Context, podUID string) (*grpc.Cl } conn, err := grpc.NewClient( - "unix://"+ateomSocketPath(podUID), + "unix://"+workerSocketPath(podUID), grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithStatsHandler(otelgrpc.NewClientHandler()), ) @@ -1780,30 +1834,6 @@ func (d *AteomDialer) DialAteomPod(ctx context.Context, podUID string) (*grpc.Cl return conn, nil } -// writeFileAtomic writes data to path by writing a temp file in the same -// directory and renaming it over the target so readers never observe a -// truncated or partially written value. -func writeFileAtomic(path string, data []byte, perm os.FileMode) error { - f, err := os.CreateTemp(filepath.Dir(path), "."+filepath.Base(path)+".tmp-*") - if err != nil { - return err - } - defer os.Remove(f.Name()) // no-op once the rename succeeds - - if _, err := f.Write(data); err != nil { - f.Close() - return err - } - if err := f.Chmod(perm); err != nil { - f.Close() - return err - } - if err := f.Close(); err != nil { - return err - } - return os.Rename(f.Name(), path) -} - // resetActorDirs empties the actor's directories and leaves them in place for // its next activation. Use removeActorDirs when the actor will not come back. func resetActorDirs(actorUID string) error { @@ -1813,8 +1843,8 @@ func resetActorDirs(actorUID string) error { // copied-up actor-image directories keeping the image's (possibly // read-only) modes, which atelet can't remove as plain root without first // making them writable. (The rootfs itself is just an empty mountpoint - // here: the overlay is mounted in the ateom pod's mount namespace, not - // atelet's, and is detached by ateom at teardown.) + // here: the overlay is mounted in the worker pod's mount namespace, not + // atelet's, and is detached by worker at teardown.) if err := resetDir("bundle dir", ateletpath.OCIBundleDir(actorUID), imagecache.RemoveAllWritable, 0o700); err != nil { return err } diff --git a/cmd/atelet/main_test.go b/cmd/atelet/main_test.go index 95dac97ac3..4939569c84 100644 --- a/cmd/atelet/main_test.go +++ b/cmd/atelet/main_test.go @@ -41,7 +41,7 @@ import ( "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/ocispec" "github.com/agent-substrate/substrate/internal/proto/ateletpb" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/pkg/objectstorage" @@ -113,7 +113,7 @@ func TestCopyLocalCheckpointLinks(t *testing.T) { t.Run("links when it can", func(t *testing.T) { srcDir, dstDir := newDirs(t) - s := &AteomHerder{} + s := &Atelet{} if err := s.copyLocalCheckpoint(context.Background(), filepath.Dir(srcDir), snapshot, srcDir, dstDir, []string{"memory-ranges"}); err != nil { t.Fatalf("copyLocalCheckpoint: %v", err) } @@ -134,7 +134,7 @@ func TestCopyLocalCheckpointLinks(t *testing.T) { linkFile = func(*os.Root, string, string) error { return unix.EXDEV } t.Cleanup(func() { linkFile = orig }) - s := &AteomHerder{} + s := &Atelet{} if err := s.copyLocalCheckpoint(context.Background(), filepath.Dir(srcDir), snapshot, srcDir, dstDir, []string{"memory-ranges"}); err != nil { t.Fatalf("copyLocalCheckpoint: %v", err) } @@ -156,7 +156,7 @@ func TestCopyLocalCheckpointLinks(t *testing.T) { if err := os.Link(filepath.Join(srcDir, snapshot, "memory-ranges"), dst); err != nil { t.Fatal(err) } - s := &AteomHerder{} + s := &Atelet{} // dst already exists, so os.Link fails with EEXIST. if err := s.copyLocalCheckpoint(context.Background(), filepath.Dir(srcDir), snapshot, srcDir, dstDir, []string{"memory-ranges"}); err == nil { t.Fatal("copyLocalCheckpoint accepted a non-EXDEV link failure, want an error") @@ -258,7 +258,7 @@ func TestPrepareOCIBundlesPause(t *testing.T) { t.Run(tc.name, func(t *testing.T) { useTempNodeDirs(t) const actorUID = "actor-uid-1" - s := &AteomHerder{imageCache: newImageVolumeStore(t)} + s := &Atelet{imageCache: newImageVolumeStore(t)} if _, err := s.prepareOCIBundles(t.Context(), actorUID, resources.ActorRef{}, spec, tc.pauseImage); err != nil { t.Fatalf("prepareOCIBundles: %v", err) } @@ -294,16 +294,16 @@ func TestPrepareOCIBundlesFillsContainerSpec(t *testing.T) { }}, } - s := &AteomHerder{imageCache: newImageVolumeStore(t)} + s := &Atelet{imageCache: newImageVolumeStore(t)} got, err := s.prepareOCIBundles(t.Context(), "actor-uid-1", resources.ActorRef{}, spec, "") if err != nil { t.Fatalf("prepareOCIBundles: %v", err) } - want := []*ateompb.ContainerSpec{{ + want := []*ateworkerpb.ContainerSpec{{ Args: []string{"/bin/app", "serve"}, Env: []string{"GREETING=hello", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"}, Capabilities: []string{"CAP_AUDIT_WRITE", "CAP_NET_ADMIN", "CAP_NET_BIND_SERVICE"}, - Resources: &ateompb.ResourceLimits{MemoryBytes: 1 << 30, CpuMillis: 500}, + Resources: &ateworkerpb.ResourceLimits{MemoryBytes: 1 << 30, CpuMillis: 500}, }} if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { t.Errorf("prepareOCIBundles container specs mismatch (-want +got):\n%s", diff) @@ -361,7 +361,7 @@ func TestCheckpointSnapshotFiles(t *testing.T) { {name: "manifest name", files: []string{"checkpoint.img", sandboxManifestName}, required: true, wantErr: true}, } { t.Run(tc.name, func(t *testing.T) { - files, data, err := checkpointSnapshotFiles(&ateompb.CheckpointWorkloadResponse{SnapshotFiles: tc.files, DataSnapshotFiles: tc.data}, tc.required) + files, data, err := checkpointSnapshotFiles(&ateworkerpb.CheckpointWorkloadResponse{SnapshotFiles: tc.files, DataSnapshotFiles: tc.data}, tc.required) if (err != nil) != tc.wantErr { t.Fatalf("checkpointSnapshotFiles() = %v, %v, %v; wantErr %v", files, data, err, tc.wantErr) } @@ -451,7 +451,7 @@ func TestCopyLocalCheckpointRejectsSymlinkOutsideRoot(t *testing.T) { t.Fatal(err) } - err := (&AteomHerder{}).copyLocalCheckpoint(context.Background(), parent, snapshotName, + err := (&Atelet{}).copyLocalCheckpoint(context.Background(), parent, snapshotName, filepath.Join(parent, "local-checkpoint"), restoreDir, []string{"checkpoint.img"}) if err == nil { t.Fatal("copyLocalCheckpoint() followed a symlink outside the local checkpoint directory") @@ -489,70 +489,21 @@ func TestCheckLocalSnapshotFilesRejectsSymlink(t *testing.T) { } } -func TestWriteFileAtomic(t *testing.T) { - dir := t.TempDir() - target := filepath.Join(dir, "actor-id") - - // One shared write over an existing value, as happens on every resume; - // each subtest checks one postcondition. - if err := os.WriteFile(target, []byte("golden-id"), 0o600); err != nil { - t.Fatalf("seeding target: %v", err) - } - if err := writeFileAtomic(target, []byte("counter-1"), 0o644); err != nil { - t.Fatalf("writeFileAtomic: %v", err) - } - - t.Run("replaces content", func(t *testing.T) { - got, err := os.ReadFile(target) - if err != nil { - t.Fatalf("reading target: %v", err) - } - if string(got) != "counter-1" { - t.Errorf("content = %q, want %q", got, "counter-1") - } - }) - - t.Run("sets permissions", func(t *testing.T) { - info, err := os.Stat(target) - if err != nil { - t.Fatalf("stat target: %v", err) - } - if perm := info.Mode().Perm(); perm != 0o644 { - t.Errorf("perm = %o, want 644", perm) - } - }) - - t.Run("leaves no temp files", func(t *testing.T) { - // The directory is visible inside the actor. - entries, err := os.ReadDir(dir) - if err != nil { - t.Fatalf("reading dir: %v", err) - } - if len(entries) != 1 { - names := make([]string, 0, len(entries)) - for _, e := range entries { - names = append(names, e.Name()) - } - t.Errorf("leftover files in identity dir: %v", names) - } - }) -} - -// Every valid atelet scope must map to its ateom counterpart. -func TestToAteomFidelity(t *testing.T) { +// Every valid atelet scope must map to its worker counterpart. +func TestToWorkerFidelity(t *testing.T) { tests := []struct { in ateletpb.SnapshotFidelity - want ateompb.SnapshotFidelity + want ateworkerpb.SnapshotFidelity }{ - {ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, - {ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES}, - {ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS}, - {ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED}, - {ateletpb.SnapshotFidelity(99), ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED}, + {ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY}, + {ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES}, + {ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS}, + {ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED}, + {ateletpb.SnapshotFidelity(99), ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED}, } for _, tc := range tests { - if got := toAteomFidelity(tc.in); got != tc.want { - t.Errorf("toAteomFidelity(%v) = %v, want %v", tc.in, got, tc.want) + if got := toWorkerFidelity(tc.in); got != tc.want { + t.Errorf("toWorkerFidelity(%v) = %v, want %v", tc.in, got, tc.want) } } } @@ -576,7 +527,7 @@ func TestFetchAssetRejectsBadHash(t *testing.T) { t.Fatalf("planting cache file: %v", err) } - s := &AteomHerder{} + s := &Atelet{} _, err := s.fetchAsset(context.Background(), assetEntry{SHA256: badHash}) if err == nil { t.Fatal("fetchAsset returned a cache hit for an invalid hash; validation must run before the os.Stat early return") @@ -615,7 +566,7 @@ func TestFetchAssetStreaming(t *testing.T) { t.Run("good asset is cached", func(t *testing.T) { nodepath.StaticFilesDir = t.TempDir() - s := &AteomHerder{anonGCSClient: fakeObjectStorage{data: content}} + s := &Atelet{anonGCSClient: fakeObjectStorage{data: content}} path, err := s.fetchAsset(context.Background(), assetEntry{URL: url, SHA256: goodHash}) if err != nil { t.Fatalf("fetchAsset: %v", err) @@ -632,7 +583,7 @@ func TestFetchAssetStreaming(t *testing.T) { t.Run("over-cap asset rejected, cache not written", func(t *testing.T) { nodepath.StaticFilesDir = t.TempDir() maxAssetBytes = 4 // content is longer than this - s := &AteomHerder{anonGCSClient: fakeObjectStorage{data: content}} + s := &Atelet{anonGCSClient: fakeObjectStorage{data: content}} _, err := s.fetchAsset(context.Background(), assetEntry{URL: url, SHA256: goodHash}) if err == nil { t.Fatal("fetchAsset accepted an over-cap asset") @@ -646,7 +597,7 @@ func TestFetchAssetStreaming(t *testing.T) { nodepath.StaticFilesDir = t.TempDir() maxAssetBytes = origCap wrongHash := strings.Repeat("a", 64) // valid 64-hex format, wrong value - s := &AteomHerder{anonGCSClient: fakeObjectStorage{data: content}} + s := &Atelet{anonGCSClient: fakeObjectStorage{data: content}} _, err := s.fetchAsset(context.Background(), assetEntry{URL: url, SHA256: wrongHash}) if err == nil { t.Fatal("fetchAsset accepted a hash mismatch") @@ -661,7 +612,7 @@ func TestFetchAssetStreaming(t *testing.T) { maxAssetBytes = origCap // The objectstorage clients tag a missing object with ErrObjectNotFound. notFound := fmt.Errorf("%w: no such object", objectstorage.ErrObjectNotFound) - s := &AteomHerder{anonGCSClient: fakeObjectStorage{err: notFound}} + s := &Atelet{anonGCSClient: fakeObjectStorage{err: notFound}} _, err := s.fetchAsset(context.Background(), assetEntry{URL: url, SHA256: goodHash}) if !errors.Is(err, objectstorage.ErrObjectNotFound) { t.Errorf("missing-object error lost the client's sentinel: %v", err) @@ -671,7 +622,7 @@ func TestFetchAssetStreaming(t *testing.T) { t.Run("malformed url is rejected", func(t *testing.T) { nodepath.StaticFilesDir = t.TempDir() maxAssetBytes = origCap - s := &AteomHerder{anonGCSClient: fakeObjectStorage{data: content}} + s := &Atelet{anonGCSClient: fakeObjectStorage{data: content}} // Invalid percent-escape: url.Parse rejects it inside objectstorage.Open. _, err := s.fetchAsset(context.Background(), assetEntry{URL: "gs://bucket/%zz", SHA256: goodHash}) if err == nil { @@ -682,7 +633,7 @@ func TestFetchAssetStreaming(t *testing.T) { t.Run("network error keeps its context wrap", func(t *testing.T) { nodepath.StaticFilesDir = t.TempDir() maxAssetBytes = origCap - s := &AteomHerder{anonGCSClient: fakeObjectStorage{err: errors.New("connection refused")}} + s := &Atelet{anonGCSClient: fakeObjectStorage{err: errors.New("connection refused")}} _, err := s.fetchAsset(context.Background(), assetEntry{URL: url, SHA256: goodHash}) if err == nil { t.Fatal("fetchAsset accepted a failing open") @@ -699,7 +650,7 @@ func TestFetchAssetStreaming(t *testing.T) { // Internal. Guards against a future removal or reordering of the validation // call at any boundary. func TestRPCBoundariesReject(t *testing.T) { - s := &AteomHerder{} + s := &Atelet{} ctx := context.Background() badUID := "../escape" // valid actor ref, invalid worker pod UID const okAtespace, okID, okActorUID = "ate-demo", "counter-1", "123e4567-e89b-12d3-a456-426614174000" @@ -750,7 +701,7 @@ func TestRPCBoundariesReject(t *testing.T) { }) } -func TestBuildAteomWorkloadSpecForwardsWakeupProbe(t *testing.T) { +func TestBuildWorkerWorkloadSpecForwardsWakeupProbe(t *testing.T) { in := &ateletpb.WorkloadSpec{ Containers: []*ateletpb.Container{ { @@ -766,51 +717,51 @@ func TestBuildAteomWorkloadSpecForwardsWakeupProbe(t *testing.T) { }, }, } - want := &ateompb.WorkloadSpec{ - Containers: []*ateompb.Container{ + want := &ateworkerpb.WorkloadSpec{ + Containers: []*ateworkerpb.Container{ { Name: "with-probe", - WakeupProbe: &ateompb.WakeupProbe{ - HttpGet: &ateompb.HttpGetAction{Path: "/health", Port: 8080}, + WakeupProbe: &ateworkerpb.WakeupProbe{ + HttpGet: &ateworkerpb.HttpGetAction{Path: "/health", Port: 8080}, TimeoutSeconds: 45, }, }, {Name: "without-probe"}, }, } - got, err := buildAteomWorkloadSpec(in, nil) + got, err := buildWorkerWorkloadSpec(in, nil) if err != nil { - t.Fatalf("buildAteomWorkloadSpec failed: %v", err) + t.Fatalf("buildWorkerWorkloadSpec failed: %v", err) } if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { - t.Errorf("buildAteomWorkloadSpec mismatch (-want +got):\n%s", diff) + t.Errorf("buildWorkerWorkloadSpec mismatch (-want +got):\n%s", diff) } } -func TestBuildAteomWorkloadSpecForwardsContainerSpecs(t *testing.T) { +func TestBuildWorkerWorkloadSpecForwardsContainerSpecs(t *testing.T) { in := &ateletpb.WorkloadSpec{ Containers: []*ateletpb.Container{{Name: "main"}, {Name: "sidecar"}}, } - containerSpecs := []*ateompb.ContainerSpec{ + containerSpecs := []*ateworkerpb.ContainerSpec{ {Args: []string{"/app", "serve"}}, {Args: []string{"/sidecar"}}, } - want := &ateompb.WorkloadSpec{ - Containers: []*ateompb.Container{ - {Name: "main", ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app", "serve"}}}, - {Name: "sidecar", ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/sidecar"}}}, + want := &ateworkerpb.WorkloadSpec{ + Containers: []*ateworkerpb.Container{ + {Name: "main", ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app", "serve"}}}, + {Name: "sidecar", ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/sidecar"}}}, }, } - got, err := buildAteomWorkloadSpec(in, containerSpecs) + got, err := buildWorkerWorkloadSpec(in, containerSpecs) if err != nil { - t.Fatalf("buildAteomWorkloadSpec failed: %v", err) + t.Fatalf("buildWorkerWorkloadSpec failed: %v", err) } if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { - t.Errorf("buildAteomWorkloadSpec mismatch (-want +got):\n%s", diff) + t.Errorf("buildWorkerWorkloadSpec mismatch (-want +got):\n%s", diff) } } -func TestBuildAteomWorkloadSpecForwardsDurableDirMounts(t *testing.T) { +func TestBuildWorkerWorkloadSpecForwardsDurableDirMounts(t *testing.T) { in := &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ {Name: "data", DurableDir: &ateletpb.DurableDirVolume{}}, @@ -837,42 +788,42 @@ func TestBuildAteomWorkloadSpecForwardsDurableDirMounts(t *testing.T) { {Name: "no-volumes"}, }, } - // ateom needs the volume NAME as well as the path: the name selects the + // worker needs the volume NAME as well as the path: the name selects the // per-volume directory on the host, and an actor may have several. - want := &ateompb.WorkloadSpec{ - Containers: []*ateompb.Container{ + want := &ateworkerpb.WorkloadSpec{ + Containers: []*ateworkerpb.Container{ { Name: "main", - DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "data", MountPath: "/home/counter"}, {VolumeName: "cache", MountPath: "/var/cache"}, }, - CsiVolumeMounts: []*ateompb.VolumeMount{ + CsiVolumeMounts: []*ateworkerpb.VolumeMount{ {VolumeName: "scratch", MountPath: "/scratch"}, }, - SystemInfoVolumeMounts: []*ateompb.SystemInfoVolumeMount{ + SystemInfoVolumeMounts: []*ateworkerpb.SystemInfoVolumeMount{ {VolumeName: "system-info", MountPath: "/run/ate"}, }, }, { Name: "sidecar", - DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "data", MountPath: "/shared"}, }, }, {Name: "no-volumes"}, }, } - got, err := buildAteomWorkloadSpec(in, nil) + got, err := buildWorkerWorkloadSpec(in, nil) if err != nil { - t.Fatalf("buildAteomWorkloadSpec failed: %v", err) + t.Fatalf("buildWorkerWorkloadSpec failed: %v", err) } if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { - t.Errorf("buildAteomWorkloadSpec mismatch (-want +got):\n%s", diff) + t.Errorf("buildWorkerWorkloadSpec mismatch (-want +got):\n%s", diff) } } -func TestBuildAteomWorkloadSpecValidation(t *testing.T) { +func TestBuildWorkerWorkloadSpecValidation(t *testing.T) { tests := []struct { name string in *ateletpb.WorkloadSpec @@ -934,7 +885,7 @@ func TestBuildAteomWorkloadSpecValidation(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - _, err := buildAteomWorkloadSpec(tc.in, nil) + _, err := buildWorkerWorkloadSpec(tc.in, nil) if err == nil { t.Fatal("expected error, got nil") } @@ -945,14 +896,14 @@ func TestBuildAteomWorkloadSpecValidation(t *testing.T) { } } -func TestToAteomEgressGateway(t *testing.T) { - if got := toAteomEgressGateway(nil); got != nil { - t.Fatalf("toAteomEgressGateway(nil) = %v, want nil", got) +func TestToWorkerEgressGateway(t *testing.T) { + if got := toWorkerEgressGateway(nil); got != nil { + t.Fatalf("toWorkerEgressGateway(nil) = %v, want nil", got) } - want := &ateompb.EgressGateway{Address: "egress.example:443"} - got := toAteomEgressGateway(&ateletpb.EgressGateway{Address: want.Address}) + want := &ateworkerpb.EgressGateway{Address: "egress.example:443"} + got := toWorkerEgressGateway(&ateletpb.EgressGateway{Address: want.Address}) if diff := cmp.Diff(want, got, protocmp.Transform()); diff != "" { - t.Errorf("toAteomEgressGateway mismatch (-want +got):\n%s", diff) + t.Errorf("toWorkerEgressGateway mismatch (-want +got):\n%s", diff) } } @@ -1197,7 +1148,7 @@ func TestDrainOnShutdownForceStopsAfterTimeout(t *testing.T) { } // Image volumes appear on their own ImageVolumeMounts field, separate from durable-dir mounts. -func TestBuildAteomWorkloadSpec_ImageVolumeMounts(t *testing.T) { +func TestBuildWorkerWorkloadSpec_ImageVolumeMounts(t *testing.T) { spec := &ateletpb.WorkloadSpec{ Volumes: []*ateletpb.Volume{ {Name: "agent", Image: &ateletpb.ImageVolumeSource{}}, @@ -1214,9 +1165,9 @@ func TestBuildAteomWorkloadSpec_ImageVolumeMounts(t *testing.T) { }}, } - got, err := buildAteomWorkloadSpec(spec, nil) + got, err := buildWorkerWorkloadSpec(spec, nil) if err != nil { - t.Fatalf("buildAteomWorkloadSpec: %v", err) + t.Fatalf("buildWorkerWorkloadSpec: %v", err) } if len(got.GetContainers()) != 1 { t.Fatalf("containers = %d, want 1", len(got.GetContainers())) @@ -1307,6 +1258,12 @@ func immediateUploadConfig() *ateletpb.UploadConfig { } } +func deferredUploadConfig() *ateletpb.UploadConfig { + return &ateletpb.UploadConfig{ + Mode: ateletpb.UploadMode_UPLOAD_MODE_DEFERRED, + } +} + func validCheckpointRequest() *ateletpb.CheckpointRequest { return &ateletpb.CheckpointRequest{ Atespace: "ate-demo", @@ -1318,6 +1275,7 @@ func validCheckpointRequest() *ateletpb.CheckpointRequest { Spec: &ateletpb.WorkloadSpec{Containers: []*ateletpb.Container{{Name: "worker"}}}, Snapshot: &ateletpb.Snapshot{ SnapshotUuid: "snap-1", + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, Object: &ateletpb.ObjectStorage{SnapshotUri: testSnapshotURI}, }, @@ -1334,6 +1292,7 @@ func validPromoteSnapshotRequest() *ateletpb.PromoteSnapshotRequest { ActorTemplateName: "counter", Snapshot: &ateletpb.Snapshot{ SnapshotUuid: "pause-snap-1", + Generation: 1, Fidelity: ateletpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, Object: &ateletpb.ObjectStorage{SnapshotUri: pausedSnapshotURI}, }, diff --git a/cmd/atelet/metrics_test.go b/cmd/atelet/metrics_test.go index 92f5ef34d2..4792a8f44c 100644 --- a/cmd/atelet/metrics_test.go +++ b/cmd/atelet/metrics_test.go @@ -177,11 +177,11 @@ func TestRecordPhasesSkipsZeroPhases(t *testing.T) { snapshotOp{fidelity: ateattr.SnapshotFidelityMemory}, phase{ateattr.SnapshotPhaseManifestFetch, 50 * time.Millisecond}, phase{ateattr.SnapshotPhaseDownload, 2 * time.Second}, - phase{ateattr.SnapshotPhaseAteomRestore, 0}, + phase{ateattr.SnapshotPhaseWorkerRestore, 0}, phase{ateattr.SnapshotPhaseTotal, 2 * time.Second}) byPhase := phaseValues(t, collectHistogram(t, reader, restoreDurationMetric)) - if _, ok := byPhase[ateattr.SnapshotPhaseAteomRestore]; ok { + if _, ok := byPhase[ateattr.SnapshotPhaseWorkerRestore]; ok { t.Error("a phase that never started was recorded as a zero observation") } if len(byPhase) != 3 { diff --git a/cmd/atelet/oci.go b/cmd/atelet/oci.go index 21bbb0040f..cc39c97d34 100644 --- a/cmd/atelet/oci.go +++ b/cmd/atelet/oci.go @@ -89,7 +89,7 @@ func prepareOCIDirectory(ctx context.Context, imageCache *imagecache.Store, acto return nil, fmt.Errorf("while clearing bundle %q: %w", bundlePath, err) } - // The bundle's rootfs is composed by ateom as an overlay mount just before + // The bundle's rootfs is composed by worker as an overlay mount just before // the workload runs: the cached image layers are the read-only lowerdirs, // and the bundle-local upper/work hold this actor's private writes (wiped // between runs, preserving the pristine-rootfs-per-run contract the old @@ -123,7 +123,7 @@ func prepareOCIDirectory(ctx context.Context, imageCache *imagecache.Store, acto } // Every bind target must exist in the rootfs for the mount to attach; - // ateom creates them through the mounted overlay (they land in the + // worker creates them through the mounted overlay (they land in the // actor's upper). var extraDirs []string for _, vm := range volumeMounts { diff --git a/cmd/atelet/rpc_logging_test.go b/cmd/atelet/rpc_logging_test.go index e9005599c2..f505f04b41 100644 --- a/cmd/atelet/rpc_logging_test.go +++ b/cmd/atelet/rpc_logging_test.go @@ -36,7 +36,7 @@ import ( // requestLogHerder answers Run and fails Restore, so the test covers the // request log of a handled and of a failed RPC. type requestLogHerder struct { - ateletpb.UnimplementedAteomHerderServer + ateletpb.UnimplementedAteletServer } func (requestLogHerder) Run(context.Context, *ateletpb.RunRequest) (*ateletpb.RunResponse, error) { @@ -47,11 +47,11 @@ func (requestLogHerder) Restore(context.Context, *ateletpb.RestoreRequest) (*ate return nil, apierror.Unavailable("asset fetch failed") } -// TestAteomHerderRequestLogMasksSignedAssetURL sends Run and Restore with a +// TestAteletRequestLogMasksSignedAssetURL sends Run and Restore with a // signed https asset URL through a server built with atelet's own options and // logger, and checks the request log carries neither the URL's user info nor // its query. -func TestAteomHerderRequestLogMasksSignedAssetURL(t *testing.T) { +func TestAteletRequestLogMasksSignedAssetURL(t *testing.T) { const ( signedURL = "https://asset-user:userinfo-secret@storage.example.com/bucket/gvisor.tar.zstd?X-Goog-Signature=query-secret&X-Goog-Expires=900" sum = "d547d81401461fd1c679c5c4fa0a6c2b8ef7dc3c22ce23c9e25dcc4c69cfd06f" @@ -62,8 +62,8 @@ func TestAteomHerderRequestLogMasksSignedAssetURL(t *testing.T) { serverboot.InitLoggerWithWriter(&logBuf) lis := bufconn.Listen(1 << 20) - srv := grpc.NewServer(ateomHerderServerOptions(insecure.NewCredentials())...) - ateletpb.RegisterAteomHerderServer(srv, requestLogHerder{}) + srv := grpc.NewServer(ateletServerOptions(insecure.NewCredentials())...) + ateletpb.RegisterAteletServer(srv, requestLogHerder{}) go srv.Serve(lis) t.Cleanup(srv.Stop) conn, err := grpc.NewClient("passthrough:///bufnet", @@ -73,7 +73,7 @@ func TestAteomHerderRequestLogMasksSignedAssetURL(t *testing.T) { t.Fatal(err) } t.Cleanup(func() { conn.Close() }) - client := ateletpb.NewAteomHerderClient(conn) + client := ateletpb.NewAteletClient(conn) assets := &ateletpb.SandboxAssets{ SandboxClass: "gvisor", @@ -96,7 +96,7 @@ func TestAteomHerderRequestLogMasksSignedAssetURL(t *testing.T) { t.Fatalf("request log contains %q from the signed asset URL:\n%s", leak, gotLog) } } - for _, method := range []string{ateletpb.AteomHerder_Run_FullMethodName, ateletpb.AteomHerder_Restore_FullMethodName} { + for _, method := range []string{ateletpb.Atelet_Run_FullMethodName, ateletpb.Atelet_Restore_FullMethodName} { var line string for l := range strings.Lines(gotLog) { if strings.Contains(l, `"method":"`+method+`"`) { diff --git a/cmd/atelet/sandbox_assets.go b/cmd/atelet/sandbox_assets.go index 3e0189e071..f4ec42da35 100644 --- a/cmd/atelet/sandbox_assets.go +++ b/cmd/atelet/sandbox_assets.go @@ -40,6 +40,7 @@ import ( "time" "github.com/agent-substrate/substrate/cmd/atelet/internal/ateletpath" + "github.com/agent-substrate/substrate/cmd/atelet/internal/atomicfile" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/proto/ateletpb" @@ -105,7 +106,7 @@ type sandboxAssetsRecord struct { ActorUID string `json:"actorUid,omitempty"` ActorTemplateAtespace string `json:"actorTemplateAtespace,omitempty"` ActorTemplateName string `json:"actorTemplateName,omitempty"` - // SnapshotFiles are the (relative) names of the files ateom wrote into the + // SnapshotFiles are the (relative) names of the files worker wrote into the // checkpoint directory, as reported by CheckpointWorkloadResponse. Recorded // in the snapshot manifest so Restore ships/downloads exactly this set // (gVisor's image files, cloud-hypervisor's snapshot set, ...). Empty in the @@ -162,7 +163,7 @@ func checkPauseImage(sandboxClass, pauseImage string) error { // several (kata-shim, cloud-hypervisor, ...). // Assets are cached, so re-fetching at Checkpoint/Restore is a no-op once // present. -func (s *AteomHerder) ensureSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord) (map[string]string, error) { +func (s *Atelet) ensureSandboxAssets(ctx context.Context, rec *sandboxAssetsRecord) (map[string]string, error) { if err := os.MkdirAll(nodepath.StaticFilesDir, 0o700); err != nil { return nil, fmt.Errorf("while creating static files dir: %w", err) } @@ -195,7 +196,7 @@ func runscPathFor(paths map[string]string) string { // fetchAsset downloads one content-addressed asset (verifying its sha256) into // the shared static-files cache and returns its local path. On a cache hit it // returns immediately. -func (s *AteomHerder) fetchAsset(ctx context.Context, entry assetEntry) (string, error) { +func (s *Atelet) fetchAsset(ctx context.Context, entry assetEntry) (string, error) { if err := resources.ValidateAssetHash(entry.SHA256); err != nil { return "", wrapFileSystemErr("while validating asset hash", err) } @@ -232,7 +233,7 @@ func (s *AteomHerder) fetchAsset(ctx context.Context, entry assetEntry) (string, // verifying its sha256) and extracts it into a content-addressed directory in // the shared static-files cache, returning the local path of the extracted // `runsc` binary. -func (s *AteomHerder) fetchGVisorRelease(ctx context.Context, entry assetEntry) (string, error) { +func (s *Atelet) fetchGVisorRelease(ctx context.Context, entry assetEntry) (string, error) { if err := resources.ValidateAssetHash(entry.SHA256); err != nil { return "", wrapFileSystemErr("while validating asset hash", err) } @@ -300,7 +301,7 @@ func (s *AteomHerder) fetchGVisorRelease(ctx context.Context, entry assetEntry) // or failed unexpectedly (see anonymousFallbackCodes). A refusal on policy or // content, and a canceled request, are final. atelet verifies the bytes it // caches whichever path fetched them, since it executes them. -func (s *AteomHerder) downloadVerified(ctx context.Context, entry assetEntry, tmpPrefix string) (string, error) { +func (s *Atelet) downloadVerified(ctx context.Context, entry assetEntry, tmpPrefix string) (string, error) { wantSum, err := hex.DecodeString(entry.SHA256) if err != nil { return "", fmt.Errorf("while parsing sha256 hash: %w", err) @@ -382,7 +383,7 @@ var anonymousFallbackCodes = []codes.Code{codes.NotFound, codes.InvalidArgument, // downloadAnonymous streams the gs:// asset entry.URL into the existing file // at path with no credentials, replacing anything a failed plugin fetch left // there, hashing as it goes, and checks the size cap and sha256. -func (s *AteomHerder) downloadAnonymous(ctx context.Context, entry assetEntry, shown, path string, wantSum []byte) error { +func (s *Atelet) downloadAnonymous(ctx context.Context, entry assetEntry, shown, path string, wantSum []byte) error { slog.DebugContext(ctx, "Streaming anonymous download from storage", slog.String("url", shown)) out, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600) if err != nil { @@ -421,7 +422,7 @@ func (s *AteomHerder) downloadAnonymous(ctx context.Context, entry assetEntry, s // The plugin's status message is dropped here and never logged or returned: // atelet's errors become actor crash messages, and a plugin's text may quote a // signed URL or other credential in a form no scrubber can recognize. -func (s *AteomHerder) fetchFromPlugin(ctx context.Context, entry assetEntry, shown, path string) codes.Code { +func (s *Atelet) fetchFromPlugin(ctx context.Context, entry assetEntry, shown, path string) codes.Code { slog.DebugContext(ctx, "Fetching sandbox asset through the object-store plugin", slog.String("url", shown)) _, err := s.assetPlugin.FetchAsset(ctx, &objectstorev1.FetchAssetRequest{ AssetUri: entry.URL, @@ -435,7 +436,7 @@ func (s *AteomHerder) fetchFromPlugin(ctx context.Context, entry assetEntry, sho // pluginError is atelet's own description of the plugin failing with code. // anonErr, if set, is why the anonymous read that followed failed; it is // atelet's own text and is kept in the message the caller receives. -func (s *AteomHerder) pluginError(ctx context.Context, entry assetEntry, shown string, code codes.Code, anonErr error) error { +func (s *Atelet) pluginError(ctx context.Context, entry assetEntry, shown string, code codes.Code, anonErr error) error { newErr := apierror.FailedPrecondition var msg string switch code { @@ -640,7 +641,7 @@ func writeSandboxRecord(actorUID string, rec *sandboxAssetsRecord) error { if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { return wrapFileSystemErr("while creating actor dir", err) } - if err := writeFileAtomic(path, data, 0o600); err != nil { + if err := atomicfile.Write(path, data, 0o600); err != nil { return wrapFileSystemErr("while writing sandbox record", err) } return nil diff --git a/cmd/atelet/sandbox_assets_plugin_test.go b/cmd/atelet/sandbox_assets_plugin_test.go index 049cd8c8ff..1610706dec 100644 --- a/cmd/atelet/sandbox_assets_plugin_test.go +++ b/cmd/atelet/sandbox_assets_plugin_test.go @@ -93,11 +93,11 @@ func (c *countingAssetClient) FetchAsset(ctx context.Context, in *objectstorev1. return c.AssetProviderClient.FetchAsset(ctx, in, opts...) } -// assetHerderFixture is an AteomHerder whose assets come from the in-tree +// assetHerderFixture is an Atelet whose assets come from the in-tree // plugin on an in-memory bucket, with an anonymous client that cannot read // that bucket. type assetHerderFixture struct { - herder *AteomHerder + herder *Atelet bucket *bucketStorage anon *bucketStorage plugin *countingAssetClient @@ -125,7 +125,7 @@ func newAssetHerderFixture(t *testing.T, scheme string) *assetHerderFixture { t.Fatal(err) } f.plugin = &countingAssetClient{AssetProviderClient: objectstoreplugintest.AssetClient(plugin)} - f.herder = &AteomHerder{anonGCSClient: f.anon, assetPlugin: f.plugin, pluginSocket: testPluginSock} + f.herder = &Atelet{anonGCSClient: f.anon, assetPlugin: f.plugin, pluginSocket: testPluginSock} return f } diff --git a/cmd/atelet/sandbox_prewarm.go b/cmd/atelet/sandbox_prewarm.go index 64f6b290ff..ab174bc1e0 100644 --- a/cmd/atelet/sandbox_prewarm.go +++ b/cmd/atelet/sandbox_prewarm.go @@ -63,7 +63,7 @@ const prewarmMaxRetries = 8 // down what that call writes into. var prewarmFinished = func(string, error) {} -// sandboxAssetFetcher is the one slice of AteomHerder the prewarmer needs. +// sandboxAssetFetcher is the one slice of Atelet the prewarmer needs. // Prewarming through the same method as the Run/Restore path keeps the two // fetches from ever diverging on cache layout or validation. type sandboxAssetFetcher interface { diff --git a/cmd/atelet/sandbox_prewarm_test.go b/cmd/atelet/sandbox_prewarm_test.go index 3492369736..2a0731c415 100644 --- a/cmd/atelet/sandbox_prewarm_test.go +++ b/cmd/atelet/sandbox_prewarm_test.go @@ -181,7 +181,7 @@ func TestPrewarmProcessRetries(t *testing.T) { if err := indexer.Add(cfg); err != nil { t.Fatalf("indexer.Add: %v", err) } - herder := &AteomHerder{anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}} + herder := &Atelet{anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}} p := newSandboxPrewarmer(herder, nil, listersv1alpha1.NewSandboxConfigLister(indexer), false) p.queue.Add(cfg.Name) @@ -217,7 +217,7 @@ func TestPrewarmProcessReappliesClassGate(t *testing.T) { } // Any fetch would fail and requeue, so NumRequeues distinguishes // "skipped" from "attempted and failed". - herder := &AteomHerder{anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}} + herder := &Atelet{anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}} p := newSandboxPrewarmer(herder, nil, listersv1alpha1.NewSandboxConfigLister(indexer), false) p.queue.Add(cfg.Name) @@ -276,7 +276,7 @@ func TestPrewarmPauseImage(t *testing.T) { cfg.Spec.Versions[0].PauseImage = pauseRef p := &sandboxPrewarmer{ - assets: &AteomHerder{anonGCSClient: fakeObjectStorage{data: content}}, + assets: &Atelet{anonGCSClient: fakeObjectStorage{data: content}}, images: okStore, } if err := p.prewarm(ctx, cfg); err != nil { @@ -289,7 +289,7 @@ func TestPrewarmPauseImage(t *testing.T) { failCfg := gvisorConfig("gvisor-default", "gs://bucket/runsc", fmt.Sprintf("%x", sha256.Sum256([]byte("other runsc")))) failCfg.Spec.Versions[0].PauseImage = pauseRef p = &sandboxPrewarmer{ - assets: &AteomHerder{anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}}, + assets: &Atelet{anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}}, images: failStore, } if err := p.prewarm(ctx, failCfg); err == nil { @@ -341,7 +341,7 @@ func TestPrewarmTimeout(t *testing.T) { cfg := gvisorConfig("gvisor-default", "gs://bucket/runsc", fmt.Sprintf("%x", sha256.Sum256([]byte("hung runsc")))) cfg.Spec.Versions[0].PauseImage = "" - p := &sandboxPrewarmer{assets: &AteomHerder{anonGCSClient: hangingObjectStorage{}}} + p := &sandboxPrewarmer{assets: &Atelet{anonGCSClient: hangingObjectStorage{}}} done := make(chan error, 1) go func() { done <- p.prewarm(context.Background(), cfg) }() @@ -376,7 +376,7 @@ func TestPrewarmDefaultVersionOnly(t *testing.T) { Assets: map[string]map[string]v1alpha1.AssetFile{runtime.GOARCH: {runscAssetName: {URL: "gs://bucket/other-runsc", SHA256: otherSHA}}}, }}, cfg.Spec.Versions...) - p := &sandboxPrewarmer{assets: &AteomHerder{anonGCSClient: fakeObjectStorage{data: content}}} + p := &sandboxPrewarmer{assets: &Atelet{anonGCSClient: fakeObjectStorage{data: content}}} if err := p.prewarm(ctx, cfg); err != nil { t.Fatalf("prewarm: %v", err) } @@ -386,7 +386,7 @@ func TestPrewarmDefaultVersionOnly(t *testing.T) { // Any fetch would fail, so a nil error means nothing was attempted. cfg.Spec.DefaultVersion = "missing" - p = &sandboxPrewarmer{assets: &AteomHerder{anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}}} + p = &sandboxPrewarmer{assets: &Atelet{anonGCSClient: fakeObjectStorage{err: errors.New("bucket unavailable")}}} if err := p.prewarm(ctx, cfg); err != nil { t.Errorf("prewarm of a config without its default version: %v", err) } @@ -433,7 +433,7 @@ func TestSandboxAssetPrewarmDownloads(t *testing.T) { factory := externalversions.NewSharedInformerFactory(client, 0) informer := factory.Api().V1alpha1().SandboxConfigs().Informer() - herder := &AteomHerder{anonGCSClient: fakeObjectStorage{data: content}} + herder := &Atelet{anonGCSClient: fakeObjectStorage{data: content}} // Handler first, informer start second, mirroring main: atelet startup // must never wait on this informer's sync, and the initial List replays // the pre-existing config into the handler as an Add. diff --git a/cmd/atelet/snapshot_transfer.go b/cmd/atelet/snapshot_transfer.go index e106921a3e..e8c54a4648 100644 --- a/cmd/atelet/snapshot_transfer.go +++ b/cmd/atelet/snapshot_transfer.go @@ -49,7 +49,7 @@ func pluginCallError(op, snapshotURI string, err error) error { } // fetchSnapshotFiles downloads the named snapshot files into dstDir. -func (s *AteomHerder) fetchSnapshotFiles(ctx context.Context, snapshotURI, dstDir string, files []string) error { +func (s *Atelet) fetchSnapshotFiles(ctx context.Context, snapshotURI, dstDir string, files []string) error { if len(files) == 0 { return nil } @@ -66,7 +66,7 @@ func (s *AteomHerder) fetchSnapshotFiles(ctx context.Context, snapshotURI, dstDi // message, and leaves the Unavailable mapping to the caller: an upload that can // be retried reports it through pluginsocket.CallError, one that cannot // does not. -func (s *AteomHerder) uploadSnapshotFiles(ctx context.Context, snapshotURI, srcDir string, files []string, opts ...grpc.CallOption) error { +func (s *Atelet) uploadSnapshotFiles(ctx context.Context, snapshotURI, srcDir string, files []string, opts ...grpc.CallOption) error { if len(files) == 0 { return nil } @@ -80,7 +80,7 @@ func (s *AteomHerder) uploadSnapshotFiles(ctx context.Context, snapshotURI, srcD // fetchManifest returns a snapshot's manifest. The returned error is the // plugin's gRPC status, so a missing manifest reports codes.NotFound. -func (s *AteomHerder) fetchManifest(ctx context.Context, snapshotURI string) ([]byte, error) { +func (s *Atelet) fetchManifest(ctx context.Context, snapshotURI string) ([]byte, error) { dir, err := s.snapshotScratch() if err != nil { return nil, err @@ -94,7 +94,7 @@ func (s *AteomHerder) fetchManifest(ctx context.Context, snapshotURI string) ([] // uploadManifest uploads manifest as the snapshot's manifest, as // uploadSnapshotFiles does. -func (s *AteomHerder) uploadManifest(ctx context.Context, snapshotURI string, manifest []byte, opts ...grpc.CallOption) error { +func (s *Atelet) uploadManifest(ctx context.Context, snapshotURI string, manifest []byte, opts ...grpc.CallOption) error { dir, err := s.snapshotScratch() if err != nil { return err @@ -108,7 +108,7 @@ func (s *AteomHerder) uploadManifest(ctx context.Context, snapshotURI string, ma // snapshotScratch creates a fresh directory for one manifest transfer. The // caller removes it. -func (s *AteomHerder) snapshotScratch() (string, error) { +func (s *Atelet) snapshotScratch() (string, error) { if err := os.MkdirAll(s.snapshotScratchDir, 0o700); err != nil { return "", fmt.Errorf("while creating snapshot scratch dir: %w", err) } diff --git a/cmd/atelet/snapshot_transfer_test.go b/cmd/atelet/snapshot_transfer_test.go index b2f89d7fb0..fb34b56e26 100644 --- a/cmd/atelet/snapshot_transfer_test.go +++ b/cmd/atelet/snapshot_transfer_test.go @@ -27,6 +27,7 @@ import ( "testing" "time" + "github.com/agent-substrate/substrate/cmd/atelet/internal/snapshottracker" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/ateinterceptors" @@ -42,20 +43,22 @@ import ( "google.golang.org/grpc/health" healthpb "google.golang.org/grpc/health/grpc_health_v1" "google.golang.org/grpc/status" + "k8s.io/utils/clock" ) -// newPluginHerder returns an AteomHerder whose snapshots go through the +// newPluginHerder returns an Atelet whose snapshots go through the // object-store plugin backed by store. -func newPluginHerder(t *testing.T, store objectstorage.ObjectStorage) *AteomHerder { +func newPluginHerder(t *testing.T, store objectstorage.ObjectStorage) *Atelet { t.Helper() // Test directories come from t.TempDir, so admit any absolute path. plugin, err := objectstoreplugin.NewNodePlugin(store, "/") if err != nil { t.Fatal(err) } - return &AteomHerder{ + return &Atelet{ objectStorePlugin: objectstoreplugintest.NodeClient(plugin), snapshotScratchDir: t.TempDir(), + snapshots: snapshottracker.New(t.Context(), clock.RealClock{}), } } @@ -182,18 +185,18 @@ func TestObjectStorePluginErrorCodeThroughAtelet(t *testing.T) { transfers := []struct { name string retryable bool - run func(ctx context.Context, s *AteomHerder) error + run func(ctx context.Context, s *Atelet) error }{ - {"download", true, func(ctx context.Context, s *AteomHerder) error { + {"download", true, func(ctx context.Context, s *Atelet) error { return s.downloadExternalCheckpoint(ctx, pausedSnapshotURI, t.TempDir(), []string{"a"}) }}, - {"paused upload", true, func(ctx context.Context, s *AteomHerder) error { + {"paused upload", true, func(ctx context.Context, s *Atelet) error { dir := t.TempDir() writeLocalSnapshot(t, dir, rec, map[string]string{"a": "a"}) _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), dir, uri) return err }}, - {"checkpoint upload", false, func(ctx context.Context, s *AteomHerder) error { + {"checkpoint upload", false, func(ctx context.Context, s *Atelet) error { dir := t.TempDir() if err := os.WriteFile(filepath.Join(dir, "a"), []byte("a"), 0o600); err != nil { return err @@ -208,7 +211,7 @@ func TestObjectStorePluginErrorCodeThroughAtelet(t *testing.T) { if pluginCode == codes.Unavailable && tr.retryable { want = codes.Unavailable } - s := &AteomHerder{ + s := &Atelet{ objectStorePlugin: failingNodePlugin{err: status.Error(pluginCode, "plugin failed")}, snapshotScratchDir: t.TempDir(), } @@ -241,19 +244,19 @@ func TestObjectStorePluginErrorKeepsCredentialsOut(t *testing.T) { transfers := []struct { name string uri string // the URI the transfer names in its error, when it takes one - run func(ctx context.Context, s *AteomHerder) error + run func(ctx context.Context, s *Atelet) error }{ - {"download", signedURI, func(ctx context.Context, s *AteomHerder) error { + {"download", signedURI, func(ctx context.Context, s *Atelet) error { return s.downloadExternalCheckpoint(ctx, signedURI, t.TempDir(), []string{"a"}) }}, - {"upload", signedURI, func(ctx context.Context, s *AteomHerder) error { + {"upload", signedURI, func(ctx context.Context, s *Atelet) error { return s.uploadSnapshotFiles(ctx, signedURI, t.TempDir(), []string{"a"}) }}, - {"manifest fetch", signedURI, func(ctx context.Context, s *AteomHerder) error { + {"manifest fetch", signedURI, func(ctx context.Context, s *Atelet) error { _, err := s.fetchManifest(ctx, signedURI) return err }}, - {"paused upload", "", func(ctx context.Context, s *AteomHerder) error { + {"paused upload", "", func(ctx context.Context, s *Atelet) error { dir := t.TempDir() writeLocalSnapshot(t, dir, rec, map[string]string{"a": "a"}) _, err := s.uploadLocalCheckpointDir(ctx, validPromoteSnapshotRequest(), dir, uri) @@ -264,7 +267,7 @@ func TestObjectStorePluginErrorKeepsCredentialsOut(t *testing.T) { for _, tr := range transfers { t.Run(pluginCode.String()+"/"+tr.name, func(t *testing.T) { logs := captureLogs(t) - s := &AteomHerder{ + s := &Atelet{ objectStorePlugin: failingNodePlugin{err: status.Error(pluginCode, "get "+signedURI+": AccountKey="+secret)}, snapshotScratchDir: t.TempDir(), } @@ -378,7 +381,7 @@ func TestCheckpointUploadWaitsOutPluginOutage(t *testing.T) { if err := pluginsocket.WaitReady(ctx, conn); err != nil { t.Fatalf("WaitReady = %v", err) } - s := &AteomHerder{ + s := &Atelet{ objectStorePlugin: objectstorev1.NewNodeProviderClient(conn), snapshotScratchDir: t.TempDir(), } @@ -439,7 +442,7 @@ func TestCheckpointUploadWaitsOutPluginOutage(t *testing.T) { // fetchManifest keeps the plugin's NotFound readable, which the paused // snapshot upload uses to tell a missing snapshot from a failed probe. func TestFetchManifestKeepsNotFound(t *testing.T) { - s := &AteomHerder{ + s := &Atelet{ objectStorePlugin: failingNodePlugin{err: status.Error(codes.NotFound, "no manifest")}, snapshotScratchDir: t.TempDir(), } diff --git a/cmd/atelet/snapshotlog_test.go b/cmd/atelet/snapshotlog_test.go index 0cf5276358..cc51812e00 100644 --- a/cmd/atelet/snapshotlog_test.go +++ b/cmd/atelet/snapshotlog_test.go @@ -147,7 +147,7 @@ func TestSnapshotLogAttrs(t *testing.T) { {ateattr.SnapshotPhaseSandboxAssets, 10 * time.Millisecond}, {ateattr.SnapshotPhaseDownload, 310 * time.Millisecond}, {ateattr.SnapshotPhaseOCIUnpack, 50 * time.Millisecond}, - {ateattr.SnapshotPhaseAteomRestore, 60 * time.Millisecond}, + {ateattr.SnapshotPhaseWorkerRestore, 60 * time.Millisecond}, {ateattr.SnapshotPhaseTotal, 420 * time.Millisecond}, }, wantStrings: map[string]string{ diff --git a/cmd/atelet/statspoller.go b/cmd/atelet/statspoller.go index 359e57378a..37de64876a 100644 --- a/cmd/atelet/statspoller.go +++ b/cmd/atelet/statspoller.go @@ -37,9 +37,9 @@ import ( "k8s.io/client-go/kubernetes" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/ateomstats" "github.com/agent-substrate/substrate/internal/nodepath" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" + "github.com/agent-substrate/substrate/internal/workerstats" ) // workerPoolLabel is the label the pool controller stamps on every worker pod @@ -49,19 +49,19 @@ import ( const workerPoolLabel = "ate.dev/worker-pool" // minActorStatsPollInterval is the floor a configured poll interval is clamped -// to. The ateoms serve GetActiveWorkloadStats from their own sampling timer, so +// to. The workers serve GetActiveWorkloadStats from their own sampling timer, so // a poll faster than their sample interval only reads the same sample again. // The floor also bounds the worker pod list each sweep sends the apiserver // (resolveWorkerPools), so do not lower it to cut lag. const minActorStatsPollInterval = 50 * time.Second -// statsRPCTimeout bounds one ateom's GetActiveWorkloadStats call. The ateom +// statsRPCTimeout bounds one worker's GetActiveWorkloadStats call. The worker // answers from memory in under a millisecond, whatever its actor count or // guest speed, so the timeout is only for a stuck socket. const statsRPCTimeout = 5 * time.Second -// statsSweepConcurrency bounds how many ateoms one sweep probes at once, one -// call per ateom whatever it hosts. It caps how many stuck sockets can hold a +// statsSweepConcurrency bounds how many workers one sweep probes at once, one +// call per worker whatever it hosts. It caps how many stuck sockets can hold a // hung call open on atelet at the same time. const statsSweepConcurrency = 8 @@ -82,16 +82,16 @@ func clampActorStatsPollInterval(ctx context.Context, configured time.Duration) } // activeStatsClient is the one RPC the poller makes, as a narrow interface so -// tests can fake an ateom without a socket. ateompb.AteomClient satisfies it. +// tests can fake a worker without a socket. ateworkerpb.WorkerClient satisfies it. type activeStatsClient interface { - GetActiveWorkloadStats(ctx context.Context, req *ateompb.GetActiveWorkloadStatsRequest, opts ...grpc.CallOption) (*ateompb.GetActiveWorkloadStatsResponse, error) + GetActiveWorkloadStats(ctx context.Context, req *ateworkerpb.GetActiveWorkloadStatsRequest, opts ...grpc.CallOption) (*ateworkerpb.GetActiveWorkloadStatsResponse, error) } -// statsPoller discovers the node's ateoms from the filesystem and turns their +// statsPoller discovers the node's workers from the filesystem and turns their // workload samples into template-level metrics. // // It holds no worker-to-actor mapping and never asks the control plane: every -// ateom registers itself on disk by creating its socket directory at boot (the +// worker registers itself on disk by creating its socket directory at boot (the // same sockets the lifecycle RPCs dial), so one readdir plus one probe per // socket is complete discovery, and discovery survives an atelet restart. // The CPU used between the old atelet's last sweep and the new one's first is @@ -103,13 +103,13 @@ type statsPoller struct { // second poll onto the same guests. interval time.Duration - // ateomsDir is the directory whose entries are worker pod UIDs - // (nodepath.AteomsDir on a real node; a fixture in tests). - ateomsDir string + // workersDir is the directory whose entries are worker pod UIDs + // (nodepath.WorkersDir on a real node; a fixture in tests). + workersDir string - // dial returns a stats client for one ateom plus the closer that releases + // dial returns a stats client for one worker plus the closer that releases // its connection; a connection lives exactly one probe. Deliberately NOT - // the lifecycle RPCs' cached AteomDialer: at one probe per ateom per + // the lifecycle RPCs' cached WorkerDialer: at one probe per worker per // minute over a local unix socket a cache saves nothing, and sweeping the // node's stale sockets through a shared cache would let telemetry evict // connections the lifecycle RPCs are using. @@ -119,7 +119,7 @@ type statsPoller struct { // node's worker pod UIDs to the pool that owns them, called once per // sweep by resolveWorkerPools, which layers cachedPools over it. A nil // return means the fetch failed. The real fetcher lists the node's pods - // by workerPoolLabel; the ateom directory name IS the worker pod UID, + // by workerPoolLabel; the worker directory name IS the worker pod UID, // which is the join key. fetchWorkerPools func(ctx context.Context) map[string]workerPoolRef @@ -130,7 +130,7 @@ type statsPoller struct { // increments, which can never be re-attributed -- onto a pool-less label // set. Safe because a pod's pool is immutable for the pod's lifetime: an // entry can be stale, never wrong. Only the sweep loop touches it; - // resolveWorkerPools prunes it to the pods whose ateom directories still + // resolveWorkerPools prunes it to the pods whose worker directories still // exist. cachedPools map[string]workerPoolRef @@ -163,7 +163,7 @@ type cpuBaseline struct { // templateAggregate is one tick's sums for one templateKey group: the // bounded label set #174 permits on a TSDB series. Actor and atespace // identity deliberately never reach a metric label; per-actor detail is on -// the ateoms' usage records. +// the workers' usage records. // // The memory fields are point-in-time sums the gauges observe. cpuDeltaUsec is // different: cpu_usage_usec is a cumulative counter per activation, so summing @@ -232,7 +232,7 @@ func (p *statsPoller) run(ctx context.Context) { } } -// tick sweeps every ateom on the node once, adds the sweep's CPU increases +// tick sweeps every worker on the node once, adds the sweep's CPU increases // onto the counters, and publishes the aggregates for the next metric // collection to observe. func (p *statsPoller) tick(ctx context.Context) { @@ -241,24 +241,24 @@ func (p *statsPoller) tick(ctx context.Context) { p.inst.publish(aggs) } -// collect probes every ateom directory, statsSweepConcurrency at a time, and +// collect probes every worker directory, statsSweepConcurrency at a time, and // aggregates the samples it gets. // // One tolerance rule covers all the noise a scan meets: any failure to dial or // call an entry means "not a target this tick", never an error worth more than // a debug line. That uniformly handles stale directories left by deleted -// worker pods (nothing garbage-collects them eagerly), ateoms that have made +// worker pods (nothing garbage-collects them eagerly), workers that have made // their directory but not yet listened, and workers torn down mid-sweep. The // no-sample answers are equally routine: an empty samples list is an idle -// worker, and a pending entry (source UNSPECIFIED) is a workload the ateom +// worker, and a pending entry (source UNSPECIFIED) is a workload the worker // has not measured (boot, restore, or an unreached guest), which adds nothing // but keeps its CPU baseline. func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggregate { - entries, err := os.ReadDir(p.ateomsDir) + entries, err := os.ReadDir(p.workersDir) if err != nil { - // A node with no ateoms directory yet has no workers to measure; the + // A node with no workers directory yet has no workers to measure; the // first RunWorkload dispatch creates it. - slog.DebugContext(ctx, "Actor stats sweep: no ateoms directory", slog.Any("err", err)) + slog.DebugContext(ctx, "Actor stats sweep: no workers directory", slog.Any("err", err)) return nil } @@ -293,14 +293,14 @@ func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggr client, closer, err := p.dial(callCtx, podUID) if err != nil { - slog.DebugContext(ctx, "Actor stats sweep: skipping ateom", slog.String("pod_uid", podUID), slog.Any("err", err)) + slog.DebugContext(ctx, "Actor stats sweep: skipping worker", slog.String("pod_uid", podUID), slog.Any("err", err)) return nil } defer closer.Close() - resp, err := client.GetActiveWorkloadStats(callCtx, &ateompb.GetActiveWorkloadStatsRequest{}) + resp, err := client.GetActiveWorkloadStats(callCtx, &ateworkerpb.GetActiveWorkloadStatsRequest{}) if err != nil { - slog.DebugContext(ctx, "Actor stats sweep: skipping ateom", slog.String("pod_uid", podUID), slog.Any("err", err)) + slog.DebugContext(ctx, "Actor stats sweep: skipping worker", slog.String("pod_uid", podUID), slog.Any("err", err)) return nil } @@ -308,11 +308,11 @@ func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggr answered[podUID] = true mu.Unlock() - // One entry per workload the ateom is hosting; empty when it is + // One entry per workload the worker is hosting; empty when it is // available. for _, sample := range resp.GetSamples() { cpuAt := cpuKey{actorUID: sample.GetActorUid(), epoch: sample.GetEpochUnixNano()} - if sample.GetSource() == ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED { + if sample.GetSource() == ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED { // Pending: hosted but not measured, so it adds nothing // and its CPU baseline carries forward. if last, ok := p.lastCPU[cpuAt]; ok { @@ -326,7 +326,7 @@ func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggr templateNamespace: sample.GetActorTemplateAtespace(), templateName: sample.GetActorTemplateName(), sandboxClass: ateattr.NormalizeSandboxClass(sample.GetSandboxClass()), - source: ateomstats.StatsSourceLabel(sample.GetSource()), + source: workerstats.StatsSourceLabel(sample.GetSource()), workerPool: pools[podUID], } mu.Lock() @@ -340,12 +340,12 @@ func (p *statsPoller) collect(ctx context.Context) map[templateKey]*templateAggr agg.memoryWorkingSetBytes = addSat(agg.memoryWorkingSetBytes, sample.GetMemoryWorkingSetBytes()) // The counter increase since the activation's baseline; the - // ateom never lets it decrease within an activation. An + // worker never lets it decrease within an activation. An // activation with no baseline that began after this atelet // started was never charged, so its value counts in full. One // that began earlier may have been charged by the atelet before // this one, so it only sets the baseline. Both times come from - // this node's clock: atelet reads only the ateoms on its node. + // this node's clock: atelet reads only the workers on its node. cpu := sample.GetCpuUsageUsec() seenCPU[cpuAt] = cpuBaseline{usec: cpu, podUID: podUID} if last, ok := p.lastCPU[cpuAt]; ok { @@ -395,7 +395,7 @@ func addSat(agg int64, v uint64) int64 { // resolveWorkerPools returns this sweep's pod-UID-to-pool map: fresh // resolutions win, a failed or partial list falls back to cachedPools, and -// the result -- rebuilt restricted to the pods whose ateom directories exist +// the result -- rebuilt restricted to the pods whose worker directories exist // -- becomes the new cache, pruning departed pods and bounding its size. The // residual unlabeled case is a pod no fetch has resolved yet, whether first // seen during an outage or omitted by a list racing the directory scan; it @@ -580,18 +580,18 @@ func (i *statsInstruments) addCPU(ctx context.Context, aggs map[templateKey]*tem // main's boot sequence so the subsystem has one obvious entry point. // // The poller dials its own short-lived connection per probe (see -// dialAteomStats) and takes no AteomDialer: the isolation from the lifecycle +// dialWorkerStats) and takes no WorkerDialer: the isolation from the lifecycle // RPCs' connection cache is structural, not just behavioral. func startStatsPoller(ctx context.Context, interval time.Duration, inst *statsInstruments, k8sClient kubernetes.Interface) { poller := &statsPoller{ - interval: interval, - ateomsDir: nodepath.AteomsDir(), + interval: interval, + workersDir: nodepath.WorkersDir(), dial: func(_ context.Context, podUID string) (activeStatsClient, io.Closer, error) { - conn, closer, err := dialAteomStats(podUID) + conn, closer, err := dialWorkerStats(podUID) if err != nil { return nil, nil, err } - return ateompb.NewAteomClient(conn), closer, nil + return ateworkerpb.NewWorkerClient(conn), closer, nil }, inst: inst, startedAt: time.Now().UnixNano(), @@ -606,13 +606,13 @@ func startStatsPoller(ctx context.Context, interval time.Duration, inst *statsIn go poller.run(ctx) } -// dialAteomStats opens the poller's short-lived connection: one per probe, -// closed by the caller, never the lifecycle RPCs' cached AteomDialer. +// dialWorkerStats opens the poller's short-lived connection: one per probe, +// closed by the caller, never the lifecycle RPCs' cached WorkerDialer. // grpc.NewClient is lazy, so this cannot block; the caller's deadline bounds // the actual connect inside the RPC. -func dialAteomStats(podUID string) (*grpc.ClientConn, io.Closer, error) { +func dialWorkerStats(podUID string) (*grpc.ClientConn, io.Closer, error) { conn, err := grpc.NewClient( - "unix://"+nodepath.AteomSocketPath(podUID), + "unix://"+nodepath.WorkerSocketPath(podUID), grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithStatsHandler(otelgrpc.NewClientHandler()), ) diff --git a/cmd/atelet/statspoller_test.go b/cmd/atelet/statspoller_test.go index 0a8e3a6c31..c17067b58e 100644 --- a/cmd/atelet/statspoller_test.go +++ b/cmd/atelet/statspoller_test.go @@ -34,16 +34,16 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" k8sfake "k8s.io/client-go/kubernetes/fake" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) -// fakeStatsAteom answers GetActiveWorkloadStats with a canned response or -// error, standing in for one ateom socket. -type fakeStatsAteom struct { - resp *ateompb.GetActiveWorkloadStatsResponse +// fakeStatsWorker answers GetActiveWorkloadStats with a canned response or +// error, standing in for one worker socket. +type fakeStatsWorker struct { + resp *ateworkerpb.GetActiveWorkloadStatsResponse err error - // mu guards the recordings below: the sweep probes ateoms concurrently. + // mu guards the recordings below: the sweep probes workers concurrently. mu sync.Mutex // calls counts probes, so tests can tell "skipped" from "never found". calls int @@ -52,7 +52,7 @@ type fakeStatsAteom struct { sawDeadline bool } -func (f *fakeStatsAteom) GetActiveWorkloadStats(ctx context.Context, req *ateompb.GetActiveWorkloadStatsRequest, opts ...grpc.CallOption) (*ateompb.GetActiveWorkloadStatsResponse, error) { +func (f *fakeStatsWorker) GetActiveWorkloadStats(ctx context.Context, req *ateworkerpb.GetActiveWorkloadStatsRequest, opts ...grpc.CallOption) (*ateworkerpb.GetActiveWorkloadStatsResponse, error) { f.mu.Lock() f.calls++ _, f.sawDeadline = ctx.Deadline() @@ -60,10 +60,10 @@ func (f *fakeStatsAteom) GetActiveWorkloadStats(ctx context.Context, req *ateomp return f.resp, f.err } -// executingResponse builds the sample an executing ateom would echo. -func executingResponse(templateNS, templateName string, class string, source ateompb.StatsSource, current, workingSet uint64) *ateompb.GetActiveWorkloadStatsResponse { - return &ateompb.GetActiveWorkloadStatsResponse{ - Samples: []*ateompb.WorkloadStatsSample{{ +// executingResponse builds the sample an executing worker would echo. +func executingResponse(templateNS, templateName string, class string, source ateworkerpb.StatsSource, current, workingSet uint64) *ateworkerpb.GetActiveWorkloadStatsResponse { + return &ateworkerpb.GetActiveWorkloadStatsResponse{ + Samples: []*ateworkerpb.WorkloadStatsSample{{ ActorTemplateAtespace: templateNS, ActorTemplateName: templateName, SandboxClass: class, @@ -75,13 +75,13 @@ func executingResponse(templateNS, templateName string, class string, source ate } // measuredSample is one measured entry of a multi-actor response. -func measuredSample(actorUID, templateNS, templateName string, current, workingSet, cpuUsec uint64) *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{ +func measuredSample(actorUID, templateNS, templateName string, current, workingSet, cpuUsec uint64) *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{ ActorUid: actorUID, ActorTemplateAtespace: templateNS, ActorTemplateName: templateName, SandboxClass: "gvisor", - Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, + Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, MemoryCurrentBytes: current, MemoryWorkingSetBytes: workingSet, CpuUsageUsec: cpuUsec, @@ -90,8 +90,8 @@ func measuredSample(actorUID, templateNS, templateName string, current, workingS // pendingSample is one pending entry: attribution present, source // UNSPECIFIED, measurements absent. -func pendingSample(actorUID, templateNS, templateName string) *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{ +func pendingSample(actorUID, templateNS, templateName string) *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{ ActorUid: actorUID, ActorTemplateAtespace: templateNS, ActorTemplateName: templateName, @@ -104,7 +104,7 @@ func pendingSample(actorUID, templateNS, templateName string) *ateompb.WorkloadS const testStartedAt = 1000 // atEpoch sets the activation a sample belongs to. -func atEpoch(s *ateompb.WorkloadStatsSample, epoch int64) *ateompb.WorkloadStatsSample { +func atEpoch(s *ateworkerpb.WorkloadStatsSample, epoch int64) *ateworkerpb.WorkloadStatsSample { s.EpochUnixNano = epoch return s } @@ -115,16 +115,16 @@ func baseline(p *statsPoller, actorUID string, epoch int64) (uint64, bool) { return b.usec, ok } -// availableResponse is an idle ateom's answer: the empty list. -func availableResponse() *ateompb.GetActiveWorkloadStatsResponse { - return &ateompb.GetActiveWorkloadStatsResponse{} +// availableResponse is an idle worker's answer: the empty list. +func availableResponse() *ateworkerpb.GetActiveWorkloadStatsResponse { + return &ateworkerpb.GetActiveWorkloadStatsResponse{} } // pendingResponse is a workload with no numbers yet, per the response // contract: attribution present, source UNSPECIFIED, measurements absent. -func pendingResponse(actorUID string) *ateompb.GetActiveWorkloadStatsResponse { - return &ateompb.GetActiveWorkloadStatsResponse{ - Samples: []*ateompb.WorkloadStatsSample{{ +func pendingResponse(actorUID string) *ateworkerpb.GetActiveWorkloadStatsResponse { + return &ateworkerpb.GetActiveWorkloadStatsResponse{ + Samples: []*ateworkerpb.WorkloadStatsSample{{ ActorUid: actorUID, ActorTemplateAtespace: "ns-a", ActorTemplateName: "tmpl-a", @@ -146,24 +146,24 @@ func (c *closeRecorder) Close() error { return nil } -// newPollerFixture builds a poller over a fixture ateoms directory with one +// newPollerFixture builds a poller over a fixture workers directory with one // subdirectory (and one fake) per entry in fakes. Dialing a UID without a fake // fails, which is the shape of a stale directory whose socket is gone. Every // successful dial hands out a recorded closer; assertClosed checks the // connections-live-exactly-one-probe contract. -func newPollerFixture(t *testing.T, fakes map[string]*fakeStatsAteom) (*statsPoller, map[string]*closeRecorder) { +func newPollerFixture(t *testing.T, fakes map[string]*fakeStatsWorker) (*statsPoller, map[string]*closeRecorder) { t.Helper() dir := t.TempDir() closers := make(map[string]*closeRecorder) for uid := range fakes { if err := os.Mkdir(filepath.Join(dir, uid), 0o700); err != nil { - t.Fatalf("creating fixture ateom dir %q: %v", uid, err) + t.Fatalf("creating fixture worker dir %q: %v", uid, err) } closers[uid] = &closeRecorder{} } return &statsPoller{ - ateomsDir: dir, - startedAt: testStartedAt, + workersDir: dir, + startedAt: testStartedAt, dial: func(_ context.Context, podUID string) (activeStatsClient, io.Closer, error) { f, ok := fakes[podUID] if !ok || f == nil { @@ -176,14 +176,14 @@ func newPollerFixture(t *testing.T, fakes map[string]*fakeStatsAteom) (*statsPol // assertClosed checks that every successfully dialed probe closed its // connection exactly once per sweep -- the RPC failing must not leak it. -func assertClosed(t *testing.T, fakes map[string]*fakeStatsAteom, closers map[string]*closeRecorder, sweeps int) { +func assertClosed(t *testing.T, fakes map[string]*fakeStatsWorker, closers map[string]*closeRecorder, sweeps int) { t.Helper() for uid, f := range fakes { if f == nil { continue // dial fails: no connection to close } if got := closers[uid].closes; got != sweeps { - t.Errorf("ateom %s connection closed %d times over %d sweeps, want %d", uid, got, sweeps, sweeps) + t.Errorf("worker %s connection closed %d times over %d sweeps, want %d", uid, got, sweeps, sweeps) } } } @@ -192,10 +192,10 @@ func TestStatsPollerCollectAggregates(t *testing.T) { // Two actors of the same template on this node, one of another, one idle // worker, one mid-boot: the same-template pair sums, the others contribute // nothing. - fakes := map[string]*fakeStatsAteom{ - "uid-1": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 1000, 700)}, - "uid-2": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 500, 300)}, - "uid-3": {resp: executingResponse("ns-b", "tmpl-b", "microvm", ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT, 42, 40)}, + fakes := map[string]*fakeStatsWorker{ + "uid-1": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 1000, 700)}, + "uid-2": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 500, 300)}, + "uid-3": {resp: executingResponse("ns-b", "tmpl-b", "microvm", ateworkerpb.StatsSource_STATS_SOURCE_GUEST_AGENT, 42, 40)}, "uid-4": {resp: availableResponse()}, "uid-5": {resp: pendingResponse("uid-5-actor")}, } @@ -217,10 +217,10 @@ func TestStatsPollerCollectAggregates(t *testing.T) { for uid, f := range fakes { if f.calls != 1 { - t.Errorf("ateom %s probed %d times, want 1", uid, f.calls) + t.Errorf("worker %s probed %d times, want 1", uid, f.calls) } if !f.sawDeadline { - t.Errorf("ateom %s probed without a deadline; every probe must carry the per-call timeout", uid) + t.Errorf("worker %s probed without a deadline; every probe must carry the per-call timeout", uid) } } assertClosed(t, fakes, closers, 1) @@ -228,10 +228,10 @@ func TestStatsPollerCollectAggregates(t *testing.T) { // TestStatsPollerCollectSkipsFailures pins the scan's one tolerance rule: a // dial or call failure means "not a target this tick", never a failed sweep. -// The healthy ateom's sample must still be aggregated. +// The healthy worker's sample must still be aggregated. func TestStatsPollerCollectSkipsFailures(t *testing.T) { - fakes := map[string]*fakeStatsAteom{ - "uid-healthy": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80)}, + fakes := map[string]*fakeStatsWorker{ + "uid-healthy": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 100, 80)}, "uid-stale": nil, // directory with no reachable socket: dial fails "uid-broken": {err: errors.New("rpc error: connection refused")}, } @@ -247,17 +247,17 @@ func TestStatsPollerCollectSkipsFailures(t *testing.T) { if diff := cmp.Diff(want, got, cmp.AllowUnexported(templateAggregate{}, templateKey{}, workerPoolRef{})); diff != "" { t.Errorf("collect() mismatch (-want +got):\n%s", diff) } - // The broken ateom's RPC failed, but its connection was dialed -- it must + // The broken worker's RPC failed, but its connection was dialed -- it must // be closed all the same. assertClosed(t, fakes, closers, 1) } -// TestStatsPollerCollectNoAteomsDir: a node whose first workload has not -// arrived has no ateoms directory, which is empty coverage, not an error. -func TestStatsPollerCollectNoAteomsDir(t *testing.T) { - p := &statsPoller{ateomsDir: filepath.Join(t.TempDir(), "does-not-exist")} +// TestStatsPollerCollectNoWorkersDir: a node whose first workload has not +// arrived has no workers directory, which is empty coverage, not an error. +func TestStatsPollerCollectNoWorkersDir(t *testing.T) { + p := &statsPoller{workersDir: filepath.Join(t.TempDir(), "does-not-exist")} if got := p.collect(context.Background()); len(got) != 0 { - t.Errorf("collect() with no ateoms dir = %v, want empty", got) + t.Errorf("collect() with no workers dir = %v, want empty", got) } } @@ -337,14 +337,14 @@ func gaugePointCount(t *testing.T, reader *sdkmetric.ManualReader, name string) // cpuResponse is executingResponse with only the CPU counter set, for the // delta tests. -func cpuResponse(actorUID string, cpuUsec uint64) *ateompb.GetActiveWorkloadStatsResponse { - return &ateompb.GetActiveWorkloadStatsResponse{ - Samples: []*ateompb.WorkloadStatsSample{{ +func cpuResponse(actorUID string, cpuUsec uint64) *ateworkerpb.GetActiveWorkloadStatsResponse { + return &ateworkerpb.GetActiveWorkloadStatsResponse{ + Samples: []*ateworkerpb.WorkloadStatsSample{{ ActorUid: actorUID, ActorTemplateAtespace: "ns-a", ActorTemplateName: "tmpl-a", SandboxClass: "gvisor", - Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, + Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, CpuUsageUsec: cpuUsec, }}, } @@ -358,8 +358,8 @@ func cpuResponse(actorUID string, cpuUsec uint64) *ateompb.GetActiveWorkloadStat // contributing and is dropped from the baselines. func TestStatsPollerCPUDeltas(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} - fake := &fakeStatsAteom{resp: cpuResponse("uid-a", 1000)} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-1": fake}) + fake := &fakeStatsWorker{resp: cpuResponse("uid-a", 1000)} + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-1": fake}) if got := p.collect(context.Background())[key].cpuDeltaUsec; got != 0 { t.Errorf("first sweep delta = %d, want 0 (an activation from before atelet started only sets its baseline)", got) @@ -372,7 +372,7 @@ func TestStatsPollerCPUDeltas(t *testing.T) { // A new activation that began after atelet started: its value is all // usage atelet has not charged. - fake.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(cpuResponse("uid-a", 250).Samples[0], testStartedAt+1), }} if got := p.collect(context.Background())[key].cpuDeltaUsec; got != 250 { @@ -395,9 +395,9 @@ func TestStatsPollerCPUDeltas(t *testing.T) { // groups under its pool, an unresolved one groups without pool labels rather // than vanishing, and the two never merge. func TestStatsPollerWorkerPoolLabels(t *testing.T) { - fakes := map[string]*fakeStatsAteom{ - "uid-pooled": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80)}, - "uid-unpooled": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 10, 8)}, + fakes := map[string]*fakeStatsWorker{ + "uid-pooled": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 100, 80)}, + "uid-unpooled": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 10, 8)}, } p, _ := newPollerFixture(t, fakes) p.fetchWorkerPools = func(context.Context) map[string]workerPoolRef { @@ -424,9 +424,9 @@ func TestStatsPollerWorkerPoolLabels(t *testing.T) { // delta is a spec-violating counter Add. Everything pins at MaxInt64 instead. func TestStatsPollerCollectSaturatesCorruptSamples(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} - fakes := map[string]*fakeStatsAteom{ - "uid-1": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, math.MaxUint64, math.MaxUint64)}, - "uid-2": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 1000, 700)}, + fakes := map[string]*fakeStatsWorker{ + "uid-1": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, math.MaxUint64, math.MaxUint64)}, + "uid-2": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 1000, 700)}, } p, _ := newPollerFixture(t, fakes) @@ -447,8 +447,8 @@ func TestStatsPollerCollectSaturatesCorruptSamples(t *testing.T) { // absurd counter value is a huge "increase"; it must clamp, not go negative. func TestStatsPollerCPUDeltaSaturatesCorruptCounter(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} - fake := &fakeStatsAteom{resp: cpuResponse("uid-a", 1000)} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-1": fake}) + fake := &fakeStatsWorker{resp: cpuResponse("uid-a", 1000)} + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-1": fake}) if got := p.collect(context.Background())[key].cpuDeltaUsec; got != 0 { t.Fatalf("first sweep delta = %d, want 0", got) @@ -495,11 +495,11 @@ func TestNewWorkerPoolFetcher(t *testing.T) { // that feeds the monotonic counter and can never be re-attributed -- on the // pooled label set. func TestStatsPollerPoolCacheSurvivesListFlap(t *testing.T) { - resp := executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) + resp := executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) resp.GetSamples()[0].ActorUid = "uid-a" resp.GetSamples()[0].CpuUsageUsec = 1000 - fake := &fakeStatsAteom{resp: resp} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-1": fake}) + fake := &fakeStatsWorker{resp: resp} + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-1": fake}) listOK := true p.fetchWorkerPools = func(context.Context) map[string]workerPoolRef { if !listOK { @@ -519,7 +519,7 @@ func TestStatsPollerPoolCacheSurvivesListFlap(t *testing.T) { // Sweep 2: the list fails AND the actor consumed CPU. Both the sample and // its delta must still group under the pool. listOK = false - resp2 := executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) + resp2 := executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) resp2.GetSamples()[0].ActorUid = "uid-a" resp2.GetSamples()[0].CpuUsageUsec = 1600 fake.resp = resp2 @@ -536,16 +536,16 @@ func TestStatsPollerPoolCacheSurvivesListFlap(t *testing.T) { } // TestStatsPollerPoolCachePrunes: the cache is rebuilt against the pods whose -// ateom directories exist, so a departed pod's entry does not linger. +// worker directories exist, so a departed pod's entry does not linger. func TestStatsPollerPoolCachePrunes(t *testing.T) { - fakes := map[string]*fakeStatsAteom{ + fakes := map[string]*fakeStatsWorker{ "uid-1": {resp: availableResponse()}, } p, _ := newPollerFixture(t, fakes) p.fetchWorkerPools = func(context.Context) map[string]workerPoolRef { return map[string]workerPoolRef{ "uid-1": {namespace: "pool-ns", name: "pool-a"}, - "uid-gone": {namespace: "pool-ns", name: "pool-a"}, // no ateom dir + "uid-gone": {namespace: "pool-ns", name: "pool-a"}, // no worker dir } } @@ -555,7 +555,7 @@ func TestStatsPollerPoolCachePrunes(t *testing.T) { t.Errorf("cachedPools lost the live pod's entry: %v", p.cachedPools) } if _, ok := p.cachedPools["uid-gone"]; ok { - t.Errorf("cachedPools kept an entry with no ateom directory: %v", p.cachedPools) + t.Errorf("cachedPools kept an entry with no worker directory: %v", p.cachedPools) } } @@ -563,8 +563,8 @@ func TestStatsPollerPoolCachePrunes(t *testing.T) { // is failing has no cache entry to fall back to -- it groups without pool // labels (the residual, documented case) and heals on the next good list. func TestStatsPollerPoolCacheMissDuringOutage(t *testing.T) { - fakes := map[string]*fakeStatsAteom{ - "uid-new": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 10, 8)}, + fakes := map[string]*fakeStatsWorker{ + "uid-new": {resp: executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 10, 8)}, } p, _ := newPollerFixture(t, fakes) p.fetchWorkerPools = func(context.Context) map[string]workerPoolRef { return nil } @@ -583,11 +583,11 @@ func TestStatsPollerPoolCacheMissDuringOutage(t *testing.T) { // all-nil flap test above -- a whole-map fallback would pass that test and // fail this one. func TestStatsPollerPoolCachePartialListFallsBack(t *testing.T) { - resp := executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) + resp := executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) resp.GetSamples()[0].ActorUid = "uid-a" resp.GetSamples()[0].CpuUsageUsec = 1000 - fake := &fakeStatsAteom{resp: resp} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-1": fake}) + fake := &fakeStatsWorker{resp: resp} + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-1": fake}) full := true p.fetchWorkerPools = func(context.Context) map[string]workerPoolRef { if !full { @@ -606,7 +606,7 @@ func TestStatsPollerPoolCachePartialListFallsBack(t *testing.T) { } full = false - resp2 := executingResponse("ns-a", "tmpl-a", "gvisor", ateompb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) + resp2 := executingResponse("ns-a", "tmpl-a", "gvisor", ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, 100, 80) resp2.GetSamples()[0].ActorUid = "uid-a" resp2.GetSamples()[0].CpuUsageUsec = 1250 fake.resp = resp2 @@ -623,8 +623,8 @@ func TestStatsPollerPoolCachePartialListFallsBack(t *testing.T) { // worker: same-template entries sum, another template gets its own key, and // a pending entry contributes nothing. func TestStatsPollerFoldsMultiActorWorker(t *testing.T) { - fakes := map[string]*fakeStatsAteom{ - "uid-w1": {resp: &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fakes := map[string]*fakeStatsWorker{ + "uid-w1": {resp: &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ measuredSample("actor-1", "ns-a", "tmpl-a", 1000, 700, 0), measuredSample("actor-2", "ns-a", "tmpl-a", 500, 300, 0), measuredSample("actor-3", "ns-b", "tmpl-b", 42, 40, 0), @@ -658,12 +658,12 @@ func TestStatsPollerFoldsMultiActorWorker(t *testing.T) { // a pending sibling neither gains a baseline nor disturbs the others'. func TestStatsPollerMultiActorCPUBaselines(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} - fake := &fakeStatsAteom{resp: &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake := &fakeStatsWorker{resp: &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1000), measuredSample("actor-2", "ns-a", "tmpl-a", 1, 1, 5000), pendingSample("actor-3", "ns-a", "tmpl-a"), }}} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-w1": fake}) + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-w1": fake}) if got := p.collect(context.Background())[key].cpuDeltaUsec; got != 0 { t.Fatalf("first sweep delta = %d, want 0 (baselines only)", got) @@ -673,7 +673,7 @@ func TestStatsPollerMultiActorCPUBaselines(t *testing.T) { } // actor-1 advances by 600, actor-2 by 250; the sum lands on the shared key. - fake.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1600), measuredSample("actor-2", "ns-a", "tmpl-a", 1, 1, 5250), pendingSample("actor-3", "ns-a", "tmpl-a"), @@ -684,7 +684,7 @@ func TestStatsPollerMultiActorCPUBaselines(t *testing.T) { // actor-2 starts a new activation; actor-1 keeps advancing. The new // activation charges its value, not a negative delta. - fake.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1700), atEpoch(measuredSample("actor-2", "ns-a", "tmpl-a", 1, 1, 40), testStartedAt+1), }} @@ -701,14 +701,14 @@ func TestStatsPollerMultiActorCPUBaselines(t *testing.T) { // the actor is measured again, not lost. func TestStatsPollerPendingKeepsCPUBaseline(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} - fake := &fakeStatsAteom{resp: &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake := &fakeStatsWorker{resp: &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1000), }}} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-w1": fake}) + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-w1": fake}) p.collect(context.Background()) // baseline 1000 - fake.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ pendingSample("actor-1", "ns-a", "tmpl-a"), }} if got := p.collect(context.Background()); len(got) != 0 { @@ -718,7 +718,7 @@ func TestStatsPollerPendingKeepsCPUBaseline(t *testing.T) { t.Fatalf("baseline after pending sweep = %d (present=%v), want 1000 kept", got, ok) } - fake.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1600), }} if got := p.collect(context.Background())[key].cpuDeltaUsec; got != 600 { @@ -731,31 +731,31 @@ func TestStatsPollerPendingKeepsCPUBaseline(t *testing.T) { // charges its first value in full only if it began after atelet started. func TestStatsPollerCPUEpochs(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} - sample := func(epoch int64, cpu uint64) *ateompb.WorkloadStatsSample { + sample := func(epoch int64, cpu uint64) *ateworkerpb.WorkloadStatsSample { return atEpoch(measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, cpu), epoch) } for _, tc := range []struct { name string - sweeps []*ateompb.WorkloadStatsSample // nil is a pending sweep - wantDelta int64 // on the last sweep + sweeps []*ateworkerpb.WorkloadStatsSample // nil is a pending sweep + wantDelta int64 // on the last sweep }{ - {"increase within an activation", []*ateompb.WorkloadStatsSample{sample(2000, 1000), sample(2000, 1400)}, 400}, - {"decrease within an activation", []*ateompb.WorkloadStatsSample{sample(2000, 1000), sample(2000, 400)}, 0}, - {"new activation after start", []*ateompb.WorkloadStatsSample{sample(2000, 1000), sample(3000, 400)}, 400}, - {"new activation across a pending sweep", []*ateompb.WorkloadStatsSample{sample(2000, 1000), nil, sample(3000, 400)}, 400}, - {"first sight of an activation after start", []*ateompb.WorkloadStatsSample{sample(2000, 700)}, 700}, - {"first sight of an activation at start", []*ateompb.WorkloadStatsSample{sample(testStartedAt, 700)}, 0}, - {"first sight of an activation before start", []*ateompb.WorkloadStatsSample{sample(testStartedAt-1, 700)}, 0}, + {"increase within an activation", []*ateworkerpb.WorkloadStatsSample{sample(2000, 1000), sample(2000, 1400)}, 400}, + {"decrease within an activation", []*ateworkerpb.WorkloadStatsSample{sample(2000, 1000), sample(2000, 400)}, 0}, + {"new activation after start", []*ateworkerpb.WorkloadStatsSample{sample(2000, 1000), sample(3000, 400)}, 400}, + {"new activation across a pending sweep", []*ateworkerpb.WorkloadStatsSample{sample(2000, 1000), nil, sample(3000, 400)}, 400}, + {"first sight of an activation after start", []*ateworkerpb.WorkloadStatsSample{sample(2000, 700)}, 700}, + {"first sight of an activation at start", []*ateworkerpb.WorkloadStatsSample{sample(testStartedAt, 700)}, 0}, + {"first sight of an activation before start", []*ateworkerpb.WorkloadStatsSample{sample(testStartedAt-1, 700)}, 0}, } { t.Run(tc.name, func(t *testing.T) { - fake := &fakeStatsAteom{} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-w1": fake}) + fake := &fakeStatsWorker{} + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-w1": fake}) var got int64 for _, sw := range tc.sweeps { if sw == nil { sw = atEpoch(pendingSample("actor-1", "ns-a", "tmpl-a"), 3000) } - fake.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{sw}} + fake.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{sw}} got = 0 if agg := p.collect(context.Background())[key]; agg != nil { got = agg.cpuDeltaUsec @@ -775,19 +775,19 @@ func TestStatsPollerCPUEpochs(t *testing.T) { func TestStatsPollerRestoreChargesEachActivationOnce(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} const before, after = 2000, 3000 - src := &fakeStatsAteom{resp: &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + src := &fakeStatsWorker{resp: &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1000), before), }}} - dst := &fakeStatsAteom{resp: availableResponse()} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-src": src, "uid-dst": dst}) + dst := &fakeStatsWorker{resp: availableResponse()} + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-src": src, "uid-dst": dst}) p.collect(context.Background()) // The source still measures its activation; the destination hosts the new // one, not yet measured. - src.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + src.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1300), before), }} - dst.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + dst.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(pendingSample("actor-1", "ns-a", "tmpl-a"), after), }} if got := p.collect(context.Background())[key].cpuDeltaUsec; got != 300 { @@ -796,7 +796,7 @@ func TestStatsPollerRestoreChargesEachActivationOnce(t *testing.T) { // Only the destination hosts it now, 200 into its activation. src.resp = availableResponse() - dst.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + dst.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 200), after), }} if got := p.collect(context.Background())[key].cpuDeltaUsec; got != 200 { @@ -812,10 +812,10 @@ func TestStatsPollerRestoreChargesEachActivationOnce(t *testing.T) { // increase rather than the activation in full. func TestStatsPollerKeepsBaselineOfSilentWorker(t *testing.T) { key := templateKey{templateNamespace: "ns-a", templateName: "tmpl-a", sandboxClass: "gvisor", source: "cgroup"} - fake := &fakeStatsAteom{resp: &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake := &fakeStatsWorker{resp: &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1000), 2000), }}} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-w1": fake}) + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-w1": fake}) p.collect(context.Background()) fake.err = errors.New("deadline exceeded") @@ -825,7 +825,7 @@ func TestStatsPollerKeepsBaselineOfSilentWorker(t *testing.T) { } fake.err = nil - fake.resp = &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake.resp = &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1600), 2000), }} if got := p.collect(context.Background())[key].cpuDeltaUsec; got != 600 { @@ -837,10 +837,10 @@ func TestStatsPollerKeepsBaselineOfSilentWorker(t *testing.T) { // that stops answering for good, as one killed without cleanup, keeps its // baselines for as long as its directory stays. func TestStatsPollerSilentWorkerKeepsBaselinesUntilDirGone(t *testing.T) { - fake := &fakeStatsAteom{resp: &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake := &fakeStatsWorker{resp: &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1000), 2000), }}} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-w1": fake}) + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-w1": fake}) p.collect(context.Background()) fake.err = errors.New("connection refused") @@ -850,7 +850,7 @@ func TestStatsPollerSilentWorkerKeepsBaselinesUntilDirGone(t *testing.T) { if _, ok := baseline(p, "actor-1", 2000); !ok { t.Fatal("baseline dropped while the silent worker's directory stays") } - if err := os.Remove(filepath.Join(p.ateomsDir, "uid-w1")); err != nil { + if err := os.Remove(filepath.Join(p.workersDir, "uid-w1")); err != nil { t.Fatal(err) } p.collect(context.Background()) @@ -862,13 +862,13 @@ func TestStatsPollerSilentWorkerKeepsBaselinesUntilDirGone(t *testing.T) { // TestStatsPollerDropsBaselineOfGoneWorker pins that a worker whose // directory is gone takes its baselines with it. func TestStatsPollerDropsBaselineOfGoneWorker(t *testing.T) { - fake := &fakeStatsAteom{resp: &ateompb.GetActiveWorkloadStatsResponse{Samples: []*ateompb.WorkloadStatsSample{ + fake := &fakeStatsWorker{resp: &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: []*ateworkerpb.WorkloadStatsSample{ atEpoch(measuredSample("actor-1", "ns-a", "tmpl-a", 1, 1, 1000), 2000), }}} - p, _ := newPollerFixture(t, map[string]*fakeStatsAteom{"uid-w1": fake}) + p, _ := newPollerFixture(t, map[string]*fakeStatsWorker{"uid-w1": fake}) p.collect(context.Background()) - if err := os.Remove(filepath.Join(p.ateomsDir, "uid-w1")); err != nil { + if err := os.Remove(filepath.Join(p.workersDir, "uid-w1")); err != nil { t.Fatal(err) } p.collect(context.Background()) diff --git a/cmd/atelet/systeminfovolume.go b/cmd/atelet/systeminfovolume.go index 05f1c444e3..9f30782148 100644 --- a/cmd/atelet/systeminfovolume.go +++ b/cmd/atelet/systeminfovolume.go @@ -258,7 +258,7 @@ func writeSystemInfoFile(root *os.Root, relPath string, data []byte) (bool, erro return true, nil } -// writeFileAtomicRoot is writeFileAtomic confined to root. The fixed temp +// writeFileAtomicRoot is atomicfile.Write confined to root. The fixed temp // name is safe because writers of an actor's volumes serialize on its lock. func writeFileAtomicRoot(root *os.Root, relPath string, data []byte, perm os.FileMode) error { dir, base := filepath.Dir(relPath), filepath.Base(relPath) diff --git a/cmd/atelet/volumes.go b/cmd/atelet/volumes.go index 2cc0ee6810..149d89ac97 100644 --- a/cmd/atelet/volumes.go +++ b/cmd/atelet/volumes.go @@ -29,8 +29,13 @@ import ( "google.golang.org/grpc/status" ) -func (s *AteomHerder) mountExternalVolumes(ctx context.Context, actorUID string, volumes []*ateletpb.Volume) error { +func (s *Atelet) mountExternalVolumes(ctx context.Context, actorUID string, volumes []*ateletpb.Volume) error { for _, vol := range volumes { + // Given that external volumes has been in the codebase for a while and + // its preview-ness is solely around the API, and that the impact of + // disabling the gate while in use would be an orphaned volume + // attachment, we are NOT checking the ExternalVolumes gate in the + // mount/unmount logic. ext := vol.GetExternal() if ext == nil { continue @@ -56,9 +61,14 @@ func (s *AteomHerder) mountExternalVolumes(ctx context.Context, actorUID string, return nil } -func (s *AteomHerder) unmountExternalVolumes(ctx context.Context, actorUID string, volumes []*ateletpb.Volume) error { +func (s *Atelet) unmountExternalVolumes(ctx context.Context, actorUID string, volumes []*ateletpb.Volume) error { var errs []error for _, vol := range volumes { + // Given that external volumes has been in the codebase for a while and + // its preview-ness is solely around the API, and that the impact of + // disabling the gate while in use would be an orphaned volume + // attachment, we are NOT checking the ExternalVolumes gate in the + // mount/unmount logic. ext := vol.GetExternal() if ext == nil { continue @@ -83,7 +93,7 @@ func (s *AteomHerder) unmountExternalVolumes(ctx context.Context, actorUID strin return errors.Join(errs...) } -func (s *AteomHerder) getPlugin(ctx context.Context, driverName string) (volume.VolumePluginWorkerPlane, error) { +func (s *Atelet) getPlugin(ctx context.Context, driverName string) (volume.VolumePluginWorkerPlane, error) { s.mu.RLock() plugin, ok := s.volumePlugins[driverName] s.mu.RUnlock() diff --git a/cmd/atelet/volumes_test.go b/cmd/atelet/volumes_test.go index cab35adcb2..7e57e36a2b 100644 --- a/cmd/atelet/volumes_test.go +++ b/cmd/atelet/volumes_test.go @@ -110,7 +110,7 @@ func TestUnmountExternalVolumes(t *testing.T) { t.Run("success", func(t *testing.T) { fake := &fakeWorkerPlugin{} - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -129,7 +129,7 @@ func TestUnmountExternalVolumes(t *testing.T) { fake := &fakeWorkerPlugin{ unmountErr: errors.New("device or resource busy"), } - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -148,7 +148,7 @@ func TestUnmountExternalVolumes(t *testing.T) { fake := &fakeWorkerPlugin{ unmountErr: fmt.Errorf("unmount failed"), } - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -194,7 +194,7 @@ func TestMountExternalVolumes(t *testing.T) { withTempActorsDir(t) fake := &fakeWorkerPlugin{} - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -243,7 +243,7 @@ func TestMountExternalVolumes(t *testing.T) { } fake := &fakeWorkerPlugin{} - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -269,7 +269,7 @@ func TestMountExternalVolumes(t *testing.T) { }, } - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{}, } @@ -288,7 +288,7 @@ func TestMountExternalVolumes(t *testing.T) { fake := &fakeWorkerPlugin{ mountErr: errors.New("mount operation failed: device or resource busy"), } - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -309,7 +309,7 @@ func TestMountExternalVolumes(t *testing.T) { fake := &fakeWorkerPlugin{ mountErr: errors.New("cannot mount volume"), } - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -335,7 +335,7 @@ func TestMountExternalVolumes(t *testing.T) { "mock-vol-2": errors.New("simulated mount error on vol-2"), }, } - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -397,7 +397,7 @@ func TestVolumeHostDirectoryCleanup(t *testing.T) { withTempActorsDir(t) fake := &fakeWorkerPlugin{} - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -431,7 +431,7 @@ func TestVolumeHostDirectoryCleanup(t *testing.T) { withTempActorsDir(t) fake := &fakeWorkerPlugin{} - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, @@ -479,7 +479,7 @@ func TestVolumeHostDirectoryCleanup(t *testing.T) { withTempActorsDir(t) fake := &fakeWorkerPlugin{} - s := &AteomHerder{ + s := &Atelet{ volumePlugins: map[string]volume.VolumePluginWorkerPlane{ "mock-driver": fake, }, diff --git a/cmd/atelet/worker_sandbox_syncer.go b/cmd/atelet/worker_sandbox_syncer.go index b8094ae63a..1378d08034 100644 --- a/cmd/atelet/worker_sandbox_syncer.go +++ b/cmd/atelet/worker_sandbox_syncer.go @@ -56,7 +56,7 @@ import ( const podUIDIndex = "uid" // orphanMinAge is how old a worker directory found on disk at startup must be -// before the syncer looks at it. The grace covers an ateom that made its own +// before the syncer looks at it. The grace covers a worker that made its own // directory before its pod reached atelet's cache. const orphanMinAge = time.Minute @@ -78,7 +78,7 @@ const syncMaxJitter = 30 * time.Second const syncMaxRetries = 8 // workerSandboxSyncer keeps a sandbox versions file in each local worker's -// directory under ateomsDir: the runtimes of the enabled versions of the +// directory under workersDir: the runtimes of the enabled versions of the // SandboxConfig the worker's pool uses. It creates a worker's directory once // it writes the pod's file and removes it when the pod is gone. type workerSandboxSyncer struct { @@ -87,13 +87,13 @@ type workerSandboxSyncer struct { pods cache.Indexer // queue carries pod UIDs. One worker drains it, so a pod's file is never // written and its directory removed at the same time. - queue workqueue.TypedRateLimitingInterface[string] - ateomsDir string // nodepath.AteomsDir(); a temp dir in tests - minAge time.Duration - maxJitter time.Duration + queue workqueue.TypedRateLimitingInterface[string] + workersDir string // nodepath.WorkersDir(); a temp dir in tests + minAge time.Duration + maxJitter time.Duration } -func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer, ateomsDir string) *workerSandboxSyncer { +func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer, workersDir string) *workerSandboxSyncer { return &workerSandboxSyncer{ resolver: resolver, pods: pods, @@ -101,21 +101,21 @@ func newWorkerSandboxSyncer(resolver *workerSandboxResolver, pods cache.Indexer, // of seconds to minutes. queue: workqueue.NewTypedRateLimitingQueue( workqueue.NewTypedItemExponentialFailureRateLimiter[string](time.Second, 5*time.Minute)), - ateomsDir: ateomsDir, - minAge: orphanMinAge, - maxJitter: syncMaxJitter, + workersDir: workersDir, + minAge: orphanMinAge, + maxJitter: syncMaxJitter, } } // startWorkerSandboxSyncer wires a syncer to the given informers, which the // caller starts, and runs it until ctx ends. pods must hold only this node's // worker pods and must not have started yet. Worker directories live under -// ateomsDir. +// workersDir. // // The syncer never holds up atelet startup: until the informers sync, or // while a list/watch fails (e.g. Forbidden while an RBAC rollout lags the // binary), files wait and existing ones stay as they are. -func startWorkerSandboxSyncer(ctx context.Context, pods coreinformers.PodInformer, workerPools informersv1alpha1.WorkerPoolInformer, configs informersv1alpha1.SandboxConfigInformer, ateomsDir string) (*workerSandboxSyncer, error) { +func startWorkerSandboxSyncer(ctx context.Context, pods coreinformers.PodInformer, workerPools informersv1alpha1.WorkerPoolInformer, configs informersv1alpha1.SandboxConfigInformer, workersDir string) (*workerSandboxSyncer, error) { if err := pods.Informer().AddIndexers(cache.Indexers{podUIDIndex: func(obj any) ([]string, error) { pod, ok := obj.(*corev1.Pod) if !ok { @@ -129,7 +129,7 @@ func startWorkerSandboxSyncer(ctx context.Context, pods coreinformers.PodInforme workerPools: workerPools.Lister(), configs: configs.Lister(), } - s := newWorkerSandboxSyncer(resolver, pods.Informer().GetIndexer(), ateomsDir) + s := newWorkerSandboxSyncer(resolver, pods.Informer().GetIndexer(), workersDir) if err := watch(ctx, "worker pod", pods.Informer(), s.podHandler()); err != nil { return nil, err } @@ -244,10 +244,10 @@ func (s *workerSandboxSyncer) enqueuePool(pool *v1alpha1.WorkerPool, maxDelay ti // one left by a pod that went away while atelet was down is removed. A // directory younger than minAge is enqueued once it reaches that age. func (s *workerSandboxSyncer) enqueueExistingDirs(ctx context.Context) { - entries, err := os.ReadDir(s.ateomsDir) + entries, err := os.ReadDir(s.workersDir) if err != nil { if !errors.Is(err, os.ErrNotExist) { - slog.WarnContext(ctx, "Could not list worker directories", slog.String("dir", s.ateomsDir), slog.Any("err", err)) + slog.WarnContext(ctx, "Could not list worker directories", slog.String("dir", s.workersDir), slog.Any("err", err)) } return } @@ -358,5 +358,5 @@ func (s *workerSandboxSyncer) removeDir(ctx context.Context, uid string) error { } func (s *workerSandboxSyncer) podDir(uid string) string { - return filepath.Join(s.ateomsDir, uid) + return filepath.Join(s.workersDir, uid) } diff --git a/cmd/atelet/worker_sandbox_syncer_test.go b/cmd/atelet/worker_sandbox_syncer_test.go index 0832838689..10557ca6f6 100644 --- a/cmd/atelet/worker_sandbox_syncer_test.go +++ b/cmd/atelet/worker_sandbox_syncer_test.go @@ -445,7 +445,7 @@ func TestWorkerSandboxSyncerExistingDirs(t *testing.T) { entries: map[string]string{"pod-a": oldDir, "pod-b": oldDir}, want: []string{"pod-a", "pod-b"}, }, { - // Its ateom may have made it before the pod reached the cache. + // Its worker may have made it before the pod reached the cache. name: "a young directory waits for minAge", entries: map[string]string{"pod-a": oldDir, "pod-b": youngDir}, want: []string{"pod-a"}, @@ -463,10 +463,10 @@ func TestWorkerSandboxSyncerExistingDirs(t *testing.T) { // Short enough to observe a delayed enqueue. s.minAge = 50 * time.Millisecond if tt.entries == nil { - s.ateomsDir = filepath.Join(s.ateomsDir, "missing") + s.workersDir = filepath.Join(s.workersDir, "missing") } for name, kind := range tt.entries { - path := filepath.Join(s.ateomsDir, name) + path := filepath.Join(s.workersDir, name) var err error switch kind { case file: diff --git a/cmd/atelet/ateomdialer_test.go b/cmd/atelet/workerdialer_test.go similarity index 63% rename from cmd/atelet/ateomdialer_test.go rename to cmd/atelet/workerdialer_test.go index 26a52d5e31..d86c186fd3 100644 --- a/cmd/atelet/ateomdialer_test.go +++ b/cmd/atelet/workerdialer_test.go @@ -21,19 +21,19 @@ import ( "google.golang.org/grpc/connectivity" ) -// TestAteomDialerClosesEvictedConns pins the eviction behavior: a connection +// TestWorkerDialerClosesEvictedConns pins the eviction behavior: a connection // pushed out of the cache must be closed, not forgotten. A forgotten conn // keeps its goroutines and buffers for the life of the process, which turns // worker pod churn past the cache size into an unbounded leak. -func TestAteomDialerClosesEvictedConns(t *testing.T) { - d := newAteomDialer(1) +func TestWorkerDialerClosesEvictedConns(t *testing.T) { + d := newWorkerDialer(1) - first, err := d.DialAteomPod(context.Background(), "pod-a") + first, err := d.DialWorkerPod(context.Background(), "pod-a") if err != nil { - t.Fatalf("DialAteomPod(pod-a) error = %v", err) + t.Fatalf("DialWorkerPod(pod-a) error = %v", err) } - if _, err := d.DialAteomPod(context.Background(), "pod-b"); err != nil { - t.Fatalf("DialAteomPod(pod-b) error = %v", err) + if _, err := d.DialWorkerPod(context.Background(), "pod-b"); err != nil { + t.Fatalf("DialWorkerPod(pod-b) error = %v", err) } if got := first.GetState(); got != connectivity.Shutdown { @@ -41,18 +41,18 @@ func TestAteomDialerClosesEvictedConns(t *testing.T) { } } -// TestAteomDialerReusesCachedConn pins the cache hit path: dialing the same +// TestWorkerDialerReusesCachedConn pins the cache hit path: dialing the same // pod twice returns the same live connection. -func TestAteomDialerReusesCachedConn(t *testing.T) { - d := newAteomDialer(2) +func TestWorkerDialerReusesCachedConn(t *testing.T) { + d := newWorkerDialer(2) - first, err := d.DialAteomPod(context.Background(), "pod-a") + first, err := d.DialWorkerPod(context.Background(), "pod-a") if err != nil { - t.Fatalf("DialAteomPod(pod-a) error = %v", err) + t.Fatalf("DialWorkerPod(pod-a) error = %v", err) } - second, err := d.DialAteomPod(context.Background(), "pod-a") + second, err := d.DialWorkerPod(context.Background(), "pod-a") if err != nil { - t.Fatalf("DialAteomPod(pod-a) again error = %v", err) + t.Fatalf("DialWorkerPod(pod-a) again error = %v", err) } if first != second { t.Errorf("second dial returned a different conn; want the cached one") diff --git a/cmd/atelet/ateomsupport.go b/cmd/atelet/workersupport.go similarity index 92% rename from cmd/atelet/ateomsupport.go rename to cmd/atelet/workersupport.go index ad3e8f5d3f..aa3bcad262 100644 --- a/cmd/atelet/ateomsupport.go +++ b/cmd/atelet/workersupport.go @@ -31,14 +31,14 @@ import ( "google.golang.org/grpc/status" ) -type ateomSupportServer struct { - ateletpb.UnimplementedAteomSupportServer +type workerSupportServer struct { + ateletpb.UnimplementedWorkerSupportServer workers ateapipb.WorkerServiceClient control ateapipb.ControlClient } -func (b *ateomSupportServer) MintActorCertificate(ctx context.Context, req *ateletpb.MintActorCertificateRequest) (*ateletpb.MintActorCertificateResponse, error) { - // Check which ateom is calling. +func (b *workerSupportServer) MintActorCertificate(ctx context.Context, req *ateletpb.MintActorCertificateRequest) (*ateletpb.MintActorCertificateResponse, error) { + // Check which worker is calling. _, err := authenticatedWorkerIdentity(ctx) if err != nil { return nil, err @@ -50,11 +50,11 @@ func (b *ateomSupportServer) MintActorCertificate(ctx context.Context, req *atel return nil, resources.ToGRPCStatusError(errs) } - // TODO(identity): Check that we believe that this ateom is running the + // TODO(identity): Check that we believe that this worker is running the // requested actor? ate-api-server will further check that we (the atelet) // are allowed to request a certificate for the actor. - resp, err := b.workers.MintAteomActorCertificate(ctx, &ateapipb.MintAteomActorCertificateRequest{ + resp, err := b.workers.MintWorkerActorCertificate(ctx, &ateapipb.MintWorkerActorCertificateRequest{ Actor: &ateapipb.ObjectRef{ Atespace: req.GetActorAtespace(), Name: req.GetActorName(), @@ -110,7 +110,7 @@ func verifyClientOnSameNode(node *substratex509.PodIdentity) func(tls.Connection // worker reports once, so an accepted call is the only thing that puts // capacity and runtimes on the Worker, and a Worker record the syncer has not // created yet is the ordinary reason for a first attempt to fail. -func (s *ateomSupportServer) RegisterWorker(ctx context.Context, req *ateletpb.RegisterWorkerRequest) (*ateletpb.RegisterWorkerResponse, error) { +func (s *workerSupportServer) RegisterWorker(ctx context.Context, req *ateletpb.RegisterWorkerRequest) (*ateletpb.RegisterWorkerResponse, error) { // Identity comes only from the mTLS certificate, never from the request: // a worker can report its own capacity and runtimes and no one else's. workerIdentity, err := authenticatedWorkerIdentity(ctx) @@ -191,7 +191,7 @@ func toAteAPISandboxRuntimes(in []*ateletpb.SandboxRuntime) []*ateapipb.SandboxR // here -- the actor is no longer assigned to this worker, or the suspend lost a // race to a resume, pause, or delete -- and the worker needs to tell those from // a transport failure it should retry. -func (s *ateomSupportServer) RequestActorSuspend(ctx context.Context, req *ateletpb.RequestActorSuspendRequest) (*ateletpb.RequestActorSuspendResponse, error) { +func (s *workerSupportServer) RequestActorSuspend(ctx context.Context, req *ateletpb.RequestActorSuspendRequest) (*ateletpb.RequestActorSuspendResponse, error) { // Identity comes only from the mTLS certificate, never from the request: a // worker can speak for the actors it hosts and no others. Which those are // is the control plane's to know, so it is checked there against the @@ -226,7 +226,7 @@ func (s *ateomSupportServer) RequestActorSuspend(ctx context.Context, req *atele } // GetActorEgressPolicy reads the egress policy of an actor. -func (s *ateomSupportServer) GetActorEgressPolicy( +func (s *workerSupportServer) GetActorEgressPolicy( ctx context.Context, req *ateletpb.GetActorEgressPolicyRequest, ) (*ateletpb.GetActorEgressPolicyResponse, error) { diff --git a/cmd/atelet/ateomsupport_test.go b/cmd/atelet/workersupport_test.go similarity index 95% rename from cmd/atelet/ateomsupport_test.go rename to cmd/atelet/workersupport_test.go index 124ab8bccd..2215bbb79c 100644 --- a/cmd/atelet/ateomsupport_test.go +++ b/cmd/atelet/workersupport_test.go @@ -89,8 +89,8 @@ type fakeWorkerService struct { ateapipb.WorkerServiceClient got []*ateapipb.RegisterWorkerRequest - mintGot []*ateapipb.MintAteomActorCertificateRequest - mintResp *ateapipb.MintAteomActorCertificateResponse + mintGot []*ateapipb.MintWorkerActorCertificateRequest + mintResp *ateapipb.MintWorkerActorCertificateResponse err error } @@ -102,7 +102,7 @@ func (s *fakeWorkerService) RegisterWorker(_ context.Context, in *ateapipb.Regis return &ateapipb.RegisterWorkerResponse{}, nil } -func (s *fakeWorkerService) MintAteomActorCertificate(_ context.Context, in *ateapipb.MintAteomActorCertificateRequest, _ ...grpc.CallOption) (*ateapipb.MintAteomActorCertificateResponse, error) { +func (s *fakeWorkerService) MintWorkerActorCertificate(_ context.Context, in *ateapipb.MintWorkerActorCertificateRequest, _ ...grpc.CallOption) (*ateapipb.MintWorkerActorCertificateResponse, error) { if s.err != nil { return nil, s.err } @@ -110,7 +110,7 @@ func (s *fakeWorkerService) MintAteomActorCertificate(_ context.Context, in *ate if s.mintResp != nil { return s.mintResp, nil } - return &ateapipb.MintAteomActorCertificateResponse{}, nil + return &ateapipb.MintWorkerActorCertificateResponse{}, nil } func TestRegisterWorker(t *testing.T) { @@ -220,7 +220,7 @@ func TestRegisterWorker(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { workers := &fakeWorkerService{err: tt.serviceErr} - svc := &ateomSupportServer{workers: workers} + svc := &workerSupportServer{workers: workers} ctx := workerContext(t, "pod-a") if tt.unauthenticated { ctx = context.Background() @@ -254,7 +254,7 @@ func (s *fakeSuspendService) RequestActorSuspend(_ context.Context, in *ateapipb func TestRequestActorSuspendNamesTheCallingWorker(t *testing.T) { workers := &fakeSuspendService{} - svc := &ateomSupportServer{workers: workers} + svc := &workerSupportServer{workers: workers} ctx := workerContext(t, "pod-a") if _, err := svc.RequestActorSuspend(ctx, &ateletpb.RequestActorSuspendRequest{ @@ -280,7 +280,7 @@ func TestRequestActorSuspendNamesTheCallingWorker(t *testing.T) { func TestRequestActorSuspendRequiresACertificate(t *testing.T) { workers := &fakeSuspendService{} - svc := &ateomSupportServer{workers: workers} + svc := &workerSupportServer{workers: workers} // No peer identity: there is no worker to attribute this to, and the // request names no other way to find one. @@ -301,7 +301,7 @@ func TestRequestActorSuspendRequiresACertificate(t *testing.T) { // that the control plane declined so it does not treat the actor as suspended. func TestRequestActorSuspendSurfacesRefusal(t *testing.T) { workers := &fakeSuspendService{err: status.Error(codes.FailedPrecondition, "Actor is RESUMING")} - svc := &ateomSupportServer{workers: workers} + svc := &workerSupportServer{workers: workers} _, err := svc.RequestActorSuspend(workerContext(t, "pod-a"), &ateletpb.RequestActorSuspendRequest{ ActorAtespace: "team-a", @@ -315,11 +315,11 @@ func TestRequestActorSuspendSurfacesRefusal(t *testing.T) { func TestMintActorCertificateForwardsToWorkerService(t *testing.T) { wantCerts := [][]byte{[]byte("cert-der-bytes")} workers := &fakeWorkerService{ - mintResp: &ateapipb.MintAteomActorCertificateResponse{ + mintResp: &ateapipb.MintWorkerActorCertificateResponse{ ActorCertificates: wantCerts, }, } - svc := &ateomSupportServer{workers: workers} + svc := &workerSupportServer{workers: workers} ctx := workerContext(t, "pod-a") resp, err := svc.MintActorCertificate(ctx, &ateletpb.MintActorCertificateRequest{ @@ -332,7 +332,7 @@ func TestMintActorCertificateForwardsToWorkerService(t *testing.T) { t.Fatalf("MintActorCertificate() failed: %v", err) } - want := []*ateapipb.MintAteomActorCertificateRequest{{ + want := []*ateapipb.MintWorkerActorCertificateRequest{{ Actor: &ateapipb.ObjectRef{ Atespace: "team-a", Name: "actor-1", @@ -350,7 +350,7 @@ func TestMintActorCertificateForwardsToWorkerService(t *testing.T) { func TestMintActorCertificateRequiresCertificate(t *testing.T) { workers := &fakeWorkerService{} - svc := &ateomSupportServer{workers: workers} + svc := &workerSupportServer{workers: workers} _, err := svc.MintActorCertificate(context.Background(), &ateletpb.MintActorCertificateRequest{ ActorAtespace: "team-a", @@ -445,7 +445,7 @@ func TestGetActorEgressPolicy(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { control := &fakeEgressPolicyControl{resp: tt.resp, err: tt.serviceErr} - svc := &ateomSupportServer{control: control} + svc := &workerSupportServer{control: control} ctx := workerContext(t, "pod-a") if tt.unauthenticated { ctx = context.Background() diff --git a/cmd/atenet/internal/root.go b/cmd/atenet/internal/root.go index c992844efc..d7d8046de8 100644 --- a/cmd/atenet/internal/root.go +++ b/cmd/atenet/internal/root.go @@ -20,15 +20,24 @@ import ( "github.com/agent-substrate/substrate/cmd/atenet/internal/router" "github.com/agent-substrate/substrate/cmd/atenet/internal/sdsmint" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/version" "github.com/spf13/cobra" ) +var previewFlags []string + var rootCmd = &cobra.Command{ Use: "atenet", Short: "atenet is a combined daemon for all networking functionality.", Long: `atenet is a combined daemon for all networking functionality.`, Version: version.String(), + PersistentPreRunE: func(cmd *cobra.Command, args []string) error { + if err := preview.Init(previewFlags...); err != nil { + return fmt.Errorf("invalid --preview: %w", err) + } + return nil + }, } func Execute() { @@ -39,6 +48,7 @@ func Execute() { } func init() { + rootCmd.PersistentFlags().StringSliceVar(&previewFlags, "preview", nil, "Preview gates to enable.") rootCmd.AddCommand(router.NewRouterCmd()) rootCmd.AddCommand(sdsmint.NewSdsmintCmd()) } diff --git a/cmd/atenet/internal/router/README.md b/cmd/atenet/internal/router/README.md index 717499df69..77be87f89b 100644 --- a/cmd/atenet/internal/router/README.md +++ b/cmd/atenet/internal/router/README.md @@ -95,10 +95,13 @@ the request legs match `ports` against it; a dataplane that does not share it gets no port enforcement. The CONNECT leg decides per connection: it refuses an actor without a policy -with rules, and for the rest hands the inner listener the port the actor -dialed, as `envoy.upstream.dynamic_port`, and the policy's SNI rules for that -port, as `dev.ate.policy.egress`. At the ClientHello the egress-policy module -reads those rules and picks the chain: `egress_tls_mitm` under an `https` +with rules, and for the rest computes hash of the policy SNI's rules for the +destination port, stores policy in the thread local store using hash as the +key and hands the inner listener the port the actor dialed, as +`envoy.upstream.dynamic_port` and the hash of the policy' as +`dev.ate.policy.egress.hash`. At the ClientHello the egress-policy module +reads the hash, and fetches egress policy from the thrad local store, +applies SNI rules and picks the chain: `egress_tls_mitm` under an `https` rule, `egress_passthrough` under a `tls_passthrough` rule, and `denied`, which matches no chain and closes the connection, when neither covers the SNI. The passthrough chain resolves the SNI and relays the bytes to it on the dialed diff --git a/cmd/atenet/internal/router/egress/credentials.go b/cmd/atenet/internal/router/egress/credentials.go index 996c361335..6ac51c3145 100644 --- a/cmd/atenet/internal/router/egress/credentials.go +++ b/cmd/atenet/internal/router/egress/credentials.go @@ -157,7 +157,7 @@ func (h *Handler) providerCredential(ctx context.Context, ref resources.ActorRef } } - // Atunnel connected to us with an ateom-for-actor SPIFFE ID; translate it + // Atunnel connected to us with a worker-for-actor SPIFFE ID; translate it // to a pure actor SPIFFE ID for plugins to make decisions on. resp, err := h.provider.FetchSecret(ctx, &credproviderpb.FetchSecretRequest{ Uri: uri, diff --git a/cmd/atenet/internal/router/egress/egress.go b/cmd/atenet/internal/router/egress/egress.go index 01b7fcfba3..0fcd11fd7c 100644 --- a/cmd/atenet/internal/router/egress/egress.go +++ b/cmd/atenet/internal/router/egress/egress.go @@ -60,7 +60,7 @@ const ( // value. // // This is the only channel that can carry a whole certificate to ext_proc - // so the gateway can verify the chain, key usages, and ateom SPIFFE URI. + // so the gateway can verify the chain, key usages, and worker SPIFFE URI. // // TODO(identity): Audit that this cannot be stomped by a header sent by the // actor. @@ -185,17 +185,21 @@ func (h *Handler) handleConnect(ctx context.Context, md *extproc.RequestMetadata return extproc.Result{}, err } rules := policy.SNIRules(dest.Port) + dynamicMetadata, err := connectMetadata(dest, rules) + if err != nil { + return extproc.Result{}, err + } slog.InfoContext(ctx, "egress tunnel opened: requests inside it are decided one by one", slog.Any("actor", ref), slog.String("leg", leg), slog.String("destination", md.Host), slog.Int("sniRules", len(rules))) res := allow() - res.DynamicMetadata = connectMetadata(dest, rules) + res.DynamicMetadata = dynamicMetadata return res, nil } // connectMetadata encodes the SNI rules for EgressPolicyMetadataNamespace and // the dialed port for EgressMetadataNamespace. The port is always set: without // it the passthrough chain falls back to its configured port instead of closing. -func connectMetadata(dest egresspolicy.Destination, rules []egresspolicy.SNIRule) *structpb.Struct { +func connectMetadata(dest egresspolicy.Destination, rules []egresspolicy.SNIRule) (*structpb.Struct, error) { values := make([]*structpb.Value, len(rules)) for i, rule := range rules { values[i] = structpb.NewStructValue(&structpb.Struct{Fields: map[string]*structpb.Value{ @@ -203,14 +207,21 @@ func connectMetadata(dest egresspolicy.Destination, rules []egresspolicy.SNIRule extproc.EgressSNIRuleModeKey: structpb.NewStringValue(string(rule.Mode)), }}) } + policyStruct := &structpb.Struct{Fields: map[string]*structpb.Value{ + extproc.EgressSNIRulesKey: structpb.NewListValue(&structpb.ListValue{Values: values}), + }} + policyJSON, err := policyStruct.MarshalJSON() + if err != nil { + return nil, fmt.Errorf("marshaling egress policy metadata: %w", err) + } return &structpb.Struct{Fields: map[string]*structpb.Value{ extproc.EgressMetadataNamespace: structpb.NewStructValue(&structpb.Struct{Fields: map[string]*structpb.Value{ extproc.EgressDialedPortKey: structpb.NewStringValue(strconv.Itoa(int(dest.Port))), }}), extproc.EgressPolicyMetadataNamespace: structpb.NewStructValue(&structpb.Struct{Fields: map[string]*structpb.Value{ - extproc.EgressSNIRulesKey: structpb.NewListValue(&structpb.ListValue{Values: values}), + extproc.EgressPolicyJSONKey: structpb.NewStringValue(string(policyJSON)), }}), - }} + }}, nil } // metadataAnswer is an answer in the egress metadata namespace. @@ -301,7 +312,7 @@ func (h *Handler) authenticateActorCertificate(md *extproc.RequestMetadata) (res } // verifyActorCertificate checks that chain[0] is a live, non-CA, client-auth -// ateom actor certificate issued by the actor-identity CA, and returns the +// worker actor certificate issued by the actor-identity CA, and returns the // ActorRef from its SPIFFE URI. // // The chain is verified here even though Envoy already did it at the handshake @@ -310,7 +321,7 @@ func (h *Handler) authenticateActorCertificate(md *extproc.RequestMetadata) (res // trusting a parsed-but-unverified certificate is a well-worn source of CVEs. // It also keeps the handler safe if the Envoy config is ever loosened, and costs // one signature check per CONNECT rather than per request. The IsCA, ClientAuth-EKU, -// and ateom SPIFFE URI checks below have no Envoy-side equivalent at all. +// and worker SPIFFE URI checks below have no Envoy-side equivalent at all. func (h *Handler) verifyActorCertificate(chain []*x509.Certificate) (resources.ActorRef, error) { leaf := chain[0] intermediates := x509.NewCertPool() @@ -339,12 +350,12 @@ func (h *Handler) verifyActorCertificate(chain []*x509.Certificate) (resources.A return resources.ActorRef{}, fmt.Errorf("actor certificate is not signed by the actor-identity CA: %w", err) } - // Check that this is an ateom certificate --- the SPIFFE URI should be of - // the form `spiffe://${trustdomain}/ateom-for-actor/${atespace}/${actor}`. + // Check that this is a worker certificate --- the SPIFFE URI should be of + // the form `spiffe://${trustdomain}/worker-for-actor/${atespace}/${actor}`. if len(leaf.URIs) != 1 { return resources.ActorRef{}, fmt.Errorf("actor certificate has %d URI SANs, want 1", len(leaf.URIs)) } - ref, err := resources.ActorRefFromAteomForActorSPIFFEURL(leaf.URIs[0]) + ref, err := resources.ActorRefFromWorkerForActorSPIFFEURL(leaf.URIs[0]) if err != nil { return resources.ActorRef{}, fmt.Errorf("while parsing actor from SPIFFE ID: %w", err) } diff --git a/cmd/atenet/internal/router/egress/egress_test.go b/cmd/atenet/internal/router/egress/egress_test.go index c14ea93bb7..bc6df80dbc 100644 --- a/cmd/atenet/internal/router/egress/egress_test.go +++ b/cmd/atenet/internal/router/egress/egress_test.go @@ -331,7 +331,7 @@ func wantStatus(t *testing.T, err error, want envoy_type.StatusCode) { func TestHandleRequestHeadersAllowsVerifiedActor(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) h := egressHandler(ca.roots(), runningActor(), nil) res, err := h.HandleRequestHeaders(context.Background(), egressMetadata(xfccHeader(leaf))) @@ -363,7 +363,7 @@ func dialedPortOf(res extproc.Result) string { // tls_passthrough SNI rules for the dialed port, most specific first. func TestConnectLegOpensForAnyRules(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) mitm := func(patterns ...string) []egresspolicy.SNIRule { rules := make([]egresspolicy.SNIRule, len(patterns)) @@ -433,10 +433,18 @@ func TestConnectLegOpensForAnyRules(t *testing.T) { // sniRulesOf reads the SNI rules from a CONNECT result. func sniRulesOf(t *testing.T, res extproc.Result) []egresspolicy.SNIRule { t.Helper() - policyStruct := res.DynamicMetadata.GetFields()[extproc.EgressPolicyMetadataNamespace].GetStructValue() - if policyStruct == nil { + nsStruct := res.DynamicMetadata.GetFields()[extproc.EgressPolicyMetadataNamespace].GetStructValue() + if nsStruct == nil { t.Fatalf("missing %q struct in DynamicMetadata", extproc.EgressPolicyMetadataNamespace) } + policyJSON := nsStruct.GetFields()[extproc.EgressPolicyJSONKey].GetStringValue() + if policyJSON == "" { + t.Fatalf("missing %q string in %q DynamicMetadata", extproc.EgressPolicyJSONKey, extproc.EgressPolicyMetadataNamespace) + } + var policyStruct structpb.Struct + if err := policyStruct.UnmarshalJSON([]byte(policyJSON)); err != nil { + t.Fatalf("unmarshaling %q in %q DynamicMetadata: %v", extproc.EgressPolicyJSONKey, extproc.EgressPolicyMetadataNamespace, err) + } listVal := policyStruct.GetFields()[extproc.EgressSNIRulesKey].GetListValue() if listVal == nil { t.Fatalf("missing %q list in %q DynamicMetadata", extproc.EgressSNIRulesKey, extproc.EgressPolicyMetadataNamespace) @@ -456,7 +464,7 @@ func sniRulesOf(t *testing.T, res extproc.Result) []egresspolicy.SNIRule { // and an actor without a policy is refused. func TestConnectLegWithoutRequestLegs(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) certificate := string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: leaf.Raw})) h := New(&egressMockClient{actor: runningActor(), policy: httpPolicy("api.example.com")}, ca.roots(), 0, nil, "") @@ -474,7 +482,7 @@ func TestConnectLegWithoutRequestLegs(t *testing.T) { func TestHandleRequestHeadersAllowsAgentgatewayCertificateAttribute(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) h := egressHandler(ca.roots(), runningActor(), nil) certificate := pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: leaf.Raw}) @@ -502,14 +510,14 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "signed by an unknown CA", xfcc: func(t *testing.T) string { - return xfccHeader(otherCA.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{})) + return xfccHeader(otherCA.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{})) }, want: envoy_type.StatusCode_Forbidden, }, { name: "expired", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.NotBefore = time.Now().Add(-2 * time.Hour) c.NotAfter = time.Now().Add(-time.Hour) }})) @@ -519,7 +527,7 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "not yet valid", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.NotBefore = time.Now().Add(time.Hour) c.NotAfter = time.Now().Add(2 * time.Hour) }})) @@ -529,7 +537,7 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "no ClientAuth EKU", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.ExtKeyUsage = []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth} }})) }, @@ -538,7 +546,7 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "is a CA certificate", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.IsCA = true c.KeyUsage |= x509.KeyUsageCertSign }})) @@ -548,7 +556,7 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "no URI SANs", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = nil }})) }, @@ -557,11 +565,11 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "multiple URI SANs", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = append(c.URIs, &url.URL{ Scheme: "spiffe", Host: "substrate-actor.local", - Path: path.Join("ateom", "actor", testEgressAtespace, "other-actor"), + Path: path.Join("worker", "actor", testEgressAtespace, "other-actor"), }) }})) }, @@ -570,11 +578,11 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "non-spiffe URI scheme", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = []*url.URL{{ Scheme: "https", Host: "substrate-actor.local", - Path: path.Join("ateom", "actor", testEgressAtespace, testEgressActor), + Path: path.Join("worker", "actor", testEgressAtespace, testEgressActor), }} }})) }, @@ -583,20 +591,20 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "empty trust domain", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = []*url.URL{{ Scheme: "spiffe", Host: "", - Path: "/" + path.Join("ateom", "actor", testEgressAtespace, testEgressActor), + Path: "/" + path.Join("worker", "actor", testEgressAtespace, testEgressActor), }} }})) }, want: envoy_type.StatusCode_Forbidden, }, { - name: "non-ateom actor SPIFFE URI", + name: "non-worker actor SPIFFE URI", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = []*url.URL{{ Scheme: "spiffe", Host: "substrate-actor.local", @@ -609,11 +617,11 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "malformed SPIFFE URI path (too few segments)", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = []*url.URL{{ Scheme: "spiffe", Host: "substrate-actor.local", - Path: path.Join("ateom", "actor", testEgressAtespace), + Path: path.Join("worker", "actor", testEgressAtespace), }} }})) }, @@ -622,11 +630,11 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "malformed SPIFFE URI path (extra segment)", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = []*url.URL{{ Scheme: "spiffe", Host: "substrate-actor.local", - Path: path.Join("ateom", "actor", testEgressAtespace, testEgressActor, "extra"), + Path: path.Join("worker", "actor", testEgressAtespace, testEgressActor, "extra"), }} }})) }, @@ -635,11 +643,11 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "invalid atespace resource name in SPIFFE URI", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = []*url.URL{{ Scheme: "spiffe", Host: "substrate-actor.local", - Path: path.Join("ateom", "actor", "INVALID_ATESPACE", testEgressActor), + Path: path.Join("worker", "actor", "INVALID_ATESPACE", testEgressActor), }} }})) }, @@ -648,11 +656,11 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "invalid actor resource name in SPIFFE URI", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = []*url.URL{{ Scheme: "spiffe", Host: "substrate-actor.local", - Path: path.Join("ateom", "actor", testEgressAtespace, "INVALID_ACTOR"), + Path: path.Join("worker", "actor", testEgressAtespace, "INVALID_ACTOR"), }} }})) }, @@ -663,7 +671,7 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { // elements means we can no longer tell which one is our peer. name: "two XFCC elements", xfcc: func(t *testing.T) string { - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) return xfccHeader(leaf) + "," + xfccHeader(leaf) }, want: envoy_type.StatusCode_Forbidden, @@ -678,7 +686,7 @@ func TestHandleRequestHeadersRejectsBadCertificates(t *testing.T) { { name: "no URI SAN at all", xfcc: func(t *testing.T) string { - return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = nil }})) + return xfccHeader(ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{mutate: func(c *x509.Certificate) { c.URIs = nil }})) }, want: envoy_type.StatusCode_Forbidden, }, @@ -738,7 +746,7 @@ func TestHandleRequestHeadersAuthorization(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { h := egressHandler(ca.roots(), tc.actor, tc.err) - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) _, err := h.HandleRequestHeaders(context.Background(), egressMetadata(xfccHeader(leaf))) wantStatus(t, err, tc.want) }) @@ -749,7 +757,7 @@ func TestHandleRequestHeadersAuthorization(t *testing.T) { // reaches it, it must fail closed rather than tunnel unauthenticated traffic. func TestHandleRequestHeadersWithoutConfiguredCA(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) h := egressHandler(nil, runningActor(), nil) _, err := h.HandleRequestHeaders(context.Background(), egressMetadata(xfccHeader(leaf))) @@ -760,7 +768,7 @@ func TestHandleRequestHeadersWithoutConfiguredCA(t *testing.T) { // not a tunnel this gateway can carry, and is refused where it can still say so. func TestHandleRequestHeadersRejectsNonAddressAuthority(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) h := egressHandler(ca.roots(), runningActor(), nil) for _, authority := range []string{"example.com:443", "93.184.216.34", ""} { @@ -774,7 +782,7 @@ func TestHandleRequestHeadersRejectsNonAddressAuthority(t *testing.T) { func TestHandleRequestHeadersRejectsNonConnect(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) h := egressHandler(ca.roots(), runningActor(), nil) md := egressMetadata(xfccHeader(leaf)) @@ -792,7 +800,7 @@ func TestParseXFCCChainPreservesPlusInPEM(t *testing.T) { // Serials differ per certificate, so mint until one encodes with a '+'. var leaf *x509.Certificate for i := 0; i < 50; i++ { - candidate := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + candidate := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) if strings.Contains(string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: candidate.Raw})), "+") { leaf = candidate break @@ -813,7 +821,7 @@ func TestParseXFCCChainPreservesPlusInPEM(t *testing.T) { func TestParseXFCCChainIncludesIntermediates(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/foo/bar", actorCertOptions{}) chain, err := parseXFCCChain(xfccHeader(leaf, ca.cert)) if err != nil { diff --git a/cmd/atenet/internal/router/egress/request.go b/cmd/atenet/internal/router/egress/request.go index 2c34fe2bb1..6313de4c58 100644 --- a/cmd/atenet/internal/router/egress/request.go +++ b/cmd/atenet/internal/router/egress/request.go @@ -197,5 +197,5 @@ func actorFromFilterState(md *extproc.RequestMetadata) (resources.ActorRef, erro if id == "" { return resources.ActorRef{}, errors.New("no actor identity in filter state") } - return resources.ActorRefFromAteomForActorSPIFFEID(id) + return resources.ActorRefFromWorkerForActorSPIFFEID(id) } diff --git a/cmd/atenet/internal/router/egress/request_test.go b/cmd/atenet/internal/router/egress/request_test.go index 505472f4c9..67589c16b6 100644 --- a/cmd/atenet/internal/router/egress/request_test.go +++ b/cmd/atenet/internal/router/egress/request_test.go @@ -31,7 +31,7 @@ import ( ) // testActorSPIFFEID is the identity filter state the CONNECT chain shares. -const testActorSPIFFEID = "spiffe://substrate-actor.local/ateom-for-actor/default/my-actor" +const testActorSPIFFEID = "spiffe://substrate-actor.local/worker-for-actor/default/my-actor" // testDialed is the IP:port the test actor's kernel dialed for a callout on // leg: the default port of the protocol that leg carries. @@ -410,7 +410,7 @@ func TestRequestLegPolicyLookup(t *testing.T) { // could be allowed through it, and warms the cache for the requests inside. func TestConnectLegRequiresAPolicy(t *testing.T) { ca := newTestCA(t, "actor-identity-ca") - leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/ateom-for-actor/default/my-actor", actorCertOptions{}) + leaf := ca.issueActorCert(t, "spiffe://substrate-actor.local/worker-for-actor/default/my-actor", actorCertOptions{}) tests := []struct { name string diff --git a/cmd/atenet/internal/router/egresspolicymanifest_test.go b/cmd/atenet/internal/router/egresspolicymanifest_test.go index ed2708ac28..5dc34d7c55 100644 --- a/cmd/atenet/internal/router/egresspolicymanifest_test.go +++ b/cmd/atenet/internal/router/egresspolicymanifest_test.go @@ -696,7 +696,7 @@ func TestEgressManifestsInnerListenerSelectsOnTheModuleVerdict(t *testing.T) { // The outer ext_proc must accept dev.ate.policy.egress, and the outer chain // must copy it after ext_proc into shared filter state for the module. -func TestEgressManifestsConnectLegHandsTheSNIRulesToTheInnerListener(t *testing.T) { +func TestEgressManifestsConnectLegHandsTheSNIRulesHashToTheInnerListener(t *testing.T) { tree := bootstrapTree(t, egressManifest) outer := outerChain(t, tree) cfg, extProcAt, filters := extProcOf(outer) @@ -708,9 +708,9 @@ func TestEgressManifestsConnectLegHandsTheSNIRulesToTheInnerListener(t *testing. t.Errorf("the CONNECT leg's ext_proc does not admit dynamic metadata in %q; the SNI rules would be dropped and every ClientHello denied", extproc.EgressPolicyMetadataNamespace) } - writers := filterStateWriters(tree, extproc.EgressPolicyMetadataNamespace) + writers := filterStateWriters(tree, extproc.EgressPolicyHashFilterStateKey) if len(writers) != 1 { - t.Fatalf("%s is set by %d filters, want exactly the CONNECT leg's copy of its answer", extproc.EgressPolicyMetadataNamespace, len(writers)) + t.Fatalf("%s is set by %d filters, want exactly the CONNECT leg's copy of its answer", extproc.EgressPolicyHashFilterStateKey, len(writers)) } entry := writers[0] var gotKeys []string @@ -724,7 +724,7 @@ func TestEgressManifestsConnectLegHandsTheSNIRulesToTheInnerListener(t *testing. key := str(v, "object_key") gotKeys = append(gotKeys, key) actionsByKey[key] = v - if key == extproc.EgressPolicyMetadataNamespace { + if key == extproc.EgressPolicyHashFilterStateKey { at = i } if got := str(v, "shared_with_upstream"); got != "ONCE" { @@ -737,7 +737,7 @@ func TestEgressManifestsConnectLegHandsTheSNIRulesToTheInnerListener(t *testing. extproc.ActorIdentityFilterStateKey, extproc.ConnectAuthorityFilterStateKey, extproc.UpstreamDynamicPortFilterStateKey, - extproc.EgressPolicyMetadataNamespace, + extproc.EgressPolicyHashFilterStateKey, } if !slices.Equal(gotKeys, wantKeys) { t.Errorf("outer chain set_filter_state actions write keys %v, want %v", gotKeys, wantKeys) @@ -748,7 +748,7 @@ func TestEgressManifestsConnectLegHandsTheSNIRulesToTheInnerListener(t *testing. for _, key := range []string{ extproc.ActorIdentityFilterStateKey, extproc.ConnectAuthorityFilterStateKey, - extproc.EgressPolicyMetadataNamespace, + extproc.EgressPolicyHashFilterStateKey, } { // Make sure factory creates hashable filter state objects, so it impacts pool selection. if got := str(actionsByKey[key], "factory_key"); got != "envoy.hashable_string" { @@ -762,8 +762,8 @@ func TestEgressManifestsConnectLegHandsTheSNIRulesToTheInnerListener(t *testing. t.Errorf("%s is set at http_filters[%d], before ext_proc at [%d]; the metadata it reads does not exist yet", extproc.EgressPolicyMetadataNamespace, at, extProcAt) } format := child(entry, "format_string") - if got := str(child(format, "text_format_source"), "inline_string"); got != extproc.EgressPolicyMetadataFormat { - t.Errorf("%s is set from %q, want %q", extproc.EgressPolicyMetadataNamespace, got, extproc.EgressPolicyMetadataFormat) + if got := str(child(format, "text_format_source"), "inline_string"); got != extproc.EgressPolicyHashFormat { + t.Errorf("%s is set from %q, want %q", extproc.EgressPolicyHashFilterStateKey, got, extproc.EgressPolicyHashFormat) } if skip, _ := entry["skip_if_empty"].(bool); !skip { t.Errorf("%s is not skip_if_empty; an absent answer would be written as unparseable JSON", extproc.EgressPolicyMetadataNamespace) diff --git a/cmd/atenet/internal/router/extproc/attributes.go b/cmd/atenet/internal/router/extproc/attributes.go index e5721d2a5a..2d0a5a717e 100644 --- a/cmd/atenet/internal/router/extproc/attributes.go +++ b/cmd/atenet/internal/router/extproc/attributes.go @@ -58,11 +58,16 @@ const ( // actor dialed, set on every allowed CONNECT. The outer chain copies it // into UpstreamDynamicPortFilterStateKey for the passthrough chain. EgressDialedPortKey = "dialed_port" - // EgressPolicyMetadataNamespace holds the SNI rules returned on CONNECT. - // The outer chain copies it as JSON into filter state of the same name for - // the egress-policy module: {"rules": [{"pattern": ..., "mode": ...}]}, + // EgressPolicyMetadataNamespace holds the SNI rules returned on CONNECT, as + // JSON under EgressPolicyJSONKey: {"rules": [{"pattern": ..., "mode": ...}]}, // most specific first. EgressPolicyMetadataNamespace = "dev.ate.policy.egress" + // EgressPolicyHashFilterStateKey holds the hash the inner listener uses to + // find those rules in the egress-policy module's per-thread store. + EgressPolicyHashFilterStateKey = "dev.ate.policy.egress.hash" + // EgressPolicyJSONKey, under EgressPolicyMetadataNamespace, holds the + // JSON-serialized SNI rules. + EgressPolicyJSONKey = "policy_json" // EgressSNIRulesKey, under EgressPolicyMetadataNamespace, is the ordered // list of rules; EgressSNIRulePatternKey and EgressSNIRuleModeKey are the // fields of each. @@ -139,8 +144,9 @@ const EgressDialHostFormat = "%DYNAMIC_METADATA(" + EgressMetadataNamespace + ": // Host wins. The request chains set it from EgressDialHostKey. const UpstreamDynamicHostFilterStateKey = "envoy.upstream.dynamic_host" -// EgressPolicyMetadataFormat renders EgressPolicyMetadataNamespace as JSON. -const EgressPolicyMetadataFormat = "%DYNAMIC_METADATA(" + EgressPolicyMetadataNamespace + ")%" +// EgressPolicyHashFormat is the set_filter_state format string that +// writes egress policy hash. +const EgressPolicyHashFormat = "%FILTER_STATE(" + EgressPolicyHashFilterStateKey + ".not_shared:PLAIN)%" // OriginalDstFilterStateKey is Envoy's filter-state key for the address an // ORIGINAL_DST cluster dials. The ingress router sets it for actor tunnels. diff --git a/cmd/atenet/internal/router/ingressauthmanifest_test.go b/cmd/atenet/internal/router/ingressauthmanifest_test.go new file mode 100644 index 0000000000..caf4051c03 --- /dev/null +++ b/cmd/atenet/internal/router/ingressauthmanifest_test.go @@ -0,0 +1,162 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package router + +import ( + "os" + "slices" + "strings" + "testing" + + "github.com/spf13/pflag" + appsv1 "k8s.io/api/apps/v1" + "sigs.k8s.io/kustomize/api/krusty" + "sigs.k8s.io/kustomize/kyaml/filesys" + "sigs.k8s.io/yaml" +) + +const routerStaticMTLSComponentPath = "../../../../manifests/ate-install/components/router-static-mtls/kustomization.yaml" + +// TestRouterIngressAuthManifestsAreValid renders the router manifest as +// ate-setup installs it, alone and with the component it layers on under +// --ingress-auth-mode=static-mtls, and checks the router would accept the +// resulting flags. The component appends to an argument list it does not own, +// and the base manifest's TLS listeners need the client auth settings, so a +// new listener in the manifest, or a new startup check here, would otherwise +// surface only as a router that refuses to start in the e2e cluster. +func TestRouterIngressAuthManifestsAreValid(t *testing.T) { + for _, tc := range []struct { + name string + componentPath string + wantMode IngressAuthMode + }{ + {name: "base", wantMode: IngressAuthDeprecatedInsecure}, + {name: "router-static-mtls", componentPath: routerStaticMTLSComponentPath, wantMode: IngressAuthStaticMTLS}, + } { + t.Run(tc.name, func(t *testing.T) { + args := renderedRouterArgs(t, tc.componentPath) + + var cfg routerConfig + fs := pflag.NewFlagSet("router", pflag.ContinueOnError) + bindRouterFlags(fs, &cfg) + if err := fs.Parse(args); err != nil { + t.Fatalf("the rendered atenet-router flags are not ones the binary accepts: %v", err) + } + + if got := cfg.IngressAuth.mode(); got != tc.wantMode { + t.Fatalf("--ingress-auth-mode = %q, want %q", got, tc.wantMode) + } + if !cfg.tlsIngressEnabled() { + t.Fatalf("the rendered flags enable no TLS ingress listener, so client auth is never exercised") + } + if err := cfg.validate(); err != nil { + t.Fatalf("the router would refuse the rendered flags: %v", err) + } + // The router reads the CA file, but Envoy is what enforces it, so + // it has to be mounted in the envoy container at the same path. + envoyMounts := renderedEnvoyMountPaths(t, tc.componentPath) + if !slices.ContainsFunc(envoyMounts, func(dir string) bool { + return strings.HasPrefix(cfg.IngressAuth.ClientCAFile, dir+"/") + }) { + t.Errorf("--ingress-client-ca-file=%s is not under any of the envoy container's mounts %v", cfg.IngressAuth.ClientCAFile, envoyMounts) + } + }) + } +} + +// renderedRouterDeployment builds the router manifest with the component at +// componentPath layered on, the way ate-setup composes them, and returns the +// atenet-router Deployment. An empty componentPath renders the manifest alone. +func renderedRouterDeployment(t *testing.T, componentPath string) appsv1.Deployment { + t.Helper() + manifest, err := os.ReadFile(routerManifestPath) + if err != nil { + t.Fatalf("reading %s: %v", routerManifestPath, err) + } + files := map[string][]byte{ + "/app/atenet-router.yaml": manifest, + "/app/kustomization.yaml": []byte("resources:\n- atenet-router.yaml\n"), + } + if componentPath != "" { + component, err := os.ReadFile(componentPath) + if err != nil { + t.Fatalf("reading %s: %v", componentPath, err) + } + files["/app/component/kustomization.yaml"] = component + files["/app/kustomization.yaml"] = []byte("resources:\n- atenet-router.yaml\ncomponents:\n- component\n") + } + + fsys := filesys.MakeFsInMemory() + for path, content := range files { + if err := fsys.WriteFile(path, content); err != nil { + t.Fatalf("staging %s: %v", path, err) + } + } + resMap, err := krusty.MakeKustomizer(krusty.MakeDefaultOptions()).Run(fsys, "/app") + if err != nil { + t.Fatalf("rendering the router manifest with %s: %v", componentPath, err) + } + for _, res := range resMap.Resources() { + if res.GetKind() != "Deployment" || res.GetName() != "atenet-router" { + continue + } + raw, err := res.AsYAML() + if err != nil { + t.Fatalf("serializing the rendered atenet-router Deployment: %v", err) + } + var deployment appsv1.Deployment + if err := yaml.Unmarshal(raw, &deployment); err != nil { + t.Fatalf("decoding the rendered atenet-router Deployment: %v", err) + } + return deployment + } + t.Fatalf("rendering with %s produced no atenet-router Deployment", componentPath) + return appsv1.Deployment{} +} + +// renderedRouterArgs returns the atenet-router container's flags, without +// the subcommand, from the manifest rendered with componentPath. +func renderedRouterArgs(t *testing.T, componentPath string) []string { + t.Helper() + for _, c := range renderedRouterDeployment(t, componentPath).Spec.Template.Spec.Containers { + if c.Name != "atenet-router" { + continue + } + if len(c.Args) == 0 || c.Args[0] != "router" { + t.Fatalf("the rendered atenet-router container's args are %v; they should invoke the router subcommand", c.Args) + } + return c.Args[1:] + } + t.Fatal("the rendered atenet-router Deployment has no container named atenet-router") + return nil +} + +// renderedEnvoyMountPaths returns the envoy container's volume mount paths +// from the manifest rendered with componentPath. +func renderedEnvoyMountPaths(t *testing.T, componentPath string) []string { + t.Helper() + for _, c := range renderedRouterDeployment(t, componentPath).Spec.Template.Spec.Containers { + if c.Name != "envoy" { + continue + } + var paths []string + for _, m := range c.VolumeMounts { + paths = append(paths, m.MountPath) + } + return paths + } + t.Fatal("the rendered atenet-router Deployment has no container named envoy") + return nil +} diff --git a/cmd/atenet/internal/router/router.go b/cmd/atenet/internal/router/router.go index 7afad07a3d..724ab979f9 100644 --- a/cmd/atenet/internal/router/router.go +++ b/cmd/atenet/internal/router/router.go @@ -40,6 +40,7 @@ import ( "github.com/agent-substrate/substrate/cmd/atenet/internal/router/extproc" "github.com/agent-substrate/substrate/cmd/atenet/internal/router/ingress" "github.com/agent-substrate/substrate/internal/ateapiauth" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/version" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -171,6 +172,9 @@ func (s *RouterServer) Run(ctx context.Context) error { return fmt.Errorf("building ateapi dial options: %w", err) } dialOpts = append(dialOpts, grpc.WithStatsHandler(otelgrpc.NewClientHandler())) + // atenet acts for the system, not a user, so it always opts in to the + // preview features the server enables. + dialOpts = append(dialOpts, preview.DialOptionsWithOptIn()...) conn, err := grpc.NewClient( s.cfg.AteapiAddr, dialOpts..., diff --git a/cmd/ateom-gvisor/internal/ateom/stopwatch.go b/cmd/ateom-gvisor/internal/ateom/stopwatch.go deleted file mode 100644 index ed0aeff8bc..0000000000 --- a/cmd/ateom-gvisor/internal/ateom/stopwatch.go +++ /dev/null @@ -1,64 +0,0 @@ -// Copyright 2026 Google LLC -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package ateom - -import ( - "log/slog" - "time" -) - -type Lap struct { - Name string - Time time.Time - StepDuration time.Duration - TotalDuration time.Duration -} - -type Stopwatch struct { - Laps []Lap -} - -var _ slog.LogValuer = (*Stopwatch)(nil) - -func (s *Stopwatch) Click(name string) { - now := time.Now() - lap := Lap{ - Name: name, - Time: now, - } - if len(s.Laps) > 0 { - lap.StepDuration = now.Sub(s.Laps[len(s.Laps)-1].Time) - lap.TotalDuration = now.Sub(s.Laps[0].Time) - } - s.Laps = append(s.Laps, lap) -} - -type FormattedLap struct { - Name string - Step string - Total string -} - -func (s *Stopwatch) LogValue() slog.Value { - formatted := []FormattedLap{} - for _, l := range s.Laps { - formatted = append(formatted, FormattedLap{ - Name: l.Name, - Step: l.StepDuration.String(), - Total: l.TotalDuration.String(), - }) - } - return slog.AnyValue(formatted) -} diff --git a/cmd/ateom-gvisor/actorpath.go b/cmd/ateworker-gvisor/actorpath.go similarity index 78% rename from cmd/ateom-gvisor/actorpath.go rename to cmd/ateworker-gvisor/actorpath.go index b2855a9a13..eef2bd318e 100644 --- a/cmd/ateom-gvisor/actorpath.go +++ b/cmd/ateworker-gvisor/actorpath.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -// The paths ateom-gvisor derives from the ActorDirs atelet sends. +// The paths ateworker-gvisor derives from the ActorDirs atelet sends. package main @@ -24,38 +24,38 @@ import ( "path/filepath" "github.com/agent-substrate/substrate/internal/imagecache" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // ociBundlePath is the container's OCI bundle. -func ociBundlePath(actorDirs *ateompb.ActorDirs, containerName string) string { +func ociBundlePath(actorDirs *ateworkerpb.ActorDirs, containerName string) string { return filepath.Join(actorDirs.GetOciBundleDir(), containerName) } // runscStateDir is runsc --root for the actor's sandbox. -func runscStateDir(actorDirs *ateompb.ActorDirs) string { +func runscStateDir(actorDirs *ateworkerpb.ActorDirs) string { return filepath.Join(actorDirs.GetRootDir(), "runsc-state") } // pidFileDir is where runsc writes .pid. -func pidFileDir(actorDirs *ateompb.ActorDirs) string { +func pidFileDir(actorDirs *ateworkerpb.ActorDirs) string { return filepath.Join(actorDirs.GetRootDir(), "pidfiles") } // pidFilePath is where runsc writes the container's pid. -func pidFilePath(actorDirs *ateompb.ActorDirs, containerName string) string { +func pidFilePath(actorDirs *ateworkerpb.ActorDirs, containerName string) string { return filepath.Join(pidFileDir(actorDirs), containerName+".pid") } // resolvConfPath is the resolver bind source outside the actor's rootfs. -func resolvConfPath(actorDirs *ateompb.ActorDirs) string { +func resolvConfPath(actorDirs *ateworkerpb.ActorDirs) string { return filepath.Join(actorDirs.GetRootDir(), "resolv.conf") } // resetRunscStateAndPidFileDirs empties both directories for a new activation. // runsc can leave mounts behind in its state directory (its null-netns), which // must be detached in this mount namespace before they can be removed. -func resetRunscStateAndPidFileDirs(actorDirs *ateompb.ActorDirs) error { +func resetRunscStateAndPidFileDirs(actorDirs *ateworkerpb.ActorDirs) error { for _, dir := range []string{runscStateDir(actorDirs), pidFileDir(actorDirs)} { if err := imagecache.UnmountAllUnder(dir); err != nil { return fmt.Errorf("while unmounting under %q: %w", dir, err) diff --git a/cmd/ateom-gvisor/actorpath_test.go b/cmd/ateworker-gvisor/actorpath_test.go similarity index 92% rename from cmd/ateom-gvisor/actorpath_test.go rename to cmd/ateworker-gvisor/actorpath_test.go index 8d24baf39a..dba4708c1d 100644 --- a/cmd/ateom-gvisor/actorpath_test.go +++ b/cmd/ateworker-gvisor/actorpath_test.go @@ -23,12 +23,12 @@ import ( "golang.org/x/sys/unix" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/roottest" ) func TestResetRunscStateAndPidFileDirs(t *testing.T) { - actorDirs := &ateompb.ActorDirs{RootDir: t.TempDir()} + actorDirs := &ateworkerpb.ActorDirs{RootDir: t.TempDir()} if err := os.MkdirAll(filepath.Join(runscStateDir(actorDirs), "stale"), 0o700); err != nil { t.Fatal(err) } @@ -46,7 +46,7 @@ func TestResetRunscStateAndPidFileDirs(t *testing.T) { // until it is detached. func TestResetRunscStateAndPidFileDirs_DetachesMounts(t *testing.T) { roottest.Require(t, "mount/unmount") - actorDirs := &ateompb.ActorDirs{RootDir: t.TempDir()} + actorDirs := &ateworkerpb.ActorDirs{RootDir: t.TempDir()} if err := os.MkdirAll(runscStateDir(actorDirs), 0o700); err != nil { t.Fatal(err) } diff --git a/cmd/ateom-gvisor/cleanup_test.go b/cmd/ateworker-gvisor/cleanup_test.go similarity index 96% rename from cmd/ateom-gvisor/cleanup_test.go rename to cmd/ateworker-gvisor/cleanup_test.go index da64818302..abbb4667f1 100644 --- a/cmd/ateom-gvisor/cleanup_test.go +++ b/cmd/ateworker-gvisor/cleanup_test.go @@ -23,7 +23,7 @@ import ( "slices" "testing" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // fakeRunsc stands in for *runsc. containers is the set runsc has a record of; @@ -80,7 +80,7 @@ func assertCalls(t *testing.T, f *fakeRunsc, want ...string) { } } -var appContainers = []*ateompb.Container{{Name: "app"}} +var appContainers = []*ateworkerpb.Container{{Name: "app"}} func TestCleanupContainers_DeletesEveryContainerRunscKnows(t *testing.T) { f := newFakeRunsc("app", "_pause") diff --git a/cmd/ateom-gvisor/durable.go b/cmd/ateworker-gvisor/durable.go similarity index 94% rename from cmd/ateom-gvisor/durable.go rename to cmd/ateworker-gvisor/durable.go index 0058a416b0..9ae6ce0338 100644 --- a/cmd/ateom-gvisor/durable.go +++ b/cmd/ateworker-gvisor/durable.go @@ -23,7 +23,7 @@ import ( "os" "path/filepath" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" ) @@ -34,7 +34,7 @@ const fsCheckpointManifest = "fscheckpoint.pb" // across containers. Every mount is emitted even when a volume is shared across // multiple mounts: runsc creates one shared tmpfs under whichever mount is // set up first and deduplicates by vfs.Filesystem when saving. -func durableFSCheckpointPaths(containers []*ateompb.Container) []string { +func durableFSCheckpointPaths(containers []*ateworkerpb.Container) []string { var paths []string for _, c := range containers { for _, m := range c.GetDurableDirVolumeMounts() { diff --git a/cmd/ateom-gvisor/durable_test.go b/cmd/ateworker-gvisor/durable_test.go similarity index 93% rename from cmd/ateom-gvisor/durable_test.go rename to cmd/ateworker-gvisor/durable_test.go index 33e14c56b5..1806bc7245 100644 --- a/cmd/ateom-gvisor/durable_test.go +++ b/cmd/ateworker-gvisor/durable_test.go @@ -22,21 +22,21 @@ import ( "slices" "testing" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) func TestDurableFSCheckpointPaths(t *testing.T) { - containers := []*ateompb.Container{ + containers := []*ateworkerpb.Container{ { Name: "app", - DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "data", MountPath: "/var/data"}, {VolumeName: "cache", MountPath: "/var/cache"}, }, }, { Name: "sidecar", - DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "logs", MountPath: "/var/log/app"}, }, }, diff --git a/cmd/ateom-gvisor/fidelity_test.go b/cmd/ateworker-gvisor/fidelity_test.go similarity index 73% rename from cmd/ateom-gvisor/fidelity_test.go rename to cmd/ateworker-gvisor/fidelity_test.go index 6e24890f4b..e058e45966 100644 --- a/cmd/ateom-gvisor/fidelity_test.go +++ b/cmd/ateworker-gvisor/fidelity_test.go @@ -24,7 +24,7 @@ import ( "google.golang.org/grpc/codes" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // TestValidateFidelity pins what this runtime serves. gVisor keeps rootfs @@ -33,14 +33,14 @@ import ( func TestValidateFidelity(t *testing.T) { for _, tc := range []struct { name string - fidelity ateompb.SnapshotFidelity + fidelity ateworkerpb.SnapshotFidelity wantErr bool }{ - {"volumes", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, false}, - {"memory", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, false}, - {"rootfs is not served", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, true}, - {"unspecified", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, true}, - {"outside the enum", ateompb.SnapshotFidelity(99), true}, + {"volumes", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, false}, + {"memory", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, false}, + {"rootfs is not served", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, true}, + {"unspecified", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, true}, + {"outside the enum", ateworkerpb.SnapshotFidelity(99), true}, } { t.Run(tc.name, func(t *testing.T) { err := validateFidelity(tc.fidelity) @@ -52,7 +52,7 @@ func TestValidateFidelity(t *testing.T) { } }) } - err := validateFidelity(ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS) + err := validateFidelity(ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS) for _, want := range []string{"SNAPSHOT_FIDELITY_VOLUMES", "SNAPSHOT_FIDELITY_MEMORY"} { if !strings.Contains(err.Error(), want) { t.Errorf("rootfs rejection %q does not list %s as supported", err, want) @@ -64,8 +64,8 @@ func TestValidateFidelity(t *testing.T) { // or restore fails validation up front, before the actor lock or any file // system work, so a misrouted request cannot leave the actor half-handled. func TestRPCsRejectRootfsBeforeTouchingTheSandbox(t *testing.T) { - s := &AteomService{} - dirs := &ateompb.ActorDirs{ + s := &AteWorkerService{} + dirs := &ateworkerpb.ActorDirs{ RootDir: "/node/actors/actor-a", OciBundleDir: "/node/actors/actor-a/bundle", CheckpointDir: "/node/actors/actor-a/checkpoint-state", @@ -74,13 +74,13 @@ func TestRPCsRejectRootfsBeforeTouchingTheSandbox(t *testing.T) { SystemInfoVolumeRootsDir: "/node/actors/actor-a/system-info", VolumesDir: "/node/actors/actor-a/volumes", } - rootfs := ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS + rootfs := ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS - _, err := s.CheckpointWorkload(context.Background(), &ateompb.CheckpointWorkloadRequest{ActorUid: "actor-a", ActorDirs: dirs, Fidelity: rootfs}) + _, err := s.CheckpointWorkload(context.Background(), &ateworkerpb.CheckpointWorkloadRequest{ActorUid: "actor-a", ActorDirs: dirs, Fidelity: rootfs}) if apierror.Code(err) != codes.InvalidArgument || !strings.Contains(err.Error(), "fidelity") { t.Errorf("CheckpointWorkload(ROOTFS) = %v, want InvalidArgument naming the fidelity", err) } - _, err = s.RestoreWorkload(context.Background(), &ateompb.RestoreWorkloadRequest{ActorUid: "actor-a", ActorDirs: dirs, Fidelity: rootfs}) + _, err = s.RestoreWorkload(context.Background(), &ateworkerpb.RestoreWorkloadRequest{ActorUid: "actor-a", ActorDirs: dirs, Fidelity: rootfs}) if apierror.Code(err) != codes.InvalidArgument || !strings.Contains(err.Error(), "fidelity") { t.Errorf("RestoreWorkload(ROOTFS) = %v, want InvalidArgument naming the fidelity", err) } diff --git a/cmd/ateom-gvisor/flags.go b/cmd/ateworker-gvisor/flags.go similarity index 100% rename from cmd/ateom-gvisor/flags.go rename to cmd/ateworker-gvisor/flags.go diff --git a/cmd/ateom-gvisor/flags_test.go b/cmd/ateworker-gvisor/flags_test.go similarity index 100% rename from cmd/ateom-gvisor/flags_test.go rename to cmd/ateworker-gvisor/flags_test.go diff --git a/cmd/ateom-gvisor/hosted.go b/cmd/ateworker-gvisor/hosted.go similarity index 80% rename from cmd/ateom-gvisor/hosted.go rename to cmd/ateworker-gvisor/hosted.go index 81f6793317..e236f5a1c9 100644 --- a/cmd/ateom-gvisor/hosted.go +++ b/cmd/ateworker-gvisor/hosted.go @@ -25,34 +25,34 @@ import ( "time" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomstats" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" "github.com/agent-substrate/substrate/internal/atunnel" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/internal/workerstats" ) // hostedActor holds one actor's attribution, network, and runtime state. type hostedActor struct { // Attribution is available during boot. attribution resources.ActorAttribution - network *ateomnet.SandboxSession + network *actornet.SandboxSession // resolvConf is the actor's resolver bind source, removed with its network. resolvConf string // Set once the containers exist. session *workloadSession // usage is this activation's epoch, CPU baseline, and latest sample. - usage *ateomstats.Activation + usage *workerstats.Activation // deadReported keeps the usage sweep to one warning per activation. deadReported atomic.Bool } // admitActor reserves capacity before network setup. An actor that is already // hosted keeps its slot; its old network is returned for the caller to close. -func (s *AteomService) admitActor(attribution resources.ActorAttribution) (*hostedActor, *ateomnet.SandboxSession, error) { +func (s *AteWorkerService) admitActor(attribution resources.ActorAttribution) (*hostedActor, *actornet.SandboxSession, error) { s.actorsMu.Lock() defer s.actorsMu.Unlock() - var stale *ateomnet.SandboxSession + var stale *actornet.SandboxSession if old, ok := s.actors[attribution.UID]; ok { stale = old.network } else if len(s.actors)+s.draining >= s.maxActors { @@ -60,13 +60,13 @@ func (s *AteomService) admitActor(attribution resources.ActorAttribution) (*host } // The sandbox cgroup is new with every activation, so its counters start // at zero. - hosted := &hostedActor{attribution: attribution, usage: ateomstats.NewActivation(time.Now(), false)} + hosted := &hostedActor{attribution: attribution, usage: workerstats.NewActivation(time.Now(), false)} s.actors[attribution.UID] = hosted return hosted, stale, nil } // hostActor sets up the actor's network, replacing any stale one. -func (s *AteomService) hostActor(ctx context.Context, attribution resources.ActorAttribution, actorDirs *ateompb.ActorDirs) (*hostedActor, error) { +func (s *AteWorkerService) hostActor(ctx context.Context, attribution resources.ActorAttribution, actorDirs *ateworkerpb.ActorDirs) (*hostedActor, error) { uid := attribution.UID if uid == "" { return nil, fmt.Errorf("actor UID is required") @@ -85,7 +85,7 @@ func (s *AteomService) hostActor(ctx context.Context, attribution resources.Acto } } - session, err := ateomnet.ServeSandbox(ctx, ateomnet.SandboxNetworkConfig{ + session, err := actornet.ServeSandbox(ctx, actornet.SandboxNetworkConfig{ ActorUID: uid, Veth: true, EgressPort: s.tunnel.EgressPort, @@ -116,7 +116,7 @@ func (s *AteomService) hostActor(ctx context.Context, attribution resources.Acto } // unhostActor removes an actor and its network. Repeated calls are safe. -func (s *AteomService) unhostActor(ctx context.Context, actorUID string) error { +func (s *AteWorkerService) unhostActor(ctx context.Context, actorUID string) error { s.actorsMu.Lock() hosted, ok := s.actors[actorUID] delete(s.actors, actorUID) @@ -146,14 +146,14 @@ func (s *AteomService) unhostActor(ctx context.Context, actorUID string) error { } // lookupActor returns the actor or nil without taking its lifecycle lock. -func (s *AteomService) lookupActor(actorUID string) *hostedActor { +func (s *AteWorkerService) lookupActor(actorUID string) *hostedActor { s.actorsMu.RLock() defer s.actorsMu.RUnlock() return s.actors[actorUID] } // hostedActors returns a snapshot of the actor map. -func (s *AteomService) hostedActors() []*hostedActor { +func (s *AteWorkerService) hostedActors() []*hostedActor { s.actorsMu.RLock() defer s.actorsMu.RUnlock() out := make([]*hostedActor, 0, len(s.actors)) @@ -164,7 +164,7 @@ func (s *AteomService) hostedActors() []*hostedActor { } // hostedSessions returns the runsc sessions for shutdown. -func (s *AteomService) hostedSessions() []*workloadSession { +func (s *AteWorkerService) hostedSessions() []*workloadSession { s.actorsMu.RLock() defer s.actorsMu.RUnlock() var out []*workloadSession @@ -177,7 +177,7 @@ func (s *AteomService) hostedSessions() []*workloadSession { } // setSession records the runsc state for an actor once its containers exist. -func (s *AteomService) setSession(actorUID string, session *workloadSession) { +func (s *AteWorkerService) setSession(actorUID string, session *workloadSession) { s.actorsMu.Lock() defer s.actorsMu.Unlock() if hosted, ok := s.actors[actorUID]; ok { @@ -186,10 +186,10 @@ func (s *AteomService) setSession(actorUID string, session *workloadSession) { } // sandboxDialer looks up the actor on each dial. -func (s *AteomService) sandboxDialer(actorUID string) atunnel.DialFunc { +func (s *AteWorkerService) sandboxDialer(actorUID string) atunnel.DialFunc { return func(ctx context.Context, network, address string) (net.Conn, error) { s.actorsMu.RLock() - var session *ateomnet.SandboxSession + var session *actornet.SandboxSession if hosted := s.actors[actorUID]; hosted != nil { session = hosted.network } diff --git a/cmd/ateom-gvisor/hosted_test.go b/cmd/ateworker-gvisor/hosted_test.go similarity index 94% rename from cmd/ateom-gvisor/hosted_test.go rename to cmd/ateworker-gvisor/hosted_test.go index 3a3738f19c..b70a76861f 100644 --- a/cmd/ateom-gvisor/hosted_test.go +++ b/cmd/ateworker-gvisor/hosted_test.go @@ -32,7 +32,7 @@ import ( func TestAdmitActorEnforcesTheCeilingConcurrently(t *testing.T) { const ceiling = 8 - s := &AteomService{ + s := &AteWorkerService{ locks: actorlock.New(), actors: map[string]*hostedActor{}, maxActors: ceiling, @@ -56,7 +56,7 @@ func TestAdmitActorEnforcesTheCeilingConcurrently(t *testing.T) { // Re-admitting an actor that is already hosted keeps its slot, so it succeeds // on a full worker and a retry cannot lose its place to another actor. func TestAdmitActorKeepsAHostedActorsSlot(t *testing.T) { - s := &AteomService{actors: map[string]*hostedActor{}, maxActors: 1} + s := &AteWorkerService{actors: map[string]*hostedActor{}, maxActors: 1} first, _, err := s.admitActor(resources.ActorAttribution{UID: "actor-a"}) if err != nil { t.Fatal(err) @@ -79,7 +79,7 @@ func TestAdmitActorKeepsAHostedActorsSlot(t *testing.T) { // TestDrainingActorsStillCountAgainstTheCeiling checks capacity during teardown. func TestDrainingActorsStillCountAgainstTheCeiling(t *testing.T) { - s := &AteomService{ + s := &AteWorkerService{ locks: actorlock.New(), actors: map[string]*hostedActor{}, maxActors: 1, @@ -92,7 +92,7 @@ func TestDrainingActorsStillCountAgainstTheCeiling(t *testing.T) { // TestGracefulShutdownWaitsForInFlightRPCs verifies checkpoints finish before shutdown. func TestGracefulShutdownWaitsForInFlightRPCs(t *testing.T) { - s := &AteomService{ + s := &AteWorkerService{ locks: actorlock.New(), inFlight: actorlock.NewInFlight(), actors: map[string]*hostedActor{}, @@ -120,7 +120,7 @@ func TestGracefulShutdownWaitsForInFlightRPCs(t *testing.T) { } func TestGracefulShutdownCancelsEveryStartup(t *testing.T) { - s := &AteomService{ + s := &AteWorkerService{ locks: actorlock.New(), inFlight: actorlock.NewInFlight(), actors: map[string]*hostedActor{}, @@ -145,11 +145,11 @@ func TestGracefulShutdownCancelsEveryStartup(t *testing.T) { // TestBeginRPCRegistersBeforeCheckingForADrain checks registration and drain rejection. func TestBeginRPCRegistersBeforeCheckingForADrain(t *testing.T) { - s := &AteomService{locks: actorlock.New(), inFlight: actorlock.NewInFlight()} + s := &AteWorkerService{locks: actorlock.New(), inFlight: actorlock.NewInFlight()} release, err := s.beginRPC("actor-a", rpcRunWorkload, func() {}) if err != nil { - t.Fatalf("beginRPC on a live ateom: %v", err) + t.Fatalf("beginRPC on a live ateworker: %v", err) } if _, ok := s.inFlight.Names()["actor-a"]; !ok { t.Error("an admitted RPC is not visible to shutdown") diff --git a/cmd/ateom-gvisor/internal/cdiinject/inject.go b/cmd/ateworker-gvisor/internal/cdiinject/inject.go similarity index 100% rename from cmd/ateom-gvisor/internal/cdiinject/inject.go rename to cmd/ateworker-gvisor/internal/cdiinject/inject.go diff --git a/cmd/ateom-gvisor/internal/cdiinject/inject_test.go b/cmd/ateworker-gvisor/internal/cdiinject/inject_test.go similarity index 98% rename from cmd/ateom-gvisor/internal/cdiinject/inject_test.go rename to cmd/ateworker-gvisor/internal/cdiinject/inject_test.go index 3ce891d3e2..d5ca98f302 100644 --- a/cmd/ateom-gvisor/internal/cdiinject/inject_test.go +++ b/cmd/ateworker-gvisor/internal/cdiinject/inject_test.go @@ -44,7 +44,7 @@ func statDev(t *testing.T, path string) (int64, int64) { const testHookBinary = "/opt/toolkit/cdi-hook" // injectFixture reads a spec from disk and applies the "all" device with the -// options the gVisor ateom uses, so these tests exercise the real combination. +// options the gVisor ateworker uses, so these tests exercise the real combination. func injectFixture(t *testing.T, bundleDir, specDir string) error { t.Helper() data, err := os.ReadFile(filepath.Join(specDir, "nvidia.json")) @@ -273,11 +273,11 @@ func TestInjectGPUIntoBundle_Idempotent(t *testing.T) { } // TestStageSonameSymlinks_ConfinedToRootfs covers the two ways the staging writes -// could escape into ateom's mount namespace, where the shared image cache and other +// could escape into ateworker's mount namespace, where the shared image cache and other // actors' bundles live: a directory the image redirects out of the rootfs, and a // SONAME read out of a library that is not a bare filename. // TestStageSonameSymlinks_ConfinedToRootfs covers the two ways the staging writes -// could escape into ateom's mount namespace, where the shared image cache and other +// could escape into ateworker's mount namespace, where the shared image cache and other // actors' bundles live: a directory the image redirects out of the rootfs, and a // SONAME read out of a library that is not a bare filename. func TestStageSonameSymlinks_ConfinedToRootfs(t *testing.T) { diff --git a/cmd/ateom-gvisor/internal/cdiinject/soname.go b/cmd/ateworker-gvisor/internal/cdiinject/soname.go similarity index 99% rename from cmd/ateom-gvisor/internal/cdiinject/soname.go rename to cmd/ateworker-gvisor/internal/cdiinject/soname.go index ea88c5c368..86763d6a78 100644 --- a/cmd/ateom-gvisor/internal/cdiinject/soname.go +++ b/cmd/ateworker-gvisor/internal/cdiinject/soname.go @@ -39,7 +39,7 @@ import ( // here. // Every path component under rootfs comes from the actor image, so the writes go // through os.Root: an image that ships a driver-mount's parent directory as a -// symlink out of the rootfs would otherwise have the kernel resolve it in ateom's +// symlink out of the rootfs would otherwise have the kernel resolve it in ateworker's // mount namespace, where the shared image cache and other actors' bundles are // mounted. Same treatment as createExtraDirs in internal/imagecache. func StageSonameSymlinks(ctx context.Context, rootfs string, mounts []specs.Mount, allowedHostDirs []string) error { diff --git a/cmd/ateom-gvisor/internal/cgroupstats/cgroupstats.go b/cmd/ateworker-gvisor/internal/cgroupstats/cgroupstats.go similarity index 96% rename from cmd/ateom-gvisor/internal/cgroupstats/cgroupstats.go rename to cmd/ateworker-gvisor/internal/cgroupstats/cgroupstats.go index 7ed8eb5124..ea3a7aa689 100644 --- a/cmd/ateom-gvisor/internal/cgroupstats/cgroupstats.go +++ b/cmd/ateworker-gvisor/internal/cgroupstats/cgroupstats.go @@ -14,7 +14,7 @@ // Package cgroupstats reads resource usage out of a cgroup v2 directory. // -// The gVisor ateom uses it to answer ateompb.Ateom/GetWorkloadStats: the sentry +// The gVisor ateworker uses it to answer ateworkerpb.Worker/GetWorkloadStats: the sentry // hosts the whole sandbox in one host process, so the sandbox's cgroup leaf is // where the workload's memory and CPU actually show up. // @@ -39,7 +39,7 @@ import ( // // Fields the kernel does not expose read as zero rather than failing the whole // sample: a partial reading is more useful than none, and the alternative is an -// ateom that reports nothing at all on a kernel missing one file. Read's doc +// ateworker that reports nothing at all on a kernel missing one file. Read's doc // comment says which fields can do this and why. type Sample struct { // MemoryCurrentBytes is memory.current: bytes currently charged to the @@ -61,7 +61,7 @@ type Sample struct { // CPUUsageUsec is cpu.stat's usage_usec: cumulative CPU time consumed by the // cgroup since it was created. Zero if the cpu controller was not delegated - // to this cgroup (see ateomcgroup.Delegate, which enables controllers + // to this cgroup (see cgroup.Delegate, which enables controllers // best-effort and carries on when one cannot be enabled). CPUUsageUsec uint64 diff --git a/cmd/ateom-gvisor/internal/cgroupstats/cgroupstats_test.go b/cmd/ateworker-gvisor/internal/cgroupstats/cgroupstats_test.go similarity index 99% rename from cmd/ateom-gvisor/internal/cgroupstats/cgroupstats_test.go rename to cmd/ateworker-gvisor/internal/cgroupstats/cgroupstats_test.go index 8b5198a295..c7728b0b11 100644 --- a/cmd/ateom-gvisor/internal/cgroupstats/cgroupstats_test.go +++ b/cmd/ateworker-gvisor/internal/cgroupstats/cgroupstats_test.go @@ -103,7 +103,7 @@ func TestRead(t *testing.T) { }, }, { - // ateomcgroup.Delegate enables controllers one at a time and carries on + // cgroup.Delegate enables controllers one at a time and carries on // when one cannot be enabled, so a cgroup with memory but no cpu is a // state this actually reaches. name: "no cpu.stat (cpu controller not delegated)", diff --git a/cmd/ateom-gvisor/kodata/internal_flags.json b/cmd/ateworker-gvisor/kodata/internal_flags.json similarity index 100% rename from cmd/ateom-gvisor/kodata/internal_flags.json rename to cmd/ateworker-gvisor/kodata/internal_flags.json diff --git a/cmd/ateom-gvisor/main.go b/cmd/ateworker-gvisor/main.go similarity index 86% rename from cmd/ateom-gvisor/main.go rename to cmd/ateworker-gvisor/main.go index 6457e2e72b..be216acb61 100644 --- a/cmd/ateom-gvisor/main.go +++ b/cmd/ateworker-gvisor/main.go @@ -33,28 +33,29 @@ import ( "time" "cloud.google.com/go/compute/metadata" - "github.com/agent-substrate/substrate/cmd/ateom-gvisor/internal/cgroupstats" + "github.com/agent-substrate/substrate/cmd/ateworker-gvisor/internal/cgroupstats" "github.com/agent-substrate/substrate/internal/actorlock" "github.com/agent-substrate/substrate/internal/actorlog" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateinterceptors" - "github.com/agent-substrate/substrate/internal/ateom" - "github.com/agent-substrate/substrate/internal/ateomcgroup" - "github.com/agent-substrate/substrate/internal/ateominit" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomphaselog" - "github.com/agent-substrate/substrate/internal/ateomstats" - "github.com/agent-substrate/substrate/internal/ateomtunnel" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" + "github.com/agent-substrate/substrate/internal/ateworker/cgroup" + "github.com/agent-substrate/substrate/internal/ateworker/phaselog" + "github.com/agent-substrate/substrate/internal/ateworker/pid1" + "github.com/agent-substrate/substrate/internal/ateworker/register" + "github.com/agent-substrate/substrate/internal/ateworker/tunnel" "github.com/agent-substrate/substrate/internal/imagecache" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/ocispec" "github.com/agent-substrate/substrate/internal/otlprelay" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/preview" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/sizing" "github.com/agent-substrate/substrate/internal/version" "github.com/agent-substrate/substrate/internal/wakeupprobe" + "github.com/agent-substrate/substrate/internal/workerstats" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "github.com/spf13/pflag" "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" @@ -74,14 +75,15 @@ func flagsConfigPath() string { var ( podUID = pflag.String("pod-uid", "", "The UID of the current pod") - tunnelConfig = ateomtunnel.RegisterFlags(pflag.CommandLine) + tunnelConfig = tunnel.RegisterFlags(pflag.CommandLine) readinessListenAddress = pflag.String("readiness-listen-address", "0.0.0.0:8080", "Address for the HTTP /readyz and /healthz probes") maxActors = pflag.Int("max-actors", 1000, "How many actors this worker will host at once") - usageSampleInterval = pflag.Duration("usage-sample-interval", time.Minute, "How often the ateom samples the resource usage of its actors. Each sample writes an ate.actor.usage_sampled record per actor, and GetActiveWorkloadStats serves the latest.") + usageSampleInterval = pflag.Duration("usage-sample-interval", time.Minute, "How often the ateworker samples the resource usage of its actors. Each sample writes an ate.actor.usage_sampled record per actor, and GetActiveWorkloadStats serves the latest.") showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") + previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.") otlpRelaySocket = pflag.String("otlp-relay-socket", nodepath.AteletOTLPSocketPath(), "Unix socket of atelet's OTLP relay to export telemetry through, keeping it off the pod network. Empty, or absent at startup, exports directly to OTEL_EXPORTER_OTLP_ENDPOINT instead.") @@ -93,12 +95,12 @@ const minUsageSampleInterval = 50 * time.Second // workloadGracePeriod is the whole budget for draining the worker on shutdown. // It needs to stay significantly less than the K8s termination grace period -// for the ateom, so the escalation to SIGKILL happens here rather than as a -// kubelet SIGKILL of ateom itself. +// for the ateworker, so the escalation to SIGKILL happens here rather than as a +// kubelet SIGKILL of ateworker itself. const workloadGracePeriod = 30 * time.Minute func main() { - ateominit.RunIfPID1() + pid1.RunIfPID1() pflag.Parse() if *showVersion { fmt.Println(version.String()) @@ -122,8 +124,11 @@ func do(ctx context.Context) error { if err := serverboot.SetLogLevel(*logLevelFlag); err != nil { return err } + if err := preview.Init(*previewFlags...); err != nil { + return fmt.Errorf("invalid --preview: %w", err) + } - slog.InfoContext(ctx, "ateom booting", slog.String("version", version.Version)) + slog.InfoContext(ctx, "ateworker booting", slog.String("version", version.Version)) if *maxActors < 0 { return fmt.Errorf("--max-actors must not be negative, got %d", *maxActors) } @@ -131,7 +136,7 @@ func do(ctx context.Context) error { return fmt.Errorf("--usage-sample-interval must be at least %v, got %v", minUsageSampleInterval, *usageSampleInterval) } - const serviceName = "ateom-gvisor" + const serviceName = "ateworker-gvisor" // Export through atelet's node-local relay when it is there, so telemetry // never touches the worker pod's network. A nil conn means it is not, and // the providers fall back to dialing the collector directly. @@ -140,7 +145,7 @@ func do(ctx context.Context) error { // ends of the same decision: Dial already treats an absent socket as a // fallback rather than an error, and atelet logs and keeps going when it // cannot serve the relay at all. What is lost here is the node-local export - // path, not the ateom's ability to run actors, and failing the worker pod + // path, not the ateworker's ability to run actors, and failing the worker pod // over its telemetry route would turn a misconfigured flag into an outage. relayConn, err := otlprelay.Dial(ctx, *otlpRelaySocket) if err != nil { @@ -156,7 +161,7 @@ func do(ctx context.Context) error { Sampling: serverboot.ResolveTraceSampling(ctx, serverboot.ParentRatioSampling(serverboot.ControlPlaneTraceRatio)), ExporterConn: relayConn, // So the spans say which path they took, including when relayConn is nil - // because the dial above failed and this ateom is exporting directly. + // because the dial above failed and this ateworker is exporting directly. RelayCapable: true, }) if err != nil { @@ -184,25 +189,25 @@ func do(ctx context.Context) error { defer serverboot.ShutdownProvider("LoggerProvider", lp.Shutdown) } - // Create ateom dir - ateomDir := nodepath.AteomPath(*podUID) + // Create ateworker dir + ateworkerDir := nodepath.WorkerPath(*podUID) if err := resources.ValidateWorkerPodUID(*podUID); err != nil { return fmt.Errorf("in resources.ValidateWorkerPodUID: %w", err) } // atelet owns the directory and the sandbox versions file in it; only the // socket below is this process's. Both outlive a container restart. - if err := os.MkdirAll(ateomDir, 0o700); err != nil { - return fmt.Errorf("in os.MkdirAll(%q): %w", ateomDir, err) + if err := os.MkdirAll(ateworkerDir, 0o700); err != nil { + return fmt.Errorf("in os.MkdirAll(%q): %w", ateworkerDir, err) } // Prepare the pod cgroup so runsc can create per-actor-container leaves under // it with real accounting. - if _, err := ateomcgroup.Delegate(ctx); err != nil { + if _, err := cgroup.Delegate(ctx); err != nil { return fmt.Errorf("while setting up cgroup delegation: %w", err) } // Clean up any old socket. - sockPath := nodepath.AteomSocketPath(*podUID) + sockPath := nodepath.WorkerSocketPath(*podUID) if err := os.RemoveAll(sockPath); err != nil { return fmt.Errorf("while removing %q: %w", sockPath, err) } @@ -213,7 +218,7 @@ func do(ctx context.Context) error { } defer func() { if err := os.Remove(sockPath); err != nil && !errors.Is(err, os.ErrNotExist) { - slog.ErrorContext(ctx, "Failed to remove the ateom socket on shutdown", slog.Any("err", err)) + slog.ErrorContext(ctx, "Failed to remove the ateworker socket on shutdown", slog.Any("err", err)) } }() @@ -223,26 +228,26 @@ func do(ctx context.Context) error { } actorLogger := actorlog.NewActorLogger(syncedWriter, metadata.OnGCE()) - tunnel, err := ateomtunnel.Start(ctx, *tunnelConfig, ateomnet.ActorHTTPUpstream) + tunnel, err := tunnel.Start(ctx, *tunnelConfig, actornet.ActorHTTPUpstream) if err != nil { return err } - ateomService := NewService(tunnel, actorLogger, *maxActors, flags) + ateworkerService := NewService(tunnel, actorLogger, *maxActors, flags) // The controller sets both from the downward API. - pool := ateomstats.Pool{Namespace: os.Getenv("POD_NAMESPACE"), Name: os.Getenv("WORKER_POOL_NAME")} + pool := workerstats.Pool{Namespace: os.Getenv("POD_NAMESPACE"), Name: os.Getenv("WORKER_POOL_NAME")} if pool.Namespace == "" || pool.Name == "" { slog.WarnContext(ctx, "Worker pool unknown; usage records will name no pool", slog.Any("pool", pool)) } - usageStdout := ateomstats.NewStdoutHandler(syncedWriter) + usageStdout := workerstats.NewStdoutHandler(syncedWriter) defer usageStdout.Close() - ateomService.usage = ateomstats.NewUsageEmitter(lp, usageStdout, pool) - defer ateomstats.StartSampler(ctx, *usageSampleInterval, func(ctx context.Context) { ateomService.sweepUsage(ctx) })() + ateworkerService.usage = workerstats.NewUsageEmitter(lp, usageStdout, pool) + defer workerstats.StartSampler(ctx, *usageSampleInterval, func(ctx context.Context) { ateworkerService.sweepUsage(ctx) })() svr := grpc.NewServer( grpc.StatsHandler(otelgrpc.NewServerHandler()), grpc.UnaryInterceptor(ateinterceptors.InternalServerUnaryInterceptor), ) - ateompb.RegisterAteomServer(svr, ateomService) + ateworkerpb.RegisterWorkerServer(svr, ateworkerService) reflection.Register(svr) readiness := &serverboot.Readiness{} @@ -257,7 +262,7 @@ func do(ctx context.Context) error { readiness.MarkNotReady() // Use a fresh context: the do() context is torn down on return, but the // shutdown must outlive it until the sandbox has stopped. - ateomService.gracefulShutdown(context.Background()) + ateworkerService.gracefulShutdown(context.Background()) // Stop the server gracefully. This blocks until all in-flight RPCs have completed. svr.GracefulStop() }() @@ -268,8 +273,8 @@ func do(ctx context.Context) error { // outlast, including the window before the Worker record exists; anything // that reaches here is a misconfiguration no restart-in-place will fix. go func() { - err := ateom.Report(ctx, ateom.ReportConfig{ - SocketPath: nodepath.AteomSupportSocket, + err := register.Report(ctx, register.ReportConfig{ + SocketPath: nodepath.WorkerSupportSocket, CredentialBundlePath: tunnelConfig.CredentialBundle, TrustBundlePath: tunnelConfig.TrustBundle, AteletSPIFFEID: tunnelConfig.BrokerIdentity, @@ -304,9 +309,9 @@ type workloadSession struct { containers []string } -// AteomService is a service for shepherding single microvm. -type AteomService struct { - ateompb.UnimplementedAteomServer +// AteWorkerService is a service for shepherding single microvm. +type AteWorkerService struct { + ateworkerpb.UnimplementedWorkerServer // Serializes lifecycle RPCs per actor. locks *actorlock.Locks @@ -322,16 +327,16 @@ type AteomService struct { maxActors int actorLogger *actorlog.ActorLogger - tunnel *ateomtunnel.Tunnel + tunnel *tunnel.Tunnel // usage writes the usage records. Nil writes none. - usage *ateomstats.UsageEmitter + usage *workerstats.UsageEmitter // shuttingDown is set once SIGTERM has been received. While true, new // workload RPCs are rejected with codes.Unavailable. shuttingDown atomic.Bool // cgroupRoot is where the sandbox's cgroup v2 leaves live: the worker pod's - // own cgroup scope, which ateomcgroup.Delegate prepares. A field rather + // own cgroup scope, which cgroup.Delegate prepares. A field rather // than a constant so tests can point GetWorkloadStats at a fixture tree. cgroupRoot string @@ -345,11 +350,11 @@ type AteomService struct { flags *flagStore } -var _ ateompb.AteomServer = (*AteomService)(nil) +var _ ateworkerpb.WorkerServer = (*AteWorkerService)(nil) -// NewService creates a new AteomService. -func NewService(tunnel *ateomtunnel.Tunnel, actorLogger *actorlog.ActorLogger, maxActors int, flags *flagStore) *AteomService { - return &AteomService{ +// NewService creates a new AteWorkerService. +func NewService(tunnel *tunnel.Tunnel, actorLogger *actorlog.ActorLogger, maxActors int, flags *flagStore) *AteWorkerService { + return &AteWorkerService{ locks: actorlock.New(), inFlight: actorlock.NewInFlight(), actors: map[string]*hostedActor{}, @@ -362,7 +367,7 @@ func NewService(tunnel *ateomtunnel.Tunnel, actorLogger *actorlog.ActorLogger, m } // beginRPC registers before checking the drain flag so shutdown cannot miss it. -func (s *AteomService) beginRPC(actorUID, name string, cancel context.CancelFunc) (func(), error) { +func (s *AteWorkerService) beginRPC(actorUID, name string, cancel context.CancelFunc) (func(), error) { release := s.inFlight.Add(actorUID, name, cancel) if err := s.rejectIfDraining(); err != nil { release() @@ -371,9 +376,9 @@ func (s *AteomService) beginRPC(actorUID, name string, cancel context.CancelFunc return release, nil } -// rejectIfDraining returns a codes.Unavailable error if ateom has begun graceful +// rejectIfDraining returns a codes.Unavailable error if ateworker has begun graceful // shutdown, so the control plane reschedules the actor onto a live worker. -func (s *AteomService) rejectIfDraining() error { +func (s *AteWorkerService) rejectIfDraining() error { if s.shuttingDown.Load() { return apierror.Unavailable("worker draining: not accepting new workloads") } @@ -390,7 +395,7 @@ func cancelStartups(ctx context.Context, inFlight *actorlock.InFlight) { // gracefulShutdown propagates SIGTERM into the sandbox and waits for the application's // containers to exit. -func (s *AteomService) gracefulShutdown(ctx context.Context) { +func (s *AteWorkerService) gracefulShutdown(ctx context.Context) { s.shuttingDown.Store(true) // If there is an active run or restore RPC, try to cancel it. This is considered // less disruptive than waiting for it to complete and then immediately sending @@ -490,7 +495,7 @@ func killContainer(ctx context.Context, rcmd containerRuntime, name string, dead } // If the error was not due to context timeout or cancellation, it means the wait command - // itself failed, so we return the error and do not escalate to SIGKILL. Ateom shutting down + // itself failed, so we return the error and do not escalate to SIGKILL. AteWorker shutting down // will kill the containers. if !errors.Is(err, context.DeadlineExceeded) && !errors.Is(err, context.Canceled) { return fmt.Errorf("wait failed: %w", err) @@ -534,7 +539,7 @@ func waitContainerStop(ctx context.Context, done <-chan error) error { } } -func containerNames(containers []*ateompb.Container) []string { +func containerNames(containers []*ateworkerpb.Container) []string { names := make([]string, 0, len(containers)) for _, c := range containers { names = append(names, c.GetName()) @@ -551,7 +556,7 @@ func validateRunscPath(p string) error { return nil } -func validateActorDirs(actorDirs *ateompb.ActorDirs) error { +func validateActorDirs(actorDirs *ateworkerpb.ActorDirs) error { if errs := resources.ValidateActorDirs(actorDirs, field.NewPath("actor_dirs")); len(errs) > 0 { return apierror.InvalidArgument("%v", errs.ToAggregate()) } @@ -561,21 +566,21 @@ func validateActorDirs(actorDirs *ateompb.ActorDirs) error { // supportedFidelities are the snapshot fidelities this runtime serves. gVisor // keeps rootfs changes inside its memory checkpoint, so it cannot capture // ROOTFS without MEMORY. -var supportedFidelities = []ateompb.SnapshotFidelity{ - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, +var supportedFidelities = []ateworkerpb.SnapshotFidelity{ + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, } // validateFidelity rejects a checkpoint or restore request whose fidelity // this runtime cannot serve. -func validateFidelity(fidelity ateompb.SnapshotFidelity) error { +func validateFidelity(fidelity ateworkerpb.SnapshotFidelity) error { if errs := resources.ValidateSnapshotFidelity(fidelity, supportedFidelities, field.NewPath("fidelity")); len(errs) > 0 { return apierror.InvalidArgument("%v", errs.ToAggregate()) } return nil } -func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkloadRequest) (resp *ateompb.RunWorkloadResponse, retErr error) { +func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.RunWorkloadRequest) (resp *ateworkerpb.RunWorkloadResponse, retErr error) { if err := validateActorDirs(req.GetActorDirs()); err != nil { return nil, err } @@ -597,11 +602,11 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload return nil, err } - if err := s.tunnel.Deactivate(ctx, ateomstats.ActorAttributionFromRequest(req)); err != nil { + if err := s.tunnel.Deactivate(ctx, workerstats.ActorAttributionFromRequest(req)); err != nil { return nil, err } - attribution := ateomstats.ActorAttributionFromRequest(req) + attribution := workerstats.ActorAttributionFromRequest(req) s.actorLogger.EmitLifecycleLog(ctx, "Actor starting", attribution) // Contract with atelet: @@ -609,7 +614,7 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload // * Correct runsc version is downloaded and placed on disk. // * All OCI bundles are set up, including for the pause container. - egress, err := s.tunnel.PrepareEgress(ctx, ateomstats.ActorAttributionFromRequest(req), req.GetEgressGateway()) + egress, err := s.tunnel.PrepareEgress(ctx, workerstats.ActorAttributionFromRequest(req), req.GetEgressGateway()) if err != nil { return nil, err } @@ -632,7 +637,7 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload if retErr != nil { cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 30*time.Second) defer cancel() - if err := s.tunnel.Deactivate(cleanupCtx, ateomstats.ActorAttributionFromRequest(req)); err != nil { + if err := s.tunnel.Deactivate(cleanupCtx, workerstats.ActorAttributionFromRequest(req)); err != nil { slog.WarnContext(cleanupCtx, "Failed to deactivate actor networking after Run failure", slog.Any("err", err)) } deleteContainers(cleanupCtx, rcmd, containersToDelete, "Run") @@ -651,7 +656,7 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload }() // Create and start pause container. The bundle rootfs is composed here — // an overlay of the node's cached image layers plus the bundle's private - // upper — because mounting is ateom's job (atelet runs with no + // upper — because mounting is ateworker's job (atelet runs with no // capabilities); runsc's gofer resolves the mount in this pod's mount // namespace. if err := imagecache.SetupBundleRootfs(ociBundlePath(req.GetActorDirs(), ocispec.PauseContainer)); err != nil { @@ -692,10 +697,10 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload } // Block until every wakeup-probe-enabled container reports 200. - if err := wakeupprobe.WaitAll(ctx, req.GetSpec().GetContainers(), ateomnet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(req.GetActorUid()))); err != nil { + if err := wakeupprobe.WaitAll(ctx, req.GetSpec().GetContainers(), actornet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(req.GetActorUid()))); err != nil { return nil, fmt.Errorf("while waiting for container wakeup probe: %w", err) } - if err := s.tunnel.Activate(ateomstats.ActorAttributionFromRequest(req), s.sandboxDialer(req.GetActorUid()), egress); err != nil { + if err := s.tunnel.Activate(workerstats.ActorAttributionFromRequest(req), s.sandboxDialer(req.GetActorUid()), egress); err != nil { return nil, err } @@ -703,12 +708,12 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload s.setSession(req.GetActorUid(), &workloadSession{rcmd: rcmd, containers: containerNames(req.GetSpec().GetContainers())}) s.recordInitial(ctx, hosted) - return &ateompb.RunWorkloadResponse{}, nil + return &ateworkerpb.RunWorkloadResponse{}, nil } // Allow checkpointing even if the pod is shutting down. This will allow actors // (or the harness) to suspend on shutdown. -func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.CheckpointWorkloadRequest) (_ *ateompb.CheckpointWorkloadResponse, err error) { +func (s *AteWorkerService) CheckpointWorkload(ctx context.Context, req *ateworkerpb.CheckpointWorkloadRequest) (_ *ateworkerpb.CheckpointWorkloadResponse, err error) { // Taken before the lock, so prep covers the wait for it. tStart := time.Now() if err := validateActorDirs(req.GetActorDirs()); err != nil { @@ -732,14 +737,14 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec // Per-phase timing, recorded on the way out so a failed checkpoint still // reports the phases it completed, and the failing step its elapsed time. - attribution := ateomstats.ActorAttributionFromRequest(req) + attribution := workerstats.ActorAttributionFromRequest(req) fidelity := req.GetFidelity() var timing checkpointTiming tLast := tStart defer func() { timing.total = time.Since(tStart) - ateomphaselog.LogSnapshotPhases(ctx, "Checkpoint timing breakdown", attribution, fidelity, - ateomphaselog.CheckpointDurationKey, err, timing.phases()) + phaselog.LogSnapshotPhases(ctx, "Checkpoint timing breakdown", attribution, fidelity, + phaselog.CheckpointDurationKey, err, timing.phases()) }() if err := s.tunnel.Deactivate(ctx, attribution); err != nil { @@ -777,7 +782,7 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec containers := req.GetSpec().GetContainers() durablePaths := durableFSCheckpointPaths(containers) switch fidelity { - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: if len(durablePaths) == 0 { return nil, fmt.Errorf("no durable-dir volumes found for VOLUMES snapshot") } @@ -786,7 +791,7 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec if err != nil { return nil, fmt.Errorf("while checkpointing durable-dir volumes: %w", err) } - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: // Checkpoint pause container (root of the sandbox), splitting durable-dir // volumes into /fs when present. sandbox, sbErr := openSandboxProcess(req.GetActorDirs()) @@ -834,7 +839,7 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec s.actorLogger.EmitLifecycleLog(ctx, "Actor checkpointed", attribution) - return &ateompb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles}, nil + return &ateworkerpb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles}, nil } // listSnapshotFiles returns the (relative) names of regular files directly under @@ -867,14 +872,14 @@ func listSnapshotFiles(dir string) ([]string, error) { // is left running: it is the sandbox, and cleanupContainers deletes the others // through it before deleting it last. Killing it first leaves the sentry a // zombie until reaped, which runsc delete mistakes for a live sandbox. -func stopContainers(ctx context.Context, rcmd containerRuntime, containers []*ateompb.Container) { +func stopContainers(ctx context.Context, rcmd containerRuntime, containers []*ateworkerpb.Container) { for _, ctr := range containers { _ = rcmd.cmdKill(ctx, ctr.GetName(), "SIGKILL") _ = rcmd.cmdWait(ctx, ctr.GetName()) } } -func cleanupContainers(ctx context.Context, rcmd containerRuntime, containers []*ateompb.Container) error { +func cleanupContainers(ctx context.Context, rcmd containerRuntime, containers []*ateworkerpb.Container) error { // Application containers first, the pause (root) container last. names := make([]string, 0, len(containers)+1) for _, ctr := range containers { @@ -920,7 +925,7 @@ func isContainerAlreadyGone(ctx context.Context, rcmd containerRuntime, name str return !slices.Contains(ids, name), nil } -func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.RestoreWorkloadRequest) (resp *ateompb.RestoreWorkloadResponse, retErr error) { +func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb.RestoreWorkloadRequest) (resp *ateworkerpb.RestoreWorkloadResponse, retErr error) { // Taken before the lock, so prep covers the wait for it. tStart := time.Now() if err := validateActorDirs(req.GetActorDirs()); err != nil { @@ -943,14 +948,14 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore // reports the phases it completed, and the failing step its elapsed time. // Registered ahead of the failure cleanup below, so on failure the total // includes that cleanup; the record is marked by error.type either way. - attribution := ateomstats.ActorAttributionFromRequest(req) + attribution := workerstats.ActorAttributionFromRequest(req) fidelity := req.GetFidelity() containers := req.GetSpec().GetContainers() var timing restoreTiming tLast := tStart defer func() { timing.total = time.Since(tStart) - attrs := ateomphaselog.SnapshotPhaseAttrs(attribution, fidelity, ateomphaselog.RestoreDurationKey, retErr, timing.phases()) + attrs := phaselog.SnapshotPhaseAttrs(attribution, fidelity, phaselog.RestoreDurationKey, retErr, timing.phases()) attrs = append(attrs, slog.Int(containerCountKey, len(containers))) slog.LogAttrs(ctx, slog.LevelInfo, "Restore timing breakdown", attrs...) }() @@ -1038,7 +1043,7 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore } defer pw.Close() switch fidelity { - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: // Create and start pause container (cold boot with durable-dir volumes restored from fs/) containersToDelete = append(containersToDelete, ocispec.PauseContainer) err := rcmd.cmdCreate(ctx, pw, ocispec.PauseContainer, restoreArgs) @@ -1051,7 +1056,7 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore if err != nil { return nil, fmt.Errorf("while starting pause container: %w", err) } - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: // Create and restore pause container (restores both process/rootfs and fs/ durable-dir volumes) containersToDelete = append(containersToDelete, ocispec.PauseContainer) err := rcmd.cmdCreate(ctx, pw, ocispec.PauseContainer, restoreArgs) @@ -1088,13 +1093,13 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore return nil, fmt.Errorf("while creating %q application container: %w", ac.GetName(), err) } switch fidelity { - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: err = rcmd.cmdStart(ctx, pw, ac.GetName()) timing.appRestore += lap(&tLast) if err != nil { return nil, fmt.Errorf("while starting %q application container: %w", ac.GetName(), err) } - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: err = rcmd.cmdRestore(ctx, pw, ac.GetName(), checkpointDir) timing.appRestore += lap(&tLast) if err != nil { @@ -1104,7 +1109,7 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore } // Block until every wakeup-probe-enabled container reports 200. - err = wakeupprobe.WaitAll(ctx, containers, ateomnet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(req.GetActorUid()))) + err = wakeupprobe.WaitAll(ctx, containers, actornet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(req.GetActorUid()))) timing.wakeupProbe = lap(&tLast) if err != nil { return nil, fmt.Errorf("while waiting for container wakeup probe: %w", err) @@ -1119,10 +1124,10 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore s.setSession(req.GetActorUid(), &workloadSession{rcmd: rcmd, containers: containerNames(containers)}) s.recordInitial(ctx, hosted) - return &ateompb.RestoreWorkloadResponse{}, nil + return &ateworkerpb.RestoreWorkloadResponse{}, nil } -func (s *AteomService) TerminateWorkload(ctx context.Context, req *ateompb.TerminateWorkloadRequest) (*ateompb.TerminateWorkloadResponse, error) { +func (s *AteWorkerService) TerminateWorkload(ctx context.Context, req *ateworkerpb.TerminateWorkloadRequest) (*ateworkerpb.TerminateWorkloadResponse, error) { if err := validateActorDirs(req.GetActorDirs()); err != nil { return nil, err } @@ -1134,7 +1139,7 @@ func (s *AteomService) TerminateWorkload(ctx context.Context, req *ateompb.Termi } defer s.locks.Unlock(req.GetActorUid()) - attribution := ateomstats.ActorAttributionFromRequest(req) + attribution := workerstats.ActorAttributionFromRequest(req) hosted := s.lookupActor(attribution.UID) if hosted != nil { @@ -1148,10 +1153,10 @@ func (s *AteomService) TerminateWorkload(ctx context.Context, req *ateompb.Termi s.actorLogger.EmitLifecycleLog(ctx, "Actor terminated", attribution) - return &ateompb.TerminateWorkloadResponse{}, nil + return &ateworkerpb.TerminateWorkloadResponse{}, nil } -func (s *AteomService) terminateWorkload(ctx context.Context, actorRef resources.ActorRef, actorUID, runscPath string, actorDirs *ateompb.ActorDirs, containers []*ateompb.Container) error { +func (s *AteWorkerService) terminateWorkload(ctx context.Context, actorRef resources.ActorRef, actorUID, runscPath string, actorDirs *ateworkerpb.ActorDirs, containers []*ateworkerpb.Container) error { var errs []error if err := s.tunnel.Deactivate(ctx, resources.ActorAttribution{Ref: actorRef, UID: actorUID}); err != nil { errs = append(errs, fmt.Errorf("while deactivating actor networking: %w", err)) diff --git a/cmd/ateom-gvisor/main_test.go b/cmd/ateworker-gvisor/main_test.go similarity index 87% rename from cmd/ateom-gvisor/main_test.go rename to cmd/ateworker-gvisor/main_test.go index cf34aaa91a..8a937d7c40 100644 --- a/cmd/ateom-gvisor/main_test.go +++ b/cmd/ateworker-gvisor/main_test.go @@ -35,29 +35,29 @@ import ( "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" ) // Every RPC rejects a request without ActorDirs before touching any state. func TestRPCsRejectMissingActorDirs(t *testing.T) { - s := &AteomService{} + s := &AteWorkerService{} ctx := context.Background() for name, call := range map[string]func() error{ "RunWorkload": func() error { - _, err := s.RunWorkload(ctx, &ateompb.RunWorkloadRequest{}) + _, err := s.RunWorkload(ctx, &ateworkerpb.RunWorkloadRequest{}) return err }, "RestoreWorkload": func() error { - _, err := s.RestoreWorkload(ctx, &ateompb.RestoreWorkloadRequest{}) + _, err := s.RestoreWorkload(ctx, &ateworkerpb.RestoreWorkloadRequest{}) return err }, "CheckpointWorkload": func() error { - _, err := s.CheckpointWorkload(ctx, &ateompb.CheckpointWorkloadRequest{}) + _, err := s.CheckpointWorkload(ctx, &ateworkerpb.CheckpointWorkloadRequest{}) return err }, "TerminateWorkload": func() error { - _, err := s.TerminateWorkload(ctx, &ateompb.TerminateWorkloadRequest{}) + _, err := s.TerminateWorkload(ctx, &ateworkerpb.TerminateWorkloadRequest{}) return err }, } { @@ -68,9 +68,9 @@ func TestRPCsRejectMissingActorDirs(t *testing.T) { } func TestRPCsRejectUntrustedRunscPath(t *testing.T) { - s := &AteomService{} + s := &AteWorkerService{} ctx := context.Background() - dirs := &ateompb.ActorDirs{ + dirs := &ateworkerpb.ActorDirs{ RootDir: "/node/actors/actor-a", OciBundleDir: "/node/actors/actor-a/bundle", CheckpointDir: "/node/actors/actor-a/checkpoint-state", @@ -81,19 +81,19 @@ func TestRPCsRejectUntrustedRunscPath(t *testing.T) { } for name, call := range map[string]func() error{ "RunWorkload": func() error { - _, err := s.RunWorkload(ctx, &ateompb.RunWorkloadRequest{ActorDirs: dirs, RunscPath: "/bin/sh"}) + _, err := s.RunWorkload(ctx, &ateworkerpb.RunWorkloadRequest{ActorDirs: dirs, RunscPath: "/bin/sh"}) return err }, "RestoreWorkload": func() error { - _, err := s.RestoreWorkload(ctx, &ateompb.RestoreWorkloadRequest{ActorDirs: dirs, RunscPath: "/bin/sh"}) + _, err := s.RestoreWorkload(ctx, &ateworkerpb.RestoreWorkloadRequest{ActorDirs: dirs, RunscPath: "/bin/sh"}) return err }, "CheckpointWorkload": func() error { - _, err := s.CheckpointWorkload(ctx, &ateompb.CheckpointWorkloadRequest{ActorDirs: dirs, RunscPath: "/bin/sh"}) + _, err := s.CheckpointWorkload(ctx, &ateworkerpb.CheckpointWorkloadRequest{ActorDirs: dirs, RunscPath: "/bin/sh"}) return err }, "TerminateWorkload": func() error { - _, err := s.TerminateWorkload(ctx, &ateompb.TerminateWorkloadRequest{ActorDirs: dirs, RunscPath: "/bin/sh"}) + _, err := s.TerminateWorkload(ctx, &ateworkerpb.TerminateWorkloadRequest{ActorDirs: dirs, RunscPath: "/bin/sh"}) return err }, } { @@ -111,7 +111,7 @@ func TestRestoreWorkloadRejectsUnsupportedFidelity(t *testing.T) { if err := os.WriteFile(runscPath, nil, 0o755); err != nil { t.Fatal(err) } - dirs := &ateompb.ActorDirs{ + dirs := &ateworkerpb.ActorDirs{ RootDir: "/node/actors/actor-a", OciBundleDir: "/node/actors/actor-a/bundle", CheckpointDir: "/node/actors/actor-a/checkpoint-state", @@ -121,10 +121,10 @@ func TestRestoreWorkloadRejectsUnsupportedFidelity(t *testing.T) { VolumesDir: "/node/actors/actor-a/volumes", } // A zero service cannot acquire locks, configure networking, or open log pipes. - s := &AteomService{} - for _, fidelity := range []ateompb.SnapshotFidelity{0, ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, 99} { + s := &AteWorkerService{} + for _, fidelity := range []ateworkerpb.SnapshotFidelity{0, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, 99} { t.Run(fidelity.String(), func(t *testing.T) { - resp, err := s.RestoreWorkload(context.Background(), &ateompb.RestoreWorkloadRequest{ + resp, err := s.RestoreWorkload(context.Background(), &ateworkerpb.RestoreWorkloadRequest{ ActorDirs: dirs, RunscPath: runscPath, Fidelity: fidelity, diff --git a/cmd/ateom-gvisor/main_unsupported.go b/cmd/ateworker-gvisor/main_unsupported.go similarity index 90% rename from cmd/ateom-gvisor/main_unsupported.go rename to cmd/ateworker-gvisor/main_unsupported.go index a62f78406e..3b18f364f5 100644 --- a/cmd/ateom-gvisor/main_unsupported.go +++ b/cmd/ateworker-gvisor/main_unsupported.go @@ -22,6 +22,6 @@ import ( ) func main() { - fmt.Fprintln(os.Stderr, "ateom-gvisor is only supported on Linux") + fmt.Fprintln(os.Stderr, "ateworker-gvisor is only supported on Linux") os.Exit(1) } diff --git a/cmd/ateom-gvisor/phaselog.go b/cmd/ateworker-gvisor/phaselog.go similarity index 91% rename from cmd/ateom-gvisor/phaselog.go rename to cmd/ateworker-gvisor/phaselog.go index 6c73cca5c0..c7190d23dd 100644 --- a/cmd/ateom-gvisor/phaselog.go +++ b/cmd/ateworker-gvisor/phaselog.go @@ -20,10 +20,10 @@ import ( "time" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/ateomphaselog" + "github.com/agent-substrate/substrate/internal/ateworker/phaselog" ) -// The phase names, suffixed onto the ateomphaselog duration keys. Kept out of +// The phase names, suffixed onto the phaselog duration keys. Kept out of // ateattr on purpose: that package's SnapshotPhase* values are the // ate.snapshot.phase metric enum, and these are not values of it. "total" is // shared so the two layers' records agree on the denominator. @@ -66,8 +66,8 @@ type restoreTiming struct { wakeupProbe, activate, total time.Duration } -func (t restoreTiming) phases() []ateomphaselog.Phase { - return []ateomphaselog.Phase{ +func (t restoreTiming) phases() []phaselog.Phase { + return []phaselog.Phase{ {Name: phasePrep, D: t.prep}, {Name: phaseEgressPrepare, D: t.egressPrepare}, {Name: phaseNetSetup, D: t.netSetup}, @@ -89,8 +89,8 @@ type checkpointTiming struct { prep, checkpoint, durableDir, teardown, total time.Duration } -func (t checkpointTiming) phases() []ateomphaselog.Phase { - return []ateomphaselog.Phase{ +func (t checkpointTiming) phases() []phaselog.Phase { + return []phaselog.Phase{ {Name: phasePrep, D: t.prep}, {Name: phaseCheckpoint, D: t.checkpoint}, {Name: phaseDurableDir, D: t.durableDir}, diff --git a/cmd/ateom-gvisor/phaselog_test.go b/cmd/ateworker-gvisor/phaselog_test.go similarity index 71% rename from cmd/ateom-gvisor/phaselog_test.go rename to cmd/ateworker-gvisor/phaselog_test.go index e8de62c51f..55d750159c 100644 --- a/cmd/ateom-gvisor/phaselog_test.go +++ b/cmd/ateworker-gvisor/phaselog_test.go @@ -24,8 +24,8 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/internal/ateomphaselog" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/ateworker/phaselog" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" ) @@ -61,8 +61,8 @@ func TestRestoreTimingPhases(t *testing.T) { activate: 2 * time.Millisecond, total: 2 * time.Second, } - attrs := ateomphaselog.SnapshotPhaseAttrs(resources.ActorAttribution{UID: "uid-abc"}, - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateomphaselog.RestoreDurationKey, nil, timing.phases()) + attrs := phaselog.SnapshotPhaseAttrs(resources.ActorAttribution{UID: "uid-abc"}, + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, phaselog.RestoreDurationKey, nil, timing.phases()) attrs = append(attrs, slog.Int(containerCountKey, 2)) seen := make(map[string]bool, len(attrs)) @@ -75,19 +75,19 @@ func TestRestoreTimingPhases(t *testing.T) { rec := renderPhaseRecord(t, attrs) for k, want := range map[string]float64{ - "ateom.actor.restore.duration.prep": 0.01, - "ateom.actor.restore.duration.egress_prepare": 0.12, - "ateom.actor.restore.duration.net_setup": 0.03, - "ateom.actor.restore.duration.pause_rootfs": 0.005, - "ateom.actor.restore.duration.pause_create": 0.2, - "ateom.actor.restore.duration.pause_restore": 0.9, - "ateom.actor.restore.duration.app_rootfs": 0.015, - "ateom.actor.restore.duration.app_create": 0.05, - "ateom.actor.restore.duration.app_restore": 0.4, - "ateom.actor.restore.duration.wakeup_probe": 0.25, - "ateom.actor.restore.duration.activate": 0.002, - "ateom.actor.restore.duration.total": 2, - "ate.actor.container.count": 2, + "ateworker.actor.restore.duration.prep": 0.01, + "ateworker.actor.restore.duration.egress_prepare": 0.12, + "ateworker.actor.restore.duration.net_setup": 0.03, + "ateworker.actor.restore.duration.pause_rootfs": 0.005, + "ateworker.actor.restore.duration.pause_create": 0.2, + "ateworker.actor.restore.duration.pause_restore": 0.9, + "ateworker.actor.restore.duration.app_rootfs": 0.015, + "ateworker.actor.restore.duration.app_create": 0.05, + "ateworker.actor.restore.duration.app_restore": 0.4, + "ateworker.actor.restore.duration.wakeup_probe": 0.25, + "ateworker.actor.restore.duration.activate": 0.002, + "ateworker.actor.restore.duration.total": 2, + "ate.actor.container.count": 2, } { got, ok := rec[k].(float64) if !ok { @@ -115,14 +115,14 @@ func TestCheckpointTimingPhases(t *testing.T) { teardown: 150 * time.Millisecond, total: 500 * time.Millisecond, } - rec := renderPhaseRecord(t, ateomphaselog.SnapshotPhaseAttrs(resources.ActorAttribution{UID: "uid-abc"}, - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, ateomphaselog.CheckpointDurationKey, nil, timing.phases())) + rec := renderPhaseRecord(t, phaselog.SnapshotPhaseAttrs(resources.ActorAttribution{UID: "uid-abc"}, + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, phaselog.CheckpointDurationKey, nil, timing.phases())) for k, want := range map[string]float64{ - "ateom.actor.checkpoint.duration.prep": 0.02, - "ateom.actor.checkpoint.duration.durable_dir": 0.3, - "ateom.actor.checkpoint.duration.teardown": 0.15, - "ateom.actor.checkpoint.duration.total": 0.5, + "ateworker.actor.checkpoint.duration.prep": 0.02, + "ateworker.actor.checkpoint.duration.durable_dir": 0.3, + "ateworker.actor.checkpoint.duration.teardown": 0.15, + "ateworker.actor.checkpoint.duration.total": 0.5, } { got, ok := rec[k].(float64) if !ok { @@ -131,7 +131,7 @@ func TestCheckpointTimingPhases(t *testing.T) { t.Errorf("%s = %v, want %v", k, got, want) } } - if v, ok := rec["ateom.actor.checkpoint.duration.checkpoint"]; ok { + if v, ok := rec["ateworker.actor.checkpoint.duration.checkpoint"]; ok { t.Errorf("checkpoint present with %v, want absent", v) } if rec["ate.snapshot.fidelity"] != "volumes" { diff --git a/cmd/ateom-gvisor/runsc.go b/cmd/ateworker-gvisor/runsc.go similarity index 95% rename from cmd/ateom-gvisor/runsc.go rename to cmd/ateworker-gvisor/runsc.go index 732c7dc950..6b5da7101e 100644 --- a/cmd/ateom-gvisor/runsc.go +++ b/cmd/ateworker-gvisor/runsc.go @@ -29,7 +29,7 @@ import ( "strings" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/sizing" ) @@ -38,13 +38,13 @@ type runsc struct { path string actorUID string // actorDirs are the actor's directories, from the request. - actorDirs *ateompb.ActorDirs + actorDirs *ateworkerpb.ActorDirs // size is the actor's declared limits. size sizing.SandboxSize // durableVolumes are the durable-dir volume names declared to the sandbox. durableVolumes []string // containers are the workload's containers, from the request. - containers []*ateompb.Container + containers []*ateworkerpb.Container // flags holds the configured flags; must not be nil. flags *flagStore } @@ -73,7 +73,7 @@ func (r *runsc) buildArgs(command, containerName string, extraArgs ...string) [] // durableVolumeNames returns the sorted, deduplicated durable-dir volume names // mounted by workload containers. -func durableVolumeNames(spec *ateompb.WorkloadSpec) []string { +func durableVolumeNames(spec *ateworkerpb.WorkloadSpec) []string { var names []string for _, c := range spec.GetContainers() { for _, m := range c.GetDurableDirVolumeMounts() { @@ -86,10 +86,10 @@ func durableVolumeNames(spec *ateompb.WorkloadSpec) []string { // pauseContainer is the sandbox root container. It only reaps, so it gets no // capabilities and mounts nothing. -func pauseContainer() *ateompb.Container { - return &ateompb.Container{ +func pauseContainer() *ateworkerpb.Container { + return &ateworkerpb.Container{ Name: ocispec.PauseContainer, - ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/pause"}}, + ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/pause"}}, } } @@ -98,7 +98,7 @@ func pauseContainer() *ateompb.Container { func (r *runsc) writeSpec(containerName string) error { container := pauseContainer() if containerName != ocispec.PauseContainer { - i := slices.IndexFunc(r.containers, func(c *ateompb.Container) bool { return c.GetName() == containerName }) + i := slices.IndexFunc(r.containers, func(c *ateworkerpb.Container) bool { return c.GetName() == containerName }) if i < 0 { return fmt.Errorf("container %q is not in the workload spec", containerName) } @@ -297,7 +297,7 @@ func (r *runsc) cmdKill(ctx context.Context, containerName, signal string) error } // cmdWait blocks until the given container's process exits. Used during -// graceful shutdown to confirm the actor has stopped before ateom exits. +// graceful shutdown to confirm the actor has stopped before ateworker exits. func (r *runsc) cmdWait(ctx context.Context, containerName string) error { slog.InfoContext(ctx, "About to run runsc wait", slog.String("container", containerName)) diff --git a/cmd/ateom-gvisor/runsc_test.go b/cmd/ateworker-gvisor/runsc_test.go similarity index 92% rename from cmd/ateom-gvisor/runsc_test.go rename to cmd/ateworker-gvisor/runsc_test.go index 3695aa8a80..94deffdbd7 100644 --- a/cmd/ateom-gvisor/runsc_test.go +++ b/cmd/ateworker-gvisor/runsc_test.go @@ -31,23 +31,23 @@ import ( "github.com/agent-substrate/substrate/internal/actorlock" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) -var testActorDirs = &ateompb.ActorDirs{ +var testActorDirs = &ateworkerpb.ActorDirs{ RootDir: "/node/actors/test-actor-123", OciBundleDir: "/node/actors/test-actor-123/bundle", } // writeSpec builds a workload container's spec from the request and the pause -// container's from ateom's own description. +// container's from ateworker's own description. func TestWriteSpec(t *testing.T) { root := t.TempDir() - actorDirs := &ateompb.ActorDirs{RootDir: root, OciBundleDir: filepath.Join(root, "bundles")} + actorDirs := &ateworkerpb.ActorDirs{RootDir: root, OciBundleDir: filepath.Join(root, "bundles")} r := &runsc{ actorUID: "uid-a", actorDirs: actorDirs, - containers: []*ateompb.Container{{Name: "app", ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app", "serve"}}}}, + containers: []*ateworkerpb.Container{{Name: "app", ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app", "serve"}}}}, } for name, wantArgs := range map[string][]string{ "app": {"/app", "serve"}, @@ -258,24 +258,24 @@ func TestRestoreArgs(t *testing.T) { } func TestRestoreWorkloadRejectsEmptyRestoreDir(t *testing.T) { - s := &AteomService{ + s := &AteWorkerService{ locks: actorlock.New(), inFlight: actorlock.NewInFlight(), } for _, tc := range []struct { name string - req *ateompb.RestoreWorkloadRequest + req *ateworkerpb.RestoreWorkloadRequest }{ { name: "nil actor_dirs", - req: &ateompb.RestoreWorkloadRequest{ActorUid: "actor-a"}, + req: &ateworkerpb.RestoreWorkloadRequest{ActorUid: "actor-a"}, }, { name: "empty restore_dir", - req: &ateompb.RestoreWorkloadRequest{ + req: &ateworkerpb.RestoreWorkloadRequest{ ActorUid: "actor-a", - ActorDirs: &ateompb.ActorDirs{ + ActorDirs: &ateworkerpb.ActorDirs{ RootDir: "/node/actors/actor-a", OciBundleDir: "/node/actors/actor-a/bundle", CheckpointDir: "/node/actors/actor-a/checkpoint-state", diff --git a/cmd/ateom-gvisor/sandboxexit.go b/cmd/ateworker-gvisor/sandboxexit.go similarity index 94% rename from cmd/ateom-gvisor/sandboxexit.go rename to cmd/ateworker-gvisor/sandboxexit.go index bc6c2eaf8d..1694a06b87 100644 --- a/cmd/ateom-gvisor/sandboxexit.go +++ b/cmd/ateworker-gvisor/sandboxexit.go @@ -26,7 +26,7 @@ import ( "golang.org/x/sys/unix" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // sandboxExitTimeout bounds the wait for a checkpointed sandbox to go away. @@ -42,7 +42,7 @@ type sandboxProcess struct { // openSandboxProcess pins the sandbox before it can exit, so a reused pid is // never mistaken for it. -func openSandboxProcess(actorDirs *ateompb.ActorDirs) (*sandboxProcess, error) { +func openSandboxProcess(actorDirs *ateworkerpb.ActorDirs) (*sandboxProcess, error) { b, err := os.ReadFile(pidFilePath(actorDirs, ocispec.PauseContainer)) if err != nil { return nil, err diff --git a/cmd/ateom-gvisor/sandboxexit_test.go b/cmd/ateworker-gvisor/sandboxexit_test.go similarity index 94% rename from cmd/ateom-gvisor/sandboxexit_test.go rename to cmd/ateworker-gvisor/sandboxexit_test.go index 5320bd07c6..2ae50e2aa6 100644 --- a/cmd/ateom-gvisor/sandboxexit_test.go +++ b/cmd/ateworker-gvisor/sandboxexit_test.go @@ -25,7 +25,7 @@ import ( "time" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // startFakeSandbox starts a process that runs until killed and records it as @@ -40,7 +40,7 @@ func startFakeSandbox(t *testing.T) (*exec.Cmd, *sandboxProcess) { _ = cmd.Process.Kill() _, _ = cmd.Process.Wait() }) - actorDirs := &ateompb.ActorDirs{RootDir: t.TempDir()} + actorDirs := &ateworkerpb.ActorDirs{RootDir: t.TempDir()} if err := os.MkdirAll(pidFileDir(actorDirs), 0o700); err != nil { t.Fatal(err) } diff --git a/cmd/ateom-gvisor/sandboxnet.go b/cmd/ateworker-gvisor/sandboxnet.go similarity index 82% rename from cmd/ateom-gvisor/sandboxnet.go rename to cmd/ateworker-gvisor/sandboxnet.go index 755e1b2ec6..5da536dddc 100644 --- a/cmd/ateom-gvisor/sandboxnet.go +++ b/cmd/ateworker-gvisor/sandboxnet.go @@ -25,13 +25,13 @@ import ( "os" "path/filepath" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomnet/dns" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // actorResolvConf writes the resolver bind source outside the actor's rootfs. -func actorResolvConf(actorDirs *ateompb.ActorDirs) (string, error) { +func actorResolvConf(actorDirs *ateworkerpb.ActorDirs) (string, error) { pod, err := os.ReadFile("/etc/resolv.conf") if err != nil { return "", fmt.Errorf("reading the worker pod resolv.conf: %w", err) @@ -40,7 +40,7 @@ func actorResolvConf(actorDirs *ateompb.ActorDirs) (string, error) { if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil { return "", fmt.Errorf("creating the actor directory: %w", err) } - if err := os.WriteFile(path, dns.SandboxResolvConf(ateomnet.ActorVethGateway, pod), 0o644); err != nil { + if err := os.WriteFile(path, dns.SandboxResolvConf(actornet.ActorVethGateway, pod), 0o644); err != nil { return "", fmt.Errorf("writing the actor resolv.conf: %w", err) } return path, nil diff --git a/cmd/ateom-gvisor/shutdown_test.go b/cmd/ateworker-gvisor/shutdown_test.go similarity index 98% rename from cmd/ateom-gvisor/shutdown_test.go rename to cmd/ateworker-gvisor/shutdown_test.go index c84fa12dfb..3a657a0451 100644 --- a/cmd/ateom-gvisor/shutdown_test.go +++ b/cmd/ateworker-gvisor/shutdown_test.go @@ -170,7 +170,7 @@ func TestKillContainer(t *testing.T) { } // TestKillContainerHonorsParentCancellation asserts that a cancelled shutdown -// context stops the drain instead of escalating: ateom is going away anyway, and +// context stops the drain instead of escalating: ateworker is going away anyway, and // the containers go down with the pod. func TestKillContainerHonorsParentCancellation(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) diff --git a/cmd/ateom-gvisor/stats.go b/cmd/ateworker-gvisor/stats.go similarity index 83% rename from cmd/ateom-gvisor/stats.go rename to cmd/ateworker-gvisor/stats.go index c2b56aec16..54ad824491 100644 --- a/cmd/ateom-gvisor/stats.go +++ b/cmd/ateworker-gvisor/stats.go @@ -24,11 +24,11 @@ import ( "path/filepath" "time" - "github.com/agent-substrate/substrate/cmd/ateom-gvisor/internal/cgroupstats" + "github.com/agent-substrate/substrate/cmd/ateworker-gvisor/internal/cgroupstats" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" ) @@ -36,7 +36,7 @@ import ( // defaultCgroupRoot is the worker pod's own cgroup scope. The worker runs in a // private cgroup namespace, so this path is the pod's cgroup rather than the // host root, and runsc's per-container leaves are its direct children (see -// ateomcgroup.Delegate). +// cgroup.Delegate). const defaultCgroupRoot = "/sys/fs/cgroup" // sandboxCgroupContainer is the container whose cgroup leaf accounts for the @@ -48,12 +48,12 @@ const defaultCgroupRoot = "/sys/fs/cgroup" // and gofer spawn in cgroup.RunInCgroup — so the sentry lands in the leaf of // the pause container, the first one RunWorkload and RestoreWorkload create. // -// The leaf is a direct child of the delegated scope rather than of ateom's own +// The leaf is a direct child of the delegated scope rather than of ateworker's own // cgroup, because runsc resolves cgroupsPath against the parent of the cgroup // it is running in: cgroup v2 forbids a cgroup from holding processes and // delegating controllers to children at once, so runsc walks up one level to -// find a directory it is allowed to create in. ateomcgroup.Delegate moves -// ateom into /sys/fs/cgroup/ateom precisely so that one level up is the +// find a directory it is allowed to create in. cgroup.Delegate moves +// ateworker into /sys/fs/cgroup/ateworker precisely so that one level up is the // delegated scope. // // The actor's own containers get leaves too, since cmdCreate shapes a @@ -88,12 +88,12 @@ const defaultCgroupRoot = "/sys/fs/cgroup" // ocispec.GVisorCgroupLeaf supplies the leaf name for both shaping and stats. const sandboxCgroupContainer = ocispec.PauseContainer -// GetWorkloadStats implements ateompb.Ateom/GetWorkloadStats. +// GetWorkloadStats implements ateworkerpb.Worker/GetWorkloadStats. // // It must not take the actor's lifecycle lock, which is held across whole // boots and checkpoints: polls would stall behind runsc, and a checkpoint // would wait on telemetry. The cgroup files are read with no lock held. -func (s *AteomService) GetWorkloadStats(ctx context.Context, req *ateompb.GetWorkloadStatsRequest) (*ateompb.GetWorkloadStatsResponse, error) { +func (s *AteWorkerService) GetWorkloadStats(ctx context.Context, req *ateworkerpb.GetWorkloadStatsRequest) (*ateworkerpb.GetWorkloadStatsResponse, error) { if req.GetActorUid() == "" { return nil, apierror.InvalidArgument("actor_uid is required") } @@ -103,14 +103,14 @@ func (s *AteomService) GetWorkloadStats(ctx context.Context, req *ateompb.GetWor // worker-to-actor mapping wants re-resolving. hosted := s.lookupActor(req.GetActorUid()) if hosted == nil { - return nil, apierror.NotFound("ateom is not executing actor %q", req.GetActorUid()) + return nil, apierror.NotFound("worker is not executing actor %q", req.GetActorUid()) } // The raw reading is dropped: the dead-sandbox check runs in the usage // sweep, which reads every hosted actor. sample, _, err := s.measure(ctx, hosted) if err != nil { // The requested actor is the active one but its cgroup is not there. - // Most often that is a poll landing in the boot: the ateom retains the + // Most often that is a poll landing in the boot: the ateworker retains the // attribution from the moment it accepts the actor, before runsc has // created the leaf. The other way in is a sandbox that went away // underneath the read, which the next CheckpointWorkload turns into the @@ -127,20 +127,20 @@ func (s *AteomService) GetWorkloadStats(ctx context.Context, req *ateompb.GetWor // can land underneath it. Pointer identity catches that: hostActor stores // a new record every time. if s.lookupActor(req.GetActorUid()) != hosted { - return nil, apierror.NotFound("ateom stopped executing actor %q while the sample was being taken", req.GetActorUid()) + return nil, apierror.NotFound("worker stopped executing actor %q while the sample was being taken", req.GetActorUid()) } // The reading advanced the activation's CPU, so the cache keeps up with it. hosted.usage.Store(sample) - return &ateompb.GetWorkloadStatsResponse{Sample: sample}, nil + return &ateworkerpb.GetWorkloadStatsResponse{Sample: sample}, nil } -// GetActiveWorkloadStats implements ateompb.Ateom/GetActiveWorkloadStats: the +// GetActiveWorkloadStats implements ateworkerpb.Worker/GetActiveWorkloadStats: the // discovery read. It serves each actor's latest sample from the sampler, so a // poll costs no read of its own; an actor not sampled yet is pending. -func (s *AteomService) GetActiveWorkloadStats(ctx context.Context, req *ateompb.GetActiveWorkloadStatsRequest) (*ateompb.GetActiveWorkloadStatsResponse, error) { +func (s *AteWorkerService) GetActiveWorkloadStats(ctx context.Context, req *ateworkerpb.GetActiveWorkloadStatsRequest) (*ateworkerpb.GetActiveWorkloadStatsResponse, error) { hosted := s.hostedActors() - samples := make([]*ateompb.WorkloadStatsSample, 0, len(hosted)) + samples := make([]*ateworkerpb.WorkloadStatsSample, 0, len(hosted)) for _, h := range hosted { sample := h.usage.Latest() if sample == nil { @@ -151,13 +151,13 @@ func (s *AteomService) GetActiveWorkloadStats(ctx context.Context, req *ateompb. // An empty list is "available", per the proto: a normal answer for a // scraper to get, not an error. - return &ateompb.GetActiveWorkloadStatsResponse{Samples: samples}, nil + return &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: samples}, nil } // sweepUsage samples every hosted actor between its initial reading and its // final record, and stores each sample and writes its periodic record. Same // lock discipline as GetWorkloadStats, for the same reasons. -func (s *AteomService) sweepUsage(ctx context.Context) { +func (s *AteWorkerService) sweepUsage(ctx context.Context) { for _, h := range s.hostedActors() { if !h.usage.Sampling() { continue @@ -189,7 +189,7 @@ func (s *AteomService) sweepUsage(ctx context.Context) { // recordInitial samples a new activation once its sandbox is up and writes its // initial record. -func (s *AteomService) recordInitial(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) recordInitial(ctx context.Context, h *hostedActor) { sample, _, err := s.measure(ctx, h) if err != nil { slog.WarnContext(ctx, "No initial usage sample", slog.String(string(ateattr.ActorUIDKey), h.attribution.UID), slog.Any("err", err)) @@ -201,7 +201,7 @@ func (s *AteomService) recordInitial(ctx context.Context, h *hostedActor) { // readFinal samples an activation while its sandbox still exists, for the final // record a successful checkpoint or terminate writes. A failed read leaves the // newest measured sample to stand in. -func (s *AteomService) readFinal(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) readFinal(ctx context.Context, h *hostedActor) { if sample, _, err := s.measure(ctx, h); err == nil { h.usage.Store(sample) } @@ -210,7 +210,7 @@ func (s *AteomService) readFinal(ctx context.Context, h *hostedActor) { // recordFinalIfEnded writes h's final record if a checkpoint or terminate tore // its activation down, which unhosts the actor even when a later step fails. // The caller holds the actor's lock. -func (s *AteomService) recordFinalIfEnded(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) recordFinalIfEnded(ctx context.Context, h *hostedActor) { if h != nil && s.lookupActor(h.attribution.UID) != h { s.recordFinal(ctx, h) } @@ -218,9 +218,9 @@ func (s *AteomService) recordFinalIfEnded(ctx context.Context, h *hostedActor) { // recordFinal writes the final record of an activation that a checkpoint or a // terminate ended, from its newest measured sample. -func (s *AteomService) recordFinal(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) recordFinal(ctx context.Context, h *hostedActor) { pending := h.usage.WithEpoch(pendingSample(&h.attribution)) - h.usage.Final(func(measured *ateompb.WorkloadStatsSample) { + h.usage.Final(func(measured *ateworkerpb.WorkloadStatsSample) { if measured == nil { measured = pending } @@ -232,8 +232,8 @@ func (s *AteomService) recordFinal(ctx context.Context, h *hostedActor) { // read reports it: attribution and the runtime family, measurements absent -- // source stays STATS_SOURCE_UNSPECIFIED, which the sample's contract defines // as "not measured" rather than "measured as zero". -func pendingSample(active *resources.ActorAttribution) *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{ +func pendingSample(active *resources.ActorAttribution) *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{ Atespace: active.Ref.Atespace, ActorName: active.Ref.Name, ActorUid: active.UID, @@ -248,9 +248,9 @@ func pendingSample(active *resources.ActorAttribution) *ateompb.WorkloadStatsSam // measure reads h's sandbox cgroup as a reading of its activation, returning // the raw cgroup reading alongside it for the dead-sandbox check. -func (s *AteomService) measure(ctx context.Context, h *hostedActor) (*ateompb.WorkloadStatsSample, cgroupstats.Sample, error) { +func (s *AteWorkerService) measure(ctx context.Context, h *hostedActor) (*ateworkerpb.WorkloadStatsSample, cgroupstats.Sample, error) { var raw cgroupstats.Sample - sample, err := h.usage.Measure(ctx, func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { + sample, err := h.usage.Measure(ctx, func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { sample, r, err := s.sampleSandbox(&h.attribution) raw = r return sample, nil, err @@ -265,7 +265,7 @@ func (s *AteomService) measure(ctx context.Context, h *hostedActor) (*ateompb.Wo // pending sample for the sweep. // The read holds no lifecycle lock, so the keyed caller re-checks the actor // record it loaded after this returns. -func (s *AteomService) sampleSandbox(active *resources.ActorAttribution) (*ateompb.WorkloadStatsSample, cgroupstats.Sample, error) { +func (s *AteWorkerService) sampleSandbox(active *resources.ActorAttribution) (*ateworkerpb.WorkloadStatsSample, cgroupstats.Sample, error) { read := s.readSandboxCgroup if read == nil { read = cgroupstats.Read @@ -276,7 +276,7 @@ func (s *AteomService) sampleSandbox(active *resources.ActorAttribution) (*ateom return nil, cgroupstats.Sample{}, err } - return &ateompb.WorkloadStatsSample{ + return &ateworkerpb.WorkloadStatsSample{ Atespace: active.Ref.Atespace, ActorName: active.Ref.Name, ActorUid: active.UID, @@ -284,7 +284,7 @@ func (s *AteomService) sampleSandbox(active *resources.ActorAttribution) (*ateom ActorTemplateName: active.TemplateName, SandboxClass: string(atev1alpha1.SandboxClassGvisor), - Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, + Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, MemoryCurrentBytes: sample.MemoryCurrentBytes, MemoryPeakBytes: sample.MemoryPeakBytes, @@ -306,7 +306,7 @@ func (s *AteomService) sampleSandbox(active *resources.ActorAttribution) (*ateom // // oom_kill counts over the cgroup's lifetime. A leaf left behind by an earlier // activation of the same actor carries its count forward. -func (s *AteomService) warnDeadSandbox(ctx context.Context, h *hostedActor, raw cgroupstats.Sample) { +func (s *AteWorkerService) warnDeadSandbox(ctx context.Context, h *hostedActor, raw cgroupstats.Sample) { if h.deadReported.Load() { return } diff --git a/cmd/ateom-gvisor/stats_test.go b/cmd/ateworker-gvisor/stats_test.go similarity index 89% rename from cmd/ateom-gvisor/stats_test.go rename to cmd/ateworker-gvisor/stats_test.go index aaa3b84bb5..e03ca0c564 100644 --- a/cmd/ateom-gvisor/stats_test.go +++ b/cmd/ateworker-gvisor/stats_test.go @@ -32,12 +32,12 @@ import ( "google.golang.org/grpc/codes" "google.golang.org/protobuf/testing/protocmp" - "github.com/agent-substrate/substrate/cmd/ateom-gvisor/internal/cgroupstats" + "github.com/agent-substrate/substrate/cmd/ateworker-gvisor/internal/cgroupstats" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/ateomstats" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/internal/workerstats" ) // The lifecycle RPCs that host and unhost actors need netlink, runsc, and @@ -64,13 +64,13 @@ var healthyCgroup = map[string]string{ // newStatsService builds a service whose cgroup root is a fixture tree. A nil // files map leaves the sandbox cgroup directory absent entirely, which is what // a torn-down sandbox looks like. -func newStatsService(t *testing.T, files map[string]string) *AteomService { +func newStatsService(t *testing.T, files map[string]string) *AteWorkerService { t.Helper() root := t.TempDir() if files != nil { writeCgroupFixture(t, root, ocispec.GVisorCgroupLeaf(testActor.UID, sandboxCgroupContainer), files) } - return &AteomService{ + return &AteWorkerService{ locks: actorlock.New(), actors: map[string]*hostedActor{}, maxActors: 1000, @@ -92,9 +92,9 @@ func writeCgroupFixture(t *testing.T, root, leaf string, files map[string]string } } -// setHostedActor makes the ateom host exactly this actor, the way RunWorkload +// setHostedActor makes the ateworker host exactly this actor, the way RunWorkload // would, or nothing when attribution is nil. -func setHostedActor(s *AteomService, attribution *resources.ActorAttribution) { +func setHostedActor(s *AteWorkerService, attribution *resources.ActorAttribution) { s.actorsMu.Lock() defer s.actorsMu.Unlock() s.actors = map[string]*hostedActor{} @@ -108,7 +108,7 @@ func TestGetWorkloadStats(t *testing.T) { setHostedActor(s, &testActor) before := time.Now().UnixNano() - got, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + got, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) after := time.Now().UnixNano() if err != nil { t.Fatalf("GetWorkloadStats() error = %v, want nil", err) @@ -120,14 +120,14 @@ func TestGetWorkloadStats(t *testing.T) { // Checked above; zeroed so the rest can be compared as a whole. got.GetSample().ObservedAtUnixNano = 0 - want := &ateompb.GetWorkloadStatsResponse{Sample: &ateompb.WorkloadStatsSample{ + want := &ateworkerpb.GetWorkloadStatsResponse{Sample: &ateworkerpb.WorkloadStatsSample{ Atespace: "space-a", ActorName: "actor-a", ActorUid: "uid-a", ActorTemplateAtespace: "ns-a", ActorTemplateName: "template-a", SandboxClass: "gvisor", - Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, + Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, MemoryCurrentBytes: 157286400, MemoryPeakBytes: 209715200, MemoryWorkingSetBytes: 136314880, @@ -143,7 +143,7 @@ func TestGetWorkloadStatsErrors(t *testing.T) { name string // files is the fixture sandbox cgroup; nil means the directory is absent. files map[string]string - // active is hosted when non-nil; nil leaves the ateom "available". + // active is hosted when non-nil; nil leaves the ateworker "available". active *resources.ActorAttribution actorUID string want codes.Code @@ -160,7 +160,7 @@ func TestGetWorkloadStatsErrors(t *testing.T) { { // Not here at all. NOT_FOUND rather than FAILED_PRECONDITION, because // what the caller should do about it is re-resolve, not retry. - name: "ateom is available", + name: "ateworker is available", files: healthyCgroup, active: nil, actorUID: "uid-a", @@ -203,7 +203,7 @@ func TestGetWorkloadStatsErrors(t *testing.T) { setHostedActor(s, tc.active) } - resp, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: tc.actorUID}) + resp, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: tc.actorUID}) if resp != nil { t.Errorf("GetWorkloadStats() returned response %v, want nil", resp) } @@ -227,18 +227,18 @@ func TestGetWorkloadStatsDoesNotTakeLock(t *testing.T) { } defer s.locks.Unlock(testActor.UID) - if _, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}); err != nil { + if _, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}); err != nil { t.Errorf("GetWorkloadStats() error = %v, want nil", err) } } -// TestAteomServiceStartsAvailable checks that a freshly constructed service +// TestAteWorkerServiceStartsAvailable checks that a freshly constructed service // retains no attribution. GetWorkloadStats's NOT_FOUND-when-available behavior -// is built on this: a non-nil zero value here would make an idle ateom report +// is built on this: a non-nil zero value here would make an idle ateworker report // an empty actor's usage instead of refusing. -func TestAteomServiceStartsAvailable(t *testing.T) { - if got := (&AteomService{}).hostedActors(); len(got) != 0 { - t.Errorf("new AteomService hosts %v, want nothing", got) +func TestAteWorkerServiceStartsAvailable(t *testing.T) { + if got := (&AteWorkerService{}).hostedActors(); len(got) != 0 { + t.Errorf("new AteWorkerService hosts %v, want nothing", got) } } @@ -257,7 +257,7 @@ func TestGetActiveWorkloadStats(t *testing.T) { // The keyed read against the same fixture is the reference: the discovery // read must produce the identical sample, since both are the same // measurement with a different addressing mode. - want, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + want, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) if err != nil { t.Fatalf("GetWorkloadStats() error = %v, want nil", err) } @@ -270,24 +270,24 @@ func TestGetActiveWorkloadStats(t *testing.T) { } // TestGetActiveWorkloadStatsAvailable pins the contract that makes the -// discovery read scrapeable: an idle ateom is an empty list, never an error. +// discovery read scrapeable: an idle ateworker is an empty list, never an error. func TestGetActiveWorkloadStatsAvailable(t *testing.T) { s := newStatsService(t, healthyCgroup) got, err := sweepAndList(s) if err != nil { - t.Fatalf("GetActiveWorkloadStats() on an available ateom: error = %v, want nil", err) + t.Fatalf("GetActiveWorkloadStats() on an available ateworker: error = %v, want nil", err) } if n := len(got.GetSamples()); n != 0 { - t.Errorf("GetActiveWorkloadStats() on an available ateom = %v, want no samples", got) + t.Errorf("GetActiveWorkloadStats() on an available ateworker = %v, want no samples", got) } } // pendingFor is the entry the discovery read answers for a workload with no // numbers yet: attribution and the runtime family, source UNSPECIFIED, // measurements absent. observed_at is zeroed by the caller before comparing. -func pendingFor(attr resources.ActorAttribution) *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{ +func pendingFor(attr resources.ActorAttribution) *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{ Atespace: attr.Ref.Atespace, ActorName: attr.Ref.Name, ActorUid: attr.UID, @@ -381,7 +381,7 @@ func TestGetWorkloadStatsTransition(t *testing.T) { return cgroupstats.Read(dir) } - _, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + _, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) if got := apierror.Code(err); got != codes.NotFound { t.Errorf("GetWorkloadStats() during transition: code = %v, want %v (err: %v)", got, codes.NotFound, err) } @@ -443,7 +443,7 @@ func TestGetActiveWorkloadStatsOneNoCgroup(t *testing.T) { t.Fatalf("GetActiveWorkloadStats() returned %d samples, want 2: %v", len(got.GetSamples()), got) } for _, sample := range got.GetSamples() { - measured := sample.GetSource() != ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED + measured := sample.GetSource() != ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED if want := sample.GetActorUid() == testActor.UID; measured != want { t.Errorf("actor %q measured = %v, want %v", sample.GetActorUid(), measured, want) } @@ -479,7 +479,7 @@ func TestGetActiveWorkloadStatsOneUnreadable(t *testing.T) { t.Fatalf("GetActiveWorkloadStats() returned %d samples, want 2: %v", len(got.GetSamples()), got) } for _, sample := range got.GetSamples() { - measured := sample.GetSource() != ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED + measured := sample.GetSource() != ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED if want := sample.GetActorUid() == testActor.UID; measured != want { t.Errorf("actor %q measured = %v, want %v", sample.GetActorUid(), measured, want) } @@ -490,16 +490,16 @@ func TestGetActiveWorkloadStatsOneUnreadable(t *testing.T) { // so the sweep samples it. It starts at the unix epoch, so a sample's epoch is // zero and the expected samples here need not name it; the epoch has tests of // its own. -func testActivation() *ateomstats.Activation { - a := ateomstats.NewActivation(time.Unix(0, 0), false) +func testActivation() *workerstats.Activation { + a := workerstats.NewActivation(time.Unix(0, 0), false) a.Initial(nil, nil) return a } // sweepAndList runs one sampler sweep, then the discovery read that serves it. -func sweepAndList(s *AteomService) (*ateompb.GetActiveWorkloadStatsResponse, error) { +func sweepAndList(s *AteWorkerService) (*ateworkerpb.GetActiveWorkloadStatsResponse, error) { s.sweepUsage(context.Background()) - return s.GetActiveWorkloadStats(context.Background(), &ateompb.GetActiveWorkloadStatsRequest{}) + return s.GetActiveWorkloadStats(context.Background(), &ateworkerpb.GetActiveWorkloadStatsRequest{}) } // deadCgroup is a sandbox leaf after the kernel OOM-killed the sentry: the leaf @@ -668,7 +668,7 @@ func TestDrainNeverKillsPauseContainer(t *testing.T) { if resources.IsValidResourceName(ocispec.PauseContainer) { t.Fatalf("%q is a valid container name; a spec could then put it on the drain's kill list", ocispec.PauseContainer) } - spec := []*ateompb.Container{{Name: "app"}, {Name: "sidecar"}} + spec := []*ateworkerpb.Container{{Name: "app"}, {Name: "sidecar"}} for _, name := range containerNames(spec) { if name == ocispec.PauseContainer { t.Fatalf("containerNames(%v) includes %q", spec, ocispec.PauseContainer) diff --git a/cmd/ateom-gvisor/usage_test.go b/cmd/ateworker-gvisor/usage_test.go similarity index 88% rename from cmd/ateom-gvisor/usage_test.go rename to cmd/ateworker-gvisor/usage_test.go index 3ae9e39aeb..e7f8d997f8 100644 --- a/cmd/ateom-gvisor/usage_test.go +++ b/cmd/ateworker-gvisor/usage_test.go @@ -26,21 +26,21 @@ import ( "github.com/agent-substrate/substrate/internal/actorlock" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/ateomstats" - "github.com/agent-substrate/substrate/internal/ateomstats/ateomstatstest" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" + "github.com/agent-substrate/substrate/internal/workerstats" + "github.com/agent-substrate/substrate/internal/workerstats/workerstatstest" ) -func withUsageRecorder(s *AteomService) *ateomstatstest.Recorder { - e, rec := ateomstatstest.NewEmitter() +func withUsageRecorder(s *AteWorkerService) *workerstatstest.Recorder { + e, rec := workerstatstest.NewEmitter() s.usage = e return rec } // hostWithEpoch hosts testActor; running means its initial reading is done. -func hostWithEpoch(s *AteomService, epoch time.Time, running bool) *hostedActor { - h := &hostedActor{attribution: testActor, usage: ateomstats.NewActivation(epoch, false)} +func hostWithEpoch(s *AteWorkerService, epoch time.Time, running bool) *hostedActor { + h := &hostedActor{attribution: testActor, usage: workerstats.NewActivation(epoch, false)} if running { h.usage.Initial(nil, nil) } @@ -55,16 +55,16 @@ func TestSweepStampsEpochAndDiscoveryServesIt(t *testing.T) { epoch := time.Unix(1700, 0) h := hostWithEpoch(s, epoch, true) - before, err := s.GetActiveWorkloadStats(context.Background(), &ateompb.GetActiveWorkloadStatsRequest{}) + before, err := s.GetActiveWorkloadStats(context.Background(), &ateworkerpb.GetActiveWorkloadStatsRequest{}) if err != nil { t.Fatal(err) } - if got := before.GetSamples()[0]; got.GetSource() != ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED || got.GetEpochUnixNano() != epoch.UnixNano() { + if got := before.GetSamples()[0]; got.GetSource() != ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED || got.GetEpochUnixNano() != epoch.UnixNano() { t.Errorf("before a sweep: source %v epoch %d, want pending with epoch %d", got.GetSource(), got.GetEpochUnixNano(), epoch.UnixNano()) } s.sweepUsage(context.Background()) - after, err := s.GetActiveWorkloadStats(context.Background(), &ateompb.GetActiveWorkloadStatsRequest{}) + after, err := s.GetActiveWorkloadStats(context.Background(), &ateworkerpb.GetActiveWorkloadStatsRequest{}) if err != nil { t.Fatal(err) } @@ -81,7 +81,7 @@ func TestGetWorkloadStatsStampsEpoch(t *testing.T) { s := newStatsService(t, healthyCgroup) epoch := time.Unix(1700, 0) hostWithEpoch(s, epoch, true) - got, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: testActor.UID}) + got, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: testActor.UID}) if err != nil { t.Fatal(err) } @@ -192,7 +192,7 @@ func TestReadFinalWritesNothingUntilRecorded(t *testing.T) { t.Fatalf("records after the pre-snapshot read = %v, want none", got) } read := h.usage.Latest() - if read == nil || read.GetSource() != ateompb.StatsSource_STATS_SOURCE_CGROUP { + if read == nil || read.GetSource() != ateworkerpb.StatsSource_STATS_SOURCE_CGROUP { t.Fatalf("pre-snapshot read stored %v, want a measured sample", read) } if err := os.RemoveAll(filepath.Join(s.cgroupRoot, ocispec.GVisorCgroupLeaf(testActor.UID, sandboxCgroupContainer))); err != nil { @@ -231,7 +231,7 @@ func TestSweepWaitsForInitialAndStopsAtFinal(t *testing.T) { func TestGetWorkloadStatsStoresItsSample(t *testing.T) { s := newStatsService(t, healthyCgroup) h := hostWithEpoch(s, time.Now(), true) - got, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: testActor.UID}) + got, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: testActor.UID}) if err != nil { t.Fatal(err) } diff --git a/cmd/ateom-microvm/actorpath.go b/cmd/ateworker-microvm/actorpath.go similarity index 74% rename from cmd/ateom-microvm/actorpath.go rename to cmd/ateworker-microvm/actorpath.go index a47f8d35f1..4d99fe30bb 100644 --- a/cmd/ateom-microvm/actorpath.go +++ b/cmd/ateworker-microvm/actorpath.go @@ -14,25 +14,25 @@ // See the License for the specific language governing permissions and // limitations under the License. -// The paths ateom-microvm derives from the ActorDirs atelet sends. The -// directory under root_dir is ateom-microvm's own: atelet does not know it. +// The paths ateworker-microvm derives from the ActorDirs atelet sends. The +// directory under root_dir is ateworker-microvm's own: atelet does not know it. package main import ( "path/filepath" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // ociBundlePath is the container's OCI bundle. -func ociBundlePath(actorDirs *ateompb.ActorDirs, containerName string) string { +func ociBundlePath(actorDirs *ateworkerpb.ActorDirs, containerName string) string { return filepath.Join(actorDirs.GetOciBundleDir(), containerName) } // rootfsUpperDir is the host directory backing the actor's rootfs overlay // uppers: one subdirectory per container (see kata.UpperWorkDirs). Local to // this binary: atelet never touches it, so it is not one of the ActorDirs. -func rootfsUpperDir(actorDirs *ateompb.ActorDirs) string { +func rootfsUpperDir(actorDirs *ateworkerpb.ActorDirs) string { return filepath.Join(actorDirs.GetRootDir(), "rootfs-upper") } diff --git a/cmd/ateom-microvm/checkpoint.go b/cmd/ateworker-microvm/checkpoint.go similarity index 87% rename from cmd/ateom-microvm/checkpoint.go rename to cmd/ateworker-microvm/checkpoint.go index bd04c97d85..394a9dee5f 100644 --- a/cmd/ateom-microvm/checkpoint.go +++ b/cmd/ateworker-microvm/checkpoint.go @@ -28,12 +28,12 @@ import ( "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/resources" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/ch" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" - "github.com/agent-substrate/substrate/internal/ateomphaselog" - "github.com/agent-substrate/substrate/internal/ateomstats" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/ch" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" + "github.com/agent-substrate/substrate/internal/ateworker/phaselog" "github.com/agent-substrate/substrate/internal/imagecache" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" + "github.com/agent-substrate/substrate/internal/workerstats" "golang.org/x/sync/errgroup" ) @@ -45,7 +45,7 @@ import ( // What the snapshot holds depends on the requested fidelity; each level // includes the ones below it: // -// - MEMORY: the whole guest. ateom drives the CH REST api-socket: pause -> +// - MEMORY: the whole guest. ateworker drives the CH REST api-socket: pause -> // snapshot file:// (config.json + state.json + sparse // memory-ranges) -> tear the VMM down. Process memory persists via the // memory snapshot; the volume and upper tars ship alongside as for ROOTFS. @@ -65,7 +65,7 @@ import ( // // Allow checkpointing even if the pod is shutting down. This will allow actors // (or the harness) to suspend on shutdown. -func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.CheckpointWorkloadRequest) (_ *ateompb.CheckpointWorkloadResponse, err error) { +func (s *AteWorkerService) CheckpointWorkload(ctx context.Context, req *ateworkerpb.CheckpointWorkloadRequest) (_ *ateworkerpb.CheckpointWorkloadResponse, err error) { if err := validateActorDirs(req.GetActorDirs()); err != nil { return nil, err } @@ -88,11 +88,11 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec // independent observations rather than a partition of the total. tStart := time.Now() var dPrep, dPause, dSnapshot, dDurable, dUpper, dTeardown time.Duration - attribution := ateomstats.ActorAttributionFromRequest(req) + attribution := workerstats.ActorAttributionFromRequest(req) scope := req.GetFidelity() defer func() { - ateomphaselog.LogSnapshotPhases(ctx, "Checkpoint timing breakdown", attribution, scope, - ateomphaselog.CheckpointDurationKey, err, []ateomphaselog.Phase{ + phaselog.LogSnapshotPhases(ctx, "Checkpoint timing breakdown", attribution, scope, + phaselog.CheckpointDurationKey, err, []phaselog.Phase{ {Name: phasePrep, D: dPrep}, {Name: phasePause, D: dPause}, {Name: phaseSnapshot, D: dSnapshot}, @@ -123,8 +123,8 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec durable := hasDurableVolumes(req.GetSpec().GetContainers()) csi := hasCsiVolumes(req.GetSpec().GetContainers()) switch scope { - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS: - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: // TODO: Revisit handling for CSI volumes since snapshots are currently quietly ignored. if !durable && !csi { return nil, apierror.FailedPrecondition( @@ -139,7 +139,7 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec hosted := s.lookupActor(actorUID) // The actor's CH was booted by RunWorkload or relaunched by RestoreWorkload; - // either way ateom owns it and tracks its api-socket. + // either way ateworker owns it and tracks its api-socket. ra := s.runningVM(actorUID) chSocket := kata.CLHSocketPath(actorUID) if ra != nil && ra.apiSocket != "" { @@ -189,7 +189,7 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec shipsRootfs := includesRootfs(scope) var durableFiles []string g, gctx := errgroup.WithContext(ctx) - if scope == ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { + if scope == ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY { g.Go(func() error { t := time.Now() d, err := s.snapshotVMState(gctx, client, ra, actorUID, checkpointDir) @@ -251,13 +251,13 @@ func (s *AteomService) CheckpointWorkload(ctx context.Context, req *ateompb.Chec slog.Duration("durable_dir", dDurable), slog.Duration("rootfs_upper", dUpper), slog.Duration("teardown", dTeardown)) s.recordFinalIfEnded(ctx, hosted) - return &ateompb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles}, nil + return &ateworkerpb.CheckpointWorkloadResponse{SnapshotFiles: snapshotFiles, DataSnapshotFiles: durableFiles}, nil } // snapshotVMState captures the paused guest into checkpointDir: the CH snapshot // (config.json + state.json + memory-ranges) plus the base-id the restore side // needs, and returns how long the snapshot itself took. -func (s *AteomService) snapshotVMState(ctx context.Context, client *ch.Client, ra *runningActor, actorUID, checkpointDir string) (time.Duration, error) { +func (s *AteWorkerService) snapshotVMState(ctx context.Context, client *ch.Client, ra *runningActor, actorUID, checkpointDir string) (time.Duration, error) { // Record the FROZEN base id (the id the guest's virtio-fs find-paths are pinned // to, /rootfs). For a cold-run actor this is its own id; for a restored // actor it is the golden id propagated via ra.baseID (set from the snapshot we @@ -332,9 +332,9 @@ func listFiles(dir string) ([]string, error) { return files, nil } -// teardownActor stops the ateom-owned CH VMM for an actor. ra may be -// nil (e.g. ateom restarted and lost in-memory state). -func (s *AteomService) teardownActor(ctx context.Context, id string, actorDirs *ateompb.ActorDirs, ra *runningActor, client *ch.Client) error { +// teardownActor stops the ateworker-owned CH VMM for an actor. ra may be +// nil (e.g. ateworker restarted and lost in-memory state). +func (s *AteWorkerService) teardownActor(ctx context.Context, id string, actorDirs *ateworkerpb.ActorDirs, ra *runningActor, client *ch.Client) error { // Stop offering the guest to GetWorkloadStats first, before anything below // makes it stop answering. Clearing it here rather than alongside the // attribution is what keeps a poll that lands mid-teardown on the @@ -368,7 +368,7 @@ func (s *AteomService) teardownActor(ctx context.Context, id string, actorDirs * _ = agent.Close() } - // Kill the CH process ateom launched. + // Kill the CH process ateworker launched. if ra.chCmd != nil && ra.chCmd.Process != nil { _ = ra.chCmd.Process.Kill() _, _ = ra.chCmd.Process.Wait() @@ -381,13 +381,13 @@ func (s *AteomService) teardownActor(ctx context.Context, id string, actorDirs * } // Sweep any leftover per-sandbox host-side state + orphaned per-sandbox - // processes. This is ateom's own cleanup (process kill + unmount + rm) — + // processes. This is ateworker's own cleanup (process kill + unmount + rm) — // it also drops the merged rootfs overlay mounts, which MUST come before // the upper-dir removal below (removing a live overlay's upperdir would // corrupt the mount rather than delete the files). s.cleanupSandboxState(ctx, id) - // Remove the rootfs upper dir: ateom owns it — atelet's actor-dir reset + // Remove the rootfs upper dir: ateworker owns it — atelet's actor-dir reset // doesn't know it — and its absence is what marks a worker as holding no // disk-backed upper. Runs after the checkpoint tar, which is already on disk. if err := os.RemoveAll(rootfsUpperDir(actorDirs)); err != nil { @@ -406,7 +406,7 @@ func (s *AteomService) teardownActor(ctx context.Context, id string, actorDirs * // TerminateWorkload stops the running actor, tears down its VMM, and cleans up // networking and overlays. -func (s *AteomService) TerminateWorkload(ctx context.Context, req *ateompb.TerminateWorkloadRequest) (*ateompb.TerminateWorkloadResponse, error) { +func (s *AteWorkerService) TerminateWorkload(ctx context.Context, req *ateworkerpb.TerminateWorkloadRequest) (*ateworkerpb.TerminateWorkloadResponse, error) { if err := validateActorDirs(req.GetActorDirs()); err != nil { return nil, err } @@ -415,7 +415,7 @@ func (s *AteomService) TerminateWorkload(ctx context.Context, req *ateompb.Termi } defer s.locks.Unlock(req.GetActorUid()) - attribution := ateomstats.ActorAttributionFromRequest(req) + attribution := workerstats.ActorAttributionFromRequest(req) hosted := s.lookupActor(attribution.UID) if hosted != nil { @@ -429,7 +429,7 @@ func (s *AteomService) TerminateWorkload(ctx context.Context, req *ateompb.Termi s.actorLogger.EmitLifecycleLog(ctx, "Actor terminated", attribution) - return &ateompb.TerminateWorkloadResponse{}, nil + return &ateworkerpb.TerminateWorkloadResponse{}, nil } // endPreviousActivation ends the activation a re-host is about to replace. A VM @@ -437,7 +437,7 @@ func (s *AteomService) TerminateWorkload(ctx context.Context, req *ateompb.Termi // left running, so it is stopped, read first for the final record. This happens // after a checkpoint that failed before its teardown, or on a retried Run or // Restore. Ending the activation also stops a late recordInitial from writing. -func (s *AteomService) endPreviousActivation(ctx context.Context, actorUID string, actorDirs *ateompb.ActorDirs) error { +func (s *AteWorkerService) endPreviousActivation(ctx context.Context, actorUID string, actorDirs *ateworkerpb.ActorDirs) error { old := s.lookupActor(actorUID) if s.runningVM(actorUID) != nil { if old != nil { @@ -454,7 +454,7 @@ func (s *AteomService) endPreviousActivation(ctx context.Context, actorUID strin } // stopActorVM tears down the actor's micro-VM, if any, keeping it hosted. -func (s *AteomService) stopActorVM(ctx context.Context, actorUID string, actorDirs *ateompb.ActorDirs) error { +func (s *AteWorkerService) stopActorVM(ctx context.Context, actorUID string, actorDirs *ateworkerpb.ActorDirs) error { ra := s.runningVM(actorUID) chSocket := kata.CLHSocketPath(actorUID) if ra != nil && ra.apiSocket != "" { @@ -463,7 +463,7 @@ func (s *AteomService) stopActorVM(ctx context.Context, actorUID string, actorDi return s.teardownActor(ctx, actorUID, actorDirs, ra, ch.NewClient(chSocket)) } -func (s *AteomService) terminateWorkload(ctx context.Context, actor resources.ActorAttribution, actorDirs *ateompb.ActorDirs) error { +func (s *AteWorkerService) terminateWorkload(ctx context.Context, actor resources.ActorAttribution, actorDirs *ateworkerpb.ActorDirs) error { var errs []error if err := s.tunnel.Deactivate(ctx, actor); err != nil { errs = append(errs, fmt.Errorf("while deactivating actor networking: %w", err)) diff --git a/cmd/ateom-microvm/csi.go b/cmd/ateworker-microvm/csi.go similarity index 81% rename from cmd/ateom-microvm/csi.go rename to cmd/ateworker-microvm/csi.go index 18aca8e55b..154d4928f9 100644 --- a/cmd/ateom-microvm/csi.go +++ b/cmd/ateworker-microvm/csi.go @@ -21,13 +21,13 @@ import ( "fmt" "os" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // hasCsiVolumes reports whether any container mounts a CSI volume. -func hasCsiVolumes(containers []*ateompb.Container) bool { +func hasCsiVolumes(containers []*ateworkerpb.Container) bool { for _, c := range containers { if len(c.GetCsiVolumeMounts()) > 0 { return true @@ -38,7 +38,7 @@ func hasCsiVolumes(containers []*ateompb.Container) bool { // stageCsiVolumes bind-mounts src, the actor's host CSI volumes directory, // into the sandbox's shared virtio-fs tree at SharedDir(actorUID)/csi. -func (s *AteomService) stageCsiVolumes(ctx context.Context, actorUID, src string) error { +func (s *AteWorkerService) stageCsiVolumes(ctx context.Context, actorUID, src string) error { if _, err := os.Stat(src); err != nil { return fmt.Errorf("while checking CSI volumes dir %q: %w", src, err) } diff --git a/cmd/ateom-microvm/csi_test.go b/cmd/ateworker-microvm/csi_test.go similarity index 79% rename from cmd/ateom-microvm/csi_test.go rename to cmd/ateworker-microvm/csi_test.go index 0d463242e7..aaa2c9bb02 100644 --- a/cmd/ateom-microvm/csi_test.go +++ b/cmd/ateworker-microvm/csi_test.go @@ -19,13 +19,13 @@ package main import ( "testing" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) func TestHasCsiVolumes(t *testing.T) { tests := []struct { name string - containers []*ateompb.Container + containers []*ateworkerpb.Container want bool }{ { @@ -35,10 +35,10 @@ func TestHasCsiVolumes(t *testing.T) { }, { name: "no CSI volumes", - containers: []*ateompb.Container{ + containers: []*ateworkerpb.Container{ { Name: "c1", - DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "data", MountPath: "/data"}, }, }, @@ -47,10 +47,10 @@ func TestHasCsiVolumes(t *testing.T) { }, { name: "has CSI volumes", - containers: []*ateompb.Container{ + containers: []*ateworkerpb.Container{ { Name: "c1", - CsiVolumeMounts: []*ateompb.VolumeMount{ + CsiVolumeMounts: []*ateworkerpb.VolumeMount{ {VolumeName: "csi-vol", MountPath: "/csi"}, }, }, @@ -59,13 +59,13 @@ func TestHasCsiVolumes(t *testing.T) { }, { name: "multiple containers, one has CSI", - containers: []*ateompb.Container{ + containers: []*ateworkerpb.Container{ { Name: "c1", }, { Name: "c2", - CsiVolumeMounts: []*ateompb.VolumeMount{ + CsiVolumeMounts: []*ateworkerpb.VolumeMount{ {VolumeName: "csi-vol", MountPath: "/csi"}, }, }, diff --git a/cmd/ateom-microvm/durable.go b/cmd/ateworker-microvm/durable.go similarity index 92% rename from cmd/ateom-microvm/durable.go rename to cmd/ateworker-microvm/durable.go index c27a3f4e1d..830ac75a77 100644 --- a/cmd/ateom-microvm/durable.go +++ b/cmd/ateworker-microvm/durable.go @@ -25,7 +25,7 @@ package main // ActorDirs.durable_dir_volume_mounts_dir and wipes them when the actor's // directories are reset. // -// ateom exposes that host directory to the guest under the single kataShared +// ateworker exposes that host directory to the guest under the single kataShared // virtio-fs share at SharedDir(actorUID)/durable, where each container's bind // is attached. // @@ -45,9 +45,9 @@ import ( "slices" "strings" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/tarutil" ) @@ -66,7 +66,7 @@ func durableTarFile(volumeName string) (string, error) { } // hasDurableVolumes reports whether any container mounts a durable-dir volume. -func hasDurableVolumes(containers []*ateompb.Container) bool { +func hasDurableVolumes(containers []*ateworkerpb.Container) bool { for _, c := range containers { if len(c.GetDurableDirVolumeMounts()) > 0 { return true @@ -77,7 +77,7 @@ func hasDurableVolumes(containers []*ateompb.Container) bool { // durableVolumeNames returns the sorted, deduplicated durable-dir volume names // mounted by workload containers. -func durableVolumeNames(containers []*ateompb.Container) []string { +func durableVolumeNames(containers []*ateworkerpb.Container) []string { var names []string for _, c := range containers { for _, m := range c.GetDurableDirVolumeMounts() { @@ -90,7 +90,7 @@ func durableVolumeNames(containers []*ateompb.Container) []string { // stageDurableVolumes bind-mounts src, the actor's host durable-dir directory, // into the sandbox's shared virtio-fs tree at SharedDir(actorUID)/durable. -func (s *AteomService) stageDurableVolumes(ctx context.Context, actorUID, src string) error { +func (s *AteWorkerService) stageDurableVolumes(ctx context.Context, actorUID, src string) error { if _, err := os.Stat(src); err != nil { return fmt.Errorf("while checking durable-dir volumes dir %q: %w", src, err) } diff --git a/cmd/ateom-microvm/durable_test.go b/cmd/ateworker-microvm/durable_test.go similarity index 92% rename from cmd/ateom-microvm/durable_test.go rename to cmd/ateworker-microvm/durable_test.go index 26a2749695..64d01ca1c5 100644 --- a/cmd/ateom-microvm/durable_test.go +++ b/cmd/ateworker-microvm/durable_test.go @@ -22,26 +22,26 @@ import ( "slices" "testing" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/tarutil" ) func TestHasDurableVolumes(t *testing.T) { tests := []struct { name string - containers []*ateompb.Container + containers []*ateworkerpb.Container want bool }{ {name: "no containers"}, { name: "container without durable volumes", - containers: []*ateompb.Container{{Name: "app"}}, + containers: []*ateworkerpb.Container{{Name: "app"}}, }, { name: "one of several containers has a durable volume", - containers: []*ateompb.Container{ + containers: []*ateworkerpb.Container{ {Name: "sidecar"}, - {Name: "app", DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + {Name: "app", DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "data", MountPath: "/home/counter"}, }}, }, @@ -58,12 +58,12 @@ func TestHasDurableVolumes(t *testing.T) { } func TestDurableVolumeNames(t *testing.T) { - containers := []*ateompb.Container{ - {Name: "app", DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + containers := []*ateworkerpb.Container{ + {Name: "app", DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "data", MountPath: "/data"}, {VolumeName: "cache", MountPath: "/cache"}, }}, - {Name: "sidecar", DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + {Name: "sidecar", DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "data", MountPath: "/shared"}, }}, } diff --git a/cmd/ateom-microvm/fidelity.go b/cmd/ateworker-microvm/fidelity.go similarity index 76% rename from cmd/ateom-microvm/fidelity.go rename to cmd/ateworker-microvm/fidelity.go index 981fd94d18..de9a7049f7 100644 --- a/cmd/ateom-microvm/fidelity.go +++ b/cmd/ateworker-microvm/fidelity.go @@ -16,7 +16,7 @@ package main -import "github.com/agent-substrate/substrate/internal/proto/ateompb" +import "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" // The fidelity ladder, lowest to highest, each level including the ones // below it: VOLUMES holds the durable volumes, ROOTFS adds each container's @@ -26,13 +26,13 @@ import "github.com/agent-substrate/substrate/internal/proto/ateompb" // includesRootfs reports whether snapshots at this fidelity carry the // containers' rootfs uppers, so a checkpoint tars them and a restore puts // them back. -func includesRootfs(f ateompb.SnapshotFidelity) bool { - return f == ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS || f == ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY +func includesRootfs(f ateworkerpb.SnapshotFidelity) bool { + return f == ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS || f == ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY } // includesMemory reports whether snapshots at this fidelity carry the guest // itself, so a restore resumes it (and its CPU counters) rather than // cold-booting a fresh one. -func includesMemory(f ateompb.SnapshotFidelity) bool { - return f == ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY +func includesMemory(f ateworkerpb.SnapshotFidelity) bool { + return f == ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY } diff --git a/cmd/ateom-microvm/fidelity_test.go b/cmd/ateworker-microvm/fidelity_test.go similarity index 73% rename from cmd/ateom-microvm/fidelity_test.go rename to cmd/ateworker-microvm/fidelity_test.go index cb65fa75e7..4634041d14 100644 --- a/cmd/ateom-microvm/fidelity_test.go +++ b/cmd/ateworker-microvm/fidelity_test.go @@ -23,7 +23,7 @@ import ( "google.golang.org/grpc/codes" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // TestFidelityLadder pins which layers each level carries: the checkpoint's @@ -31,13 +31,13 @@ import ( // these two predicates. func TestFidelityLadder(t *testing.T) { for _, tc := range []struct { - fidelity ateompb.SnapshotFidelity + fidelity ateworkerpb.SnapshotFidelity wantRootfs, wantMemory bool }{ - {ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, false, false}, - {ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, false, false}, - {ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, true, false}, - {ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, true, true}, + {ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, false, false}, + {ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, false, false}, + {ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, true, false}, + {ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, true, true}, } { t.Run(tc.fidelity.String(), func(t *testing.T) { if got := includesRootfs(tc.fidelity); got != tc.wantRootfs { @@ -57,14 +57,14 @@ func TestFidelityLadder(t *testing.T) { func TestValidateFidelity(t *testing.T) { for _, tc := range []struct { name string - fidelity ateompb.SnapshotFidelity + fidelity ateworkerpb.SnapshotFidelity wantErr bool }{ - {"volumes", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, false}, - {"rootfs", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, false}, - {"memory", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, false}, - {"unspecified", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, true}, - {"outside the enum", ateompb.SnapshotFidelity(99), true}, + {"volumes", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, false}, + {"rootfs", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, false}, + {"memory", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, false}, + {"unspecified", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, true}, + {"outside the enum", ateworkerpb.SnapshotFidelity(99), true}, } { t.Run(tc.name, func(t *testing.T) { err := validateFidelity(tc.fidelity) diff --git a/cmd/ateom-microvm/hosted.go b/cmd/ateworker-microvm/hosted.go similarity index 75% rename from cmd/ateom-microvm/hosted.go rename to cmd/ateworker-microvm/hosted.go index 549286f4f2..0c33ae705b 100644 --- a/cmd/ateom-microvm/hosted.go +++ b/cmd/ateworker-microvm/hosted.go @@ -23,15 +23,15 @@ import ( "net" "time" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/ateomcgroup" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" - "github.com/agent-substrate/substrate/internal/ateomstats" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" + "github.com/agent-substrate/substrate/internal/ateworker/cgroup" "github.com/agent-substrate/substrate/internal/atunnel" + "github.com/agent-substrate/substrate/internal/netns" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/sizing" + "github.com/agent-substrate/substrate/internal/workerstats" ) // hostedActor holds one actor's attribution, network, and runtime state. @@ -40,7 +40,7 @@ type hostedActor struct { attribution resources.ActorAttribution // Mutable fields are guarded by actorsMu. - network *ateomnet.SandboxSession + network *actornet.SandboxSession // Nil until boot publishes the VM. vm *runningActor // Nil when no guest stats connection is available. @@ -48,30 +48,30 @@ type hostedActor struct { // usage is this activation's epoch, CPU baseline, and latest sample. // Immutable after admission. - usage *ateomstats.Activation + usage *workerstats.Activation } // admitActor reserves capacity before network setup. An actor that is already // hosted keeps its slot; its old network is returned for the caller to close. // resumesGuest is true when the activation resumes a guest from a snapshot, // whose CPU counters come back with it. -func (s *AteomService) admitActor(attribution resources.ActorAttribution, resumesGuest bool) (*hostedActor, *ateomnet.SandboxSession, error) { +func (s *AteWorkerService) admitActor(attribution resources.ActorAttribution, resumesGuest bool) (*hostedActor, *actornet.SandboxSession, error) { s.actorsMu.Lock() defer s.actorsMu.Unlock() - var stale *ateomnet.SandboxSession + var stale *actornet.SandboxSession if old, ok := s.actors[attribution.UID]; ok { stale = old.network } else if len(s.actors)+s.draining >= s.maxActors { return nil, nil, apierror.ResourceExhausted("worker is full: %d actors", s.maxActors) } - hosted := &hostedActor{attribution: attribution, usage: ateomstats.NewActivation(time.Now(), resumesGuest)} + hosted := &hostedActor{attribution: attribution, usage: workerstats.NewActivation(time.Now(), resumesGuest)} s.actors[attribution.UID] = hosted return hosted, stale, nil } // hostActor sets up the actor's network, replacing any stale one. resumesGuest // is as for admitActor. -func (s *AteomService) hostActor(ctx context.Context, attribution resources.ActorAttribution, resumesGuest bool) (*hostedActor, error) { +func (s *AteWorkerService) hostActor(ctx context.Context, attribution resources.ActorAttribution, resumesGuest bool) (*hostedActor, error) { uid := attribution.UID if uid == "" { return nil, fmt.Errorf("actor UID is required") @@ -91,7 +91,7 @@ func (s *AteomService) hostActor(ctx context.Context, attribution resources.Acto } // The tap and atunnel share a namespace; the guest owns the other end. - session, err := ateomnet.ServeSandbox(ctx, ateomnet.SandboxNetworkConfig{ + session, err := actornet.ServeSandbox(ctx, actornet.SandboxNetworkConfig{ ActorUID: uid, EgressPort: s.tunnel.EgressPort, DNSPolicy: s.tunnel.ActorEgressPolicy(attribution), @@ -110,19 +110,19 @@ func (s *AteomService) hostActor(ctx context.Context, attribution resources.Acto } // actorLeaf opens the actor's cgroup, or returns nil when the worker has none. -func (s *AteomService) actorLeaf(actorUID string, size sizing.SandboxSize) (*ateomcgroup.ActorLeaf, error) { +func (s *AteWorkerService) actorLeaf(actorUID string, size sizing.SandboxSize) (*cgroup.ActorLeaf, error) { if !s.actorCgroups { return nil, nil } - return ateomcgroup.OpenActorLeaf(actorUID, size.MilliCPU) + return cgroup.OpenActorLeaf(actorUID, size.MilliCPU) } // cleanupSandboxState kills what is left of the actor's host processes and // clears its sandbox directories. -func (s *AteomService) cleanupSandboxState(ctx context.Context, actorUID string) { +func (s *AteWorkerService) cleanupSandboxState(ctx context.Context, actorUID string) { if s.actorCgroups { killCtx, cancel := context.WithTimeout(ctx, 5*time.Second) - if err := ateomcgroup.KillActorLeaf(killCtx, actorUID); err != nil { + if err := cgroup.KillActorLeaf(killCtx, actorUID); err != nil { slog.WarnContext(ctx, "Failed to kill the actor's leftover processes", slog.Any("err", err)) } cancel() @@ -132,7 +132,7 @@ func (s *AteomService) cleanupSandboxState(ctx context.Context, actorUID string) // unhostActor removes an actor, its network, and its cgroup. Repeated calls are // safe. The caller must stop the VM first. -func (s *AteomService) unhostActor(ctx context.Context, actorUID string) error { +func (s *AteWorkerService) unhostActor(ctx context.Context, actorUID string) error { s.actorsMu.Lock() hosted, ok := s.actors[actorUID] delete(s.actors, actorUID) @@ -151,7 +151,7 @@ func (s *AteomService) unhostActor(ctx context.Context, actorUID string) error { }() if s.actorCgroups { - if err := ateomcgroup.RemoveActorLeaf(actorUID); err != nil { + if err := cgroup.RemoveActorLeaf(actorUID); err != nil { slog.WarnContext(ctx, "Failed to remove the actor's cgroup", slog.Any("err", err)) } } @@ -166,14 +166,14 @@ func (s *AteomService) unhostActor(ctx context.Context, actorUID string) error { } // lookupActor returns the actor or nil without taking its lifecycle lock. -func (s *AteomService) lookupActor(actorUID string) *hostedActor { +func (s *AteWorkerService) lookupActor(actorUID string) *hostedActor { s.actorsMu.RLock() defer s.actorsMu.RUnlock() return s.actors[actorUID] } // hostedActors returns a snapshot of the actor map. -func (s *AteomService) hostedActors() []*hostedActor { +func (s *AteWorkerService) hostedActors() []*hostedActor { s.actorsMu.RLock() defer s.actorsMu.RUnlock() out := make([]*hostedActor, 0, len(s.actors)) @@ -184,7 +184,7 @@ func (s *AteomService) hostedActors() []*hostedActor { } // setRunningVM publishes the live micro-VM for an actor. -func (s *AteomService) setRunningVM(actorUID string, vm *runningActor) { +func (s *AteWorkerService) setRunningVM(actorUID string, vm *runningActor) { s.actorsMu.Lock() defer s.actorsMu.Unlock() if hosted, ok := s.actors[actorUID]; ok { @@ -193,7 +193,7 @@ func (s *AteomService) setRunningVM(actorUID string, vm *runningActor) { } // runningVM is the live micro-VM for an actor, or nil. -func (s *AteomService) runningVM(actorUID string) *runningActor { +func (s *AteWorkerService) runningVM(actorUID string) *runningActor { hosted := s.lookupActor(actorUID) if hosted == nil { return nil @@ -204,7 +204,7 @@ func (s *AteomService) runningVM(actorUID string) *runningActor { } // setGuestStats sets or clears the actor's stats target. -func (s *AteomService) setGuestStats(actorUID string, guest *guestStatsTarget) { +func (s *AteWorkerService) setGuestStats(actorUID string, guest *guestStatsTarget) { s.actorsMu.Lock() defer s.actorsMu.Unlock() if hosted, ok := s.actors[actorUID]; ok { @@ -213,14 +213,14 @@ func (s *AteomService) setGuestStats(actorUID string, guest *guestStatsTarget) { } // guestOf is h's stats target, or nil. -func (s *AteomService) guestOf(h *hostedActor) *guestStatsTarget { +func (s *AteWorkerService) guestOf(h *hostedActor) *guestStatsTarget { s.actorsMu.RLock() defer s.actorsMu.RUnlock() return h.guest } // sandboxNetNS is where an actor's tap and atunnel's sockets live, or -1. -func (s *AteomService) sandboxNetNS(actorUID string) netns.Handle { +func (s *AteWorkerService) sandboxNetNS(actorUID string) netns.Handle { hosted := s.lookupActor(actorUID) if hosted == nil { return -1 @@ -234,11 +234,11 @@ func (s *AteomService) sandboxNetNS(actorUID string) netns.Handle { } // sandboxDialer connects through the actor's gateway namespace. -func (s *AteomService) sandboxDialer(actorUID string) atunnel.DialFunc { +func (s *AteWorkerService) sandboxDialer(actorUID string) atunnel.DialFunc { return func(ctx context.Context, network, address string) (net.Conn, error) { s.actorsMu.RLock() hosted := s.actors[actorUID] - var session *ateomnet.SandboxSession + var session *actornet.SandboxSession if hosted != nil { session = hosted.network } diff --git a/cmd/ateom-microvm/hosted_test.go b/cmd/ateworker-microvm/hosted_test.go similarity index 96% rename from cmd/ateom-microvm/hosted_test.go rename to cmd/ateworker-microvm/hosted_test.go index 7acf915d6d..9c60d5cff1 100644 --- a/cmd/ateom-microvm/hosted_test.go +++ b/cmd/ateworker-microvm/hosted_test.go @@ -28,7 +28,7 @@ import ( // Re-admitting an actor that is already hosted keeps its slot, so it succeeds // on a full worker and a retry cannot lose its place to another actor. func TestAdmitActorKeepsAHostedActorsSlot(t *testing.T) { - s := &AteomService{actors: map[string]*hostedActor{}, maxActors: 1} + s := &AteWorkerService{actors: map[string]*hostedActor{}, maxActors: 1} first, _, err := s.admitActor(resources.ActorAttribution{UID: "actor-a"}, false) if err != nil { t.Fatal(err) diff --git a/cmd/ateom-microvm/internal/agentstats/agentstats.go b/cmd/ateworker-microvm/internal/agentstats/agentstats.go similarity index 93% rename from cmd/ateom-microvm/internal/agentstats/agentstats.go rename to cmd/ateworker-microvm/internal/agentstats/agentstats.go index 581658d87e..ad9b83ed67 100644 --- a/cmd/ateom-microvm/internal/agentstats/agentstats.go +++ b/cmd/ateworker-microvm/internal/agentstats/agentstats.go @@ -13,9 +13,9 @@ // limitations under the License. // Package agentstats turns the kata guest agent's per-container cgroup -// accounting into the resource-usage sample ateom reports. +// accounting into the resource-usage sample ateworker reports. // -// The micro-VM ateom uses it to answer ateompb.Ateom/GetWorkloadStats. The host +// The micro-VM ateworker uses it to answer ateworkerpb.Worker/GetWorkloadStats. The host // cgroup is the wrong place to look on this runtime: the guest's RAM is a fixed // allocation cloud-hypervisor takes at boot, so the host cgroup reads roughly // the same whether the actor is idle or saturated. The numbers that move with @@ -24,16 +24,16 @@ // // This package is deliberately pure — it converts an already-fetched // agentpb.CgroupStats and never talks to a guest — which keeps it testable -// without a live micro-VM and, unlike the rest of the micro-VM ateom, without +// without a live micro-VM and, unlike the rest of the micro-VM ateworker, without // the linux build tag. package agentstats import ( - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" ) // Sample is a point-in-time reading for one container, or the sum of several. -// It carries the same four numbers as the gVisor ateom's cgroupstats.Sample, +// It carries the same four numbers as the gVisor ateworker's cgroupstats.Sample, // because both feed the same four fields of GetWorkloadStatsResponse. // // Anything the guest did not report reads as zero rather than failing the whole @@ -80,7 +80,7 @@ const ( // may be nil — the agent answers without cgroup stats for a container it has no // accounting for, which is a normal state for one that has exited rather than // an error. The caller decides what an all-zero container means; see the -// summing in the micro-VM ateom's GetWorkloadStats. +// summing in the micro-VM ateworker's GetWorkloadStats. func FromCgroupStats(cs *agentpb.CgroupStats) Sample { mem := cs.GetMemoryStats().GetUsage() current := mem.GetUsage() @@ -136,7 +136,7 @@ func inactiveFileBytes(cs *agentpb.CgroupStats) (uint64, bool) { // it is exact. // // The guest kernel does track the true figure: every container cgroup sits -// under the shared /ateomchv parent (see StartRootfsContainer and the +// under the shared /actor parent (see StartRootfsContainer and the // CgroupsPath defaults in internal/kata), so with hierarchical accounting the // parent's memory.peak is the actor-level maximum this sum approximates. The // kata-agent just has no RPC that reads a cgroup by path — StatsContainer is diff --git a/cmd/ateom-microvm/internal/agentstats/agentstats_test.go b/cmd/ateworker-microvm/internal/agentstats/agentstats_test.go similarity index 98% rename from cmd/ateom-microvm/internal/agentstats/agentstats_test.go rename to cmd/ateworker-microvm/internal/agentstats/agentstats_test.go index 6384763d8e..3bcc79bc48 100644 --- a/cmd/ateom-microvm/internal/agentstats/agentstats_test.go +++ b/cmd/ateworker-microvm/internal/agentstats/agentstats_test.go @@ -19,7 +19,7 @@ import ( "github.com/google/go-cmp/cmp" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" ) // cgroupStats builds an agent reading: usage/max bytes, the memory.stat entries diff --git a/cmd/ateom-microvm/internal/ch/api.go b/cmd/ateworker-microvm/internal/ch/api.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/api.go rename to cmd/ateworker-microvm/internal/ch/api.go diff --git a/cmd/ateom-microvm/internal/ch/ch.go b/cmd/ateworker-microvm/internal/ch/ch.go similarity index 98% rename from cmd/ateom-microvm/internal/ch/ch.go rename to cmd/ateworker-microvm/internal/ch/ch.go index 0921635d7c..38be196b03 100644 --- a/cmd/ateom-microvm/internal/ch/ch.go +++ b/cmd/ateworker-microvm/internal/ch/ch.go @@ -17,8 +17,8 @@ // kata creates at /run/vc/vm//clh-api.sock), plus relaunching a fresh VMM // from a snapshot directory for restore. // -// This is the snapshot/restore half of the ateom-microvm model: kata -// owns RUN (boot the micro-VM + run the OCI container), and ateom drives the CH +// This is the snapshot/restore half of the ateworker-microvm model: kata +// owns RUN (boot the micro-VM + run the OCI container), and ateworker drives the CH // REST API underneath for suspend (pause+snapshot) and owns the bare-CH // relaunch for restore (see LaunchVMM + RestoreWithNetFDs in restorefds.go). The // REST wire format is the one cloud-hypervisor documents for snapshot/restore. diff --git a/cmd/ateom-microvm/internal/ch/ch_test.go b/cmd/ateworker-microvm/internal/ch/ch_test.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/ch_test.go rename to cmd/ateworker-microvm/internal/ch/ch_test.go diff --git a/cmd/ateom-microvm/internal/ch/createvm.go b/cmd/ateworker-microvm/internal/ch/createvm.go similarity index 96% rename from cmd/ateom-microvm/internal/ch/createvm.go rename to cmd/ateworker-microvm/internal/ch/createvm.go index 466e45afd1..9019294cf2 100644 --- a/cmd/ateom-microvm/internal/ch/createvm.go +++ b/cmd/ateworker-microvm/internal/ch/createvm.go @@ -20,7 +20,7 @@ import ( ) // VmConfig is the body of /api/v1/vm.create — the subset of cloud-hypervisor's -// VmConfig ateom sets to boot the kata guest. Modeled on kata's clh driver +// VmConfig ateworker sets to boot the kata guest. Modeled on kata's clh driver // (src/runtime/virtcontainers/clh.go). vm.create + vm.boot are issued with PUT. type VmConfig struct { Cpus CpusConfig `json:"cpus"` @@ -59,7 +59,7 @@ type CpusConfig struct { // MemoryConfig sets guest RAM. Shared=true makes CH back RAM with a memfd, which // is what lets vm.snapshot write a SPARSE image (the memory-only snapshot the -// rest of ateom relies on). +// rest of ateworker relies on). type MemoryConfig struct { Size int64 `json:"size"` Shared bool `json:"shared"` @@ -95,7 +95,7 @@ type ConsoleConfig struct { // VsockConfig is the hybrid-vsock the kata-agent listens on. Cid is the guest // CID (kata uses 3); Socket is the host unix socket (kata.VsockSocketPath) that -// ateom then dials (DialAgent) to drive the agent. +// ateworker then dials (DialAgent) to drive the agent. type VsockConfig struct { Cid int64 `json:"cid"` Socket string `json:"socket"` diff --git a/cmd/ateom-microvm/internal/ch/guestclock.go b/cmd/ateworker-microvm/internal/ch/guestclock.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/guestclock.go rename to cmd/ateworker-microvm/internal/ch/guestclock.go diff --git a/cmd/ateom-microvm/internal/ch/guestclock_test.go b/cmd/ateworker-microvm/internal/ch/guestclock_test.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/guestclock_test.go rename to cmd/ateworker-microvm/internal/ch/guestclock_test.go diff --git a/cmd/ateom-microvm/internal/ch/merge.go b/cmd/ateworker-microvm/internal/ch/merge.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/merge.go rename to cmd/ateworker-microvm/internal/ch/merge.go diff --git a/cmd/ateom-microvm/internal/ch/merge_test.go b/cmd/ateworker-microvm/internal/ch/merge_test.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/merge_test.go rename to cmd/ateworker-microvm/internal/ch/merge_test.go diff --git a/cmd/ateom-microvm/internal/ch/prefault.go b/cmd/ateworker-microvm/internal/ch/prefault.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/prefault.go rename to cmd/ateworker-microvm/internal/ch/prefault.go diff --git a/cmd/ateom-microvm/internal/ch/prefault_test.go b/cmd/ateworker-microvm/internal/ch/prefault_test.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/prefault_test.go rename to cmd/ateworker-microvm/internal/ch/prefault_test.go diff --git a/cmd/ateom-microvm/internal/ch/restorefds.go b/cmd/ateworker-microvm/internal/ch/restorefds.go similarity index 100% rename from cmd/ateom-microvm/internal/ch/restorefds.go rename to cmd/ateworker-microvm/internal/ch/restorefds.go diff --git a/cmd/ateom-microvm/internal/kata/agentclient.go b/cmd/ateworker-microvm/internal/kata/agentclient.go similarity index 95% rename from cmd/ateom-microvm/internal/kata/agentclient.go rename to cmd/ateworker-microvm/internal/kata/agentclient.go index ab04ebc77b..ff39a0c991 100644 --- a/cmd/ateom-microvm/internal/kata/agentclient.go +++ b/cmd/ateworker-microvm/internal/kata/agentclient.go @@ -25,7 +25,7 @@ import ( "strings" "time" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" "github.com/containerd/ttrpc" "google.golang.org/protobuf/types/known/emptypb" ) @@ -84,8 +84,8 @@ func DebugConsoleDump(ctx context.Context, vsockPath, cmd string) string { return out.String() } -// AgentClient is a thin ttrpc client for the kata-agent RPCs ateom drives -// directly. ateom owns the cloud-hypervisor boot (no kata shim) and drives the +// AgentClient is a thin ttrpc client for the kata-agent RPCs ateworker drives +// directly. ateworker owns the cloud-hypervisor boot (no kata shim) and drives the // kata-agent over ttrpc itself: alongside UpdateInterface/UpdateRoutes for guest // networking, it issues CreateContainer/StartContainer to assemble the container // rootfs directly, instead of relying on the kata runtime's hooks @@ -170,7 +170,7 @@ func (a *AgentClient) StartContainer(ctx context.Context, containerID string) er // CreateSandbox establishes the agent's sandbox context (sandbox id, hostname, // sandbox pidns) before any container is created. The kata shim normally issues -// this once at VM boot; ateom (no shim) must call it itself so the agent has a +// this once at VM boot; ateworker (no shim) must call it itself so the agent has a // sandbox to attach containers to. Storages carries the shared virtio-fs mount // (the overlay lowers); each container's rootfs is assembled per-container. // Mirrors grpc.AgentService/CreateSandbox (returns google.protobuf.Empty). @@ -182,7 +182,7 @@ func (a *AgentClient) CreateSandbox(ctx context.Context, req *agentpb.CreateSand } // UpdateInterface configures a guest network interface (the kata shim's job, which -// ateom does itself). The agent matches the link by HwAddr, then applies the +// ateworker does itself). The agent matches the link by HwAddr, then applies the // name/IP/MTU. Mirrors grpc.AgentService/UpdateInterface (returns the resulting // Interface). func (a *AgentClient) UpdateInterface(ctx context.Context, iface *agentpb.Interface) error { @@ -230,7 +230,7 @@ func (a *AgentClient) SignalProcess(ctx context.Context, containerID, execID str // WaitProcess blocks until the identified guest process exits and returns its // exit status (mimics waitpid(2)). Used during graceful shutdown to confirm the -// actor has stopped before ateom tears the VM down. Mirrors +// actor has stopped before ateworker tears the VM down. Mirrors // grpc.AgentService/WaitProcess. func (a *AgentClient) WaitProcess(ctx context.Context, containerID, execID string) (int32, error) { resp := &agentpb.WaitProcessResponse{} @@ -246,7 +246,7 @@ func (a *AgentClient) WaitProcess(ctx context.Context, containerID, execID strin // buffered (up to max), so callers loop until it returns an error — the agent // returns an error/EOF-like status once the stream ends (container exit / connection // close). Mirrors grpc.AgentService/ReadStdout. The kata-agent keys the stream by -// ExecId, which ateom sets equal to ContainerId. +// ExecId, which ateworker sets equal to ContainerId. func (a *AgentClient) ReadStdout(ctx context.Context, containerID, execID string, max uint32) ([]byte, error) { resp := &agentpb.ReadStreamResponse{} req := &agentpb.ReadStreamRequest{ContainerId: containerID, ExecId: execID, Len: max} @@ -274,7 +274,7 @@ func (a *AgentClient) ReadStderr(ctx context.Context, containerID, execID string // // It returns only the cgroup half of the response; the network counters // alongside it are per-guest-interface rather than per-container and are not -// what ateom reports. A nil return with a nil error means the agent answered +// what ateworker reports. A nil return with a nil error means the agent answered // without cgroup stats for this container — it has no accounting for it, which // is a normal state for one that has exited. What that means for the actor is // the caller's call: the summing in GetWorkloadStats folds it in as a zero @@ -308,7 +308,7 @@ type StreamReader struct { } // NewStdioReader returns an io.Reader over the container's stdout (stderr=false) -// or stderr (stderr=true). execID equals containerID (ateom sets ExecId == +// or stderr (stderr=true). execID equals containerID (ateworker sets ExecId == // ContainerId when it creates the container). func NewStdioReader(ctx context.Context, ac *AgentClient, containerID, execID string, stderr bool) *StreamReader { return &StreamReader{ctx: ctx, ac: ac, containerID: containerID, execID: execID, stderr: stderr} diff --git a/cmd/ateom-microvm/internal/kata/agentclient_linux_test.go b/cmd/ateworker-microvm/internal/kata/agentclient_linux_test.go similarity index 96% rename from cmd/ateom-microvm/internal/kata/agentclient_linux_test.go rename to cmd/ateworker-microvm/internal/kata/agentclient_linux_test.go index c81977b2bf..a27b40b397 100644 --- a/cmd/ateom-microvm/internal/kata/agentclient_linux_test.go +++ b/cmd/ateworker-microvm/internal/kata/agentclient_linux_test.go @@ -25,7 +25,7 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" "github.com/containerd/ttrpc" "google.golang.org/protobuf/types/known/emptypb" ) diff --git a/cmd/ateom-microvm/internal/kata/cleanup_linux.go b/cmd/ateworker-microvm/internal/kata/cleanup_linux.go similarity index 98% rename from cmd/ateom-microvm/internal/kata/cleanup_linux.go rename to cmd/ateworker-microvm/internal/kata/cleanup_linux.go index 12f383b882..6f7fd6ae5c 100644 --- a/cmd/ateom-microvm/internal/kata/cleanup_linux.go +++ b/cmd/ateworker-microvm/internal/kata/cleanup_linux.go @@ -33,7 +33,7 @@ import ( // CleanupSandboxState removes leftover host-side state for a sandbox id (the // virtio-fs shared sandbox dir and the per-VM runtime dir), lazily unmounting // anything still mounted underneath them first, and kills orphaned per-sandbox -// processes. ateom owns the cloud-hypervisor boot directly (no kata shim, no +// processes. ateworker owns the cloud-hypervisor boot directly (no kata shim, no // containerd), so a failed Create does not fully self-clean; the deterministic // sandbox id (= actor name) then collides on the next attempt: "listen unix // .../virtiofsd.sock: bind: address already in use", "Could not bind mount diff --git a/cmd/ateom-microvm/internal/kata/cleanup_linux_test.go b/cmd/ateworker-microvm/internal/kata/cleanup_linux_test.go similarity index 100% rename from cmd/ateom-microvm/internal/kata/cleanup_linux_test.go rename to cmd/ateworker-microvm/internal/kata/cleanup_linux_test.go diff --git a/cmd/ateom-microvm/internal/kata/config.go b/cmd/ateworker-microvm/internal/kata/config.go similarity index 84% rename from cmd/ateom-microvm/internal/kata/config.go rename to cmd/ateworker-microvm/internal/kata/config.go index ba95222e43..f1350ed343 100644 --- a/cmd/ateom-microvm/internal/kata/config.go +++ b/cmd/ateworker-microvm/internal/kata/config.go @@ -18,15 +18,7 @@ import ( "strings" ) -const ( - // TODO(#1724): Tune the following values for Substrate actors. - // DefaultMemoryMiB is the default guest memory size (MiB). - DefaultMemoryMiB = 2048 - // DefaultVCPUs is the default guest vCPU count. - DefaultVCPUs = 1 -) - -// BaseKernelParams is the guest kernel command line parameters ateom boots with +// BaseKernelParams is the guest kernel command line parameters ateworker boots with const BaseKernelParams = "cgroup_no_v1=all systemd.unified_cgroup_hierarchy=1" // WithAgentDebug enables guest agent debug logging and the debug console on vsock 1026 diff --git a/cmd/ateom-microvm/internal/kata/config_test.go b/cmd/ateworker-microvm/internal/kata/config_test.go similarity index 100% rename from cmd/ateom-microvm/internal/kata/config_test.go rename to cmd/ateworker-microvm/internal/kata/config_test.go diff --git a/cmd/ateom-microvm/internal/kata/kata.go b/cmd/ateworker-microvm/internal/kata/kata.go similarity index 84% rename from cmd/ateom-microvm/internal/kata/kata.go rename to cmd/ateworker-microvm/internal/kata/kata.go index ad49047e66..e0ea270248 100644 --- a/cmd/ateom-microvm/internal/kata/kata.go +++ b/cmd/ateworker-microvm/internal/kata/kata.go @@ -12,8 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package kata holds the helpers ateom uses to boot and drive a kata guest in a -// cloud-hypervisor micro-VM without the kata shim: ateom boots cloud-hypervisor +// Package kata holds the helpers ateworker uses to boot and drive a kata guest in a +// cloud-hypervisor micro-VM without the kata shim: ateworker boots cloud-hypervisor // itself (see internal/ch), then drives the stock kata-agent over its hybrid-vsock // ttrpc API (DialAgent / AgentClient) to create the sandbox and run each container // on its host-merged rootfs (overlay_linux.go). @@ -31,7 +31,7 @@ import ( const vcVMDir = "/run/vc/vm" // CLHSocketPath returns the default cloud-hypervisor API socket path for the -// sandbox with the given id (the per-sandbox runtime dir). ateom records the +// sandbox with the given id (the per-sandbox runtime dir). ateworker records the // actual api-socket it launched the VMM on, but uses this as the fallback. func CLHSocketPath(id string) string { return filepath.Join(vcVMDir, id, "clh-api.sock") diff --git a/cmd/ateom-microvm/internal/kata/overlay_linux.go b/cmd/ateworker-microvm/internal/kata/overlay_linux.go similarity index 97% rename from cmd/ateom-microvm/internal/kata/overlay_linux.go rename to cmd/ateworker-microvm/internal/kata/overlay_linux.go index 8415670c08..9e06d03e4b 100644 --- a/cmd/ateom-microvm/internal/kata/overlay_linux.go +++ b/cmd/ateworker-microvm/internal/kata/overlay_linux.go @@ -18,7 +18,7 @@ package kata // Each container's rootfs is an overlay assembled ON THE HOST — the stock kata // arrangement (containerd's overlay snapshotter does the same): lower = the OCI image -// bundle, upper/work = host directories (see cmd/ateom-microvm/rootfsupper.go), merged +// bundle, upper/work = host directories (see cmd/ateworker-microvm/rootfsupper.go), merged // by the host kernel and served to the guest over the ONE kataShared virtio-fs share. // The guest runs the container on that directory directly; it never mounts an overlay // itself. Rootfs writes therefore cost host disk, not guest RAM, and persist across @@ -38,7 +38,7 @@ import ( "syscall" "time" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" "github.com/agent-substrate/substrate/internal/ocispec" specs "github.com/opencontainers/runtime-spec/specs-go" "golang.org/x/sys/unix" @@ -71,7 +71,7 @@ func VirtiofsdSocketPath(id string) string { return filepath.Join(VMDir(id), "vi // /work. Both properties are load-bearing — the kernel requires upperdir // and workdir on the same filesystem and rejects a nested workdir. The // snapshot only carries each upperdir's contents (see -// cmd/ateom-microvm/rootfsupper.go); ateom creates this layout itself. +// cmd/ateworker-microvm/rootfsupper.go); ateworker creates this layout itself. // Covered by regression tests. func UpperWorkDirs(upperBase, containerID string) (upper, work string) { return filepath.Join(upperBase, containerID, "fs"), filepath.Join(upperBase, containerID, "work") @@ -248,7 +248,7 @@ func StageImageVolume(ctx context.Context, src, id, cid, volumeName string) erro // // The merged path is identical on every node — find-paths migration re-opens the // guest's open files by path — given a deterministic image unpack plus the upper -// re-materialized from the snapshot tar (see cmd/ateom-microvm/rootfsupper.go). +// re-materialized from the snapshot tar (see cmd/ateworker-microvm/rootfsupper.go). func StageMergedRootfs(ctx context.Context, bundleRootfs, upperBase, restoreID, cid string) error { if cid == "" { return fmt.Errorf("StageMergedRootfs: empty container id") @@ -447,10 +447,10 @@ func (a *AgentClient) CreateSandboxForActor(ctx context.Context, opts CreateSand func (a *AgentClient) StartRootfsContainer(ctx context.Context, cid string, spec *specs.Spec) error { pbSpec := SpecToAgentPB(spec) pbSpec.Root = &agentpb.Root{Path: GuestSharedRootfs(cid), Readonly: false} - // Per-container cgroup under the shared /ateomchv parent, so the guest + // Per-container cgroup under the shared /actor parent, so the guest // kernel accounts an actor's containers hierarchically (see agentstats). if pbSpec.Linux != nil { - pbSpec.Linux.CgroupsPath = "/ateomchv/" + cid + pbSpec.Linux.CgroupsPath = "/actor/" + cid } if err := a.CreateContainer(ctx, &agentpb.CreateContainerRequest{ ContainerId: cid, diff --git a/cmd/ateom-microvm/internal/kata/overlay_linux_test.go b/cmd/ateworker-microvm/internal/kata/overlay_linux_test.go similarity index 99% rename from cmd/ateom-microvm/internal/kata/overlay_linux_test.go rename to cmd/ateworker-microvm/internal/kata/overlay_linux_test.go index 8119eb1b09..be491697ce 100644 --- a/cmd/ateom-microvm/internal/kata/overlay_linux_test.go +++ b/cmd/ateworker-microvm/internal/kata/overlay_linux_test.go @@ -66,7 +66,7 @@ func TestEnsureOCIMountpoints(t *testing.T) { // rejects a workdir nested inside (or equal to) upperdir — so they must be // SIBLINGS under the container's subdirectory of the actor's upper base. func TestUpperWorkDirsAreSiblings(t *testing.T) { - const base = "/var/lib/ateom-gvisor/actors/uid/rootfs-upper" + const base = "/var/lib/ateworker-gvisor/actors/uid/rootfs-upper" upper, work := UpperWorkDirs(base, "app") cidDir := filepath.Join(base, "app") if filepath.Dir(upper) != cidDir || filepath.Dir(work) != cidDir { diff --git a/cmd/ateom-microvm/internal/kata/overlay_wait_test.go b/cmd/ateworker-microvm/internal/kata/overlay_wait_test.go similarity index 100% rename from cmd/ateom-microvm/internal/kata/overlay_wait_test.go rename to cmd/ateworker-microvm/internal/kata/overlay_wait_test.go diff --git a/cmd/ateom-microvm/internal/kata/restore.go b/cmd/ateworker-microvm/internal/kata/restore.go similarity index 100% rename from cmd/ateom-microvm/internal/kata/restore.go rename to cmd/ateworker-microvm/internal/kata/restore.go diff --git a/cmd/ateom-microvm/internal/kata/specconv.go b/cmd/ateworker-microvm/internal/kata/specconv.go similarity index 98% rename from cmd/ateom-microvm/internal/kata/specconv.go rename to cmd/ateworker-microvm/internal/kata/specconv.go index f176cf06a4..87f0f2194c 100644 --- a/cmd/ateom-microvm/internal/kata/specconv.go +++ b/cmd/ateworker-microvm/internal/kata/specconv.go @@ -17,7 +17,7 @@ package kata import ( - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" specs "github.com/opencontainers/runtime-spec/specs-go" ) diff --git a/cmd/ateom-microvm/internal/kata/specconv_linux_test.go b/cmd/ateworker-microvm/internal/kata/specconv_linux_test.go similarity index 100% rename from cmd/ateom-microvm/internal/kata/specconv_linux_test.go rename to cmd/ateworker-microvm/internal/kata/specconv_linux_test.go diff --git a/cmd/ateom-microvm/internal/third_party/kata/LICENSE b/cmd/ateworker-microvm/internal/third_party/kata/LICENSE similarity index 100% rename from cmd/ateom-microvm/internal/third_party/kata/LICENSE rename to cmd/ateworker-microvm/internal/third_party/kata/LICENSE diff --git a/cmd/ateom-microvm/internal/third_party/kata/PROVENANCE.md b/cmd/ateworker-microvm/internal/third_party/kata/PROVENANCE.md similarity index 89% rename from cmd/ateom-microvm/internal/third_party/kata/PROVENANCE.md rename to cmd/ateworker-microvm/internal/third_party/kata/PROVENANCE.md index 52676b2a8e..e9d8cdbd06 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/PROVENANCE.md +++ b/cmd/ateworker-microvm/internal/third_party/kata/PROVENANCE.md @@ -1,7 +1,7 @@ # third_party/kata — vendored kata-containers sources Source copied from [kata-containers](https://github.com/kata-containers/kata-containers) -and used by `cmd/ateom-microvm`. The upstream `LICENSE` in this directory covers +and used by `cmd/ateworker-microvm`. The upstream `LICENSE` in this directory covers everything under it. - **Upstream:** github.com/kata-containers/kata-containers @@ -17,7 +17,7 @@ everything under it. ## agentpb — kata-agent ttrpc protobufs A copy of the kata-agent protocol-buffer API, used to drive the kata-agent over ttrpc -when ateom boots the guest. +when ateworker boots the guest. - **Path:** `src/libs/protocols/protos/{agent,oci,types,csi}.proto` @@ -41,7 +41,7 @@ the message types are needed. ### RPCs actually used -ateom drives a small subset of `AgentService`: +ateworker drives a small subset of `AgentService`: `CreateSandbox`, `CreateContainer`, `StartContainer`, `UpdateInterface`, `UpdateRoutes`, `AddARPNeighbors`, `ReadStdout`, `ReadStderr`. @@ -50,5 +50,5 @@ ateom drives a small subset of `AgentService`: 1. Check out the matching kata-containers tag and copy `src/libs/protocols/protos/{agent,oci,types,csi}.proto` into `agentpb/`. 2. Set each `option go_package` to - `github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb;agentpb`. + `github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpb`. 3. Regenerate with `protoc --go_out=...` (protoc-gen-go), matching the versions above. diff --git a/cmd/ateom-microvm/internal/third_party/kata/agentpb/agent.pb.go b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/agent.pb.go similarity index 99% rename from cmd/ateom-microvm/internal/third_party/kata/agentpb/agent.pb.go rename to cmd/ateworker-microvm/internal/third_party/kata/agentpb/agent.pb.go index bb578f792d..d1d5e570b4 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/agentpb/agent.pb.go +++ b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/agent.pb.go @@ -4694,7 +4694,7 @@ const file_agent_proto_rawDesc = "" + "\vAddSwapPath\x12\x18.grpc.AddSwapPathRequest\x1a\x16.google.protobuf.Empty\x12E\n" + "\x0eGetVolumeStats\x12\x18.grpc.VolumeStatsRequest\x1a\x19.grpc.VolumeStatsResponse\x12A\n" + "\fResizeVolume\x12\x19.grpc.ResizeVolumeRequest\x1a\x16.google.protobuf.Empty\x12;\n" + - "\tSetPolicy\x12\x16.grpc.SetPolicyRequest\x1a\x16.google.protobuf.EmptyBVZTgithub.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata/agentpb;agentpbb\x06proto3" + "\tSetPolicy\x12\x16.grpc.SetPolicyRequest\x1a\x16.google.protobuf.EmptyBfZdgithub.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpbb\x06proto3" var ( file_agent_proto_rawDescOnce sync.Once diff --git a/cmd/ateom-microvm/internal/third_party/kata/agentpb/agent.proto b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/agent.proto similarity index 99% rename from cmd/ateom-microvm/internal/third_party/kata/agentpb/agent.proto rename to cmd/ateworker-microvm/internal/third_party/kata/agentpb/agent.proto index 462dc9732f..eff52661e4 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/agentpb/agent.proto +++ b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/agent.proto @@ -7,7 +7,7 @@ syntax = "proto3"; -option go_package = "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb;agentpb"; +option go_package = "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpb"; package grpc; diff --git a/cmd/ateom-microvm/internal/third_party/kata/agentpb/csi.pb.go b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/csi.pb.go similarity index 98% rename from cmd/ateom-microvm/internal/third_party/kata/agentpb/csi.pb.go rename to cmd/ateworker-microvm/internal/third_party/kata/agentpb/csi.pb.go index 66d1e8722c..d68026183e 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/agentpb/csi.pb.go +++ b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/csi.pb.go @@ -286,7 +286,7 @@ const file_csi_proto_rawDesc = "" + "\x06INODES\x10\x02\"G\n" + "\x0fVolumeCondition\x12\x1a\n" + "\babnormal\x18\x01 \x01(\bR\babnormal\x12\x18\n" + - "\amessage\x18\x02 \x01(\tR\amessageBVZTgithub.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata/agentpb;agentpbb\x06proto3" + "\amessage\x18\x02 \x01(\tR\amessageBfZdgithub.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpbb\x06proto3" var ( file_csi_proto_rawDescOnce sync.Once diff --git a/cmd/ateom-microvm/internal/third_party/kata/agentpb/csi.proto b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/csi.proto similarity index 96% rename from cmd/ateom-microvm/internal/third_party/kata/agentpb/csi.proto rename to cmd/ateworker-microvm/internal/third_party/kata/agentpb/csi.proto index 6bdf4de818..21d2f89c59 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/agentpb/csi.proto +++ b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/csi.proto @@ -4,7 +4,7 @@ // syntax = "proto3"; -option go_package = "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb;agentpb"; +option go_package = "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpb"; package grpc; diff --git a/cmd/ateom-microvm/internal/third_party/kata/agentpb/oci.pb.go b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/oci.pb.go similarity index 99% rename from cmd/ateom-microvm/internal/third_party/kata/agentpb/oci.pb.go rename to cmd/ateworker-microvm/internal/third_party/kata/agentpb/oci.pb.go index b5f82d0d23..7a704e0855 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/agentpb/oci.pb.go +++ b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/oci.pb.go @@ -2618,7 +2618,7 @@ const file_oci_proto_rawDesc = "" + "\n" + "\bErrnoRet\"5\n" + "\rLinuxIntelRdt\x12$\n" + - "\rL3CacheSchema\x18\x01 \x01(\tR\rL3CacheSchemaBVZTgithub.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata/agentpb;agentpbb\x06proto3" + "\rL3CacheSchema\x18\x01 \x01(\tR\rL3CacheSchemaBfZdgithub.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpbb\x06proto3" var ( file_oci_proto_rawDescOnce sync.Once diff --git a/cmd/ateom-microvm/internal/third_party/kata/agentpb/oci.proto b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/oci.proto similarity index 99% rename from cmd/ateom-microvm/internal/third_party/kata/agentpb/oci.proto rename to cmd/ateworker-microvm/internal/third_party/kata/agentpb/oci.proto index 1d47dd6044..f28cae63b0 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/agentpb/oci.proto +++ b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/oci.proto @@ -6,7 +6,7 @@ // syntax = "proto3"; -option go_package = "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb;agentpb"; +option go_package = "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpb"; package grpc; diff --git a/cmd/ateom-microvm/internal/third_party/kata/agentpb/types.pb.go b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/types.pb.go similarity index 99% rename from cmd/ateom-microvm/internal/third_party/kata/agentpb/types.pb.go rename to cmd/ateworker-microvm/internal/third_party/kata/agentpb/types.pb.go index c2c849ddfa..1a600ed555 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/agentpb/types.pb.go +++ b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/types.pb.go @@ -508,7 +508,7 @@ const file_types_proto_rawDesc = "" + "\x13FSGroupChangePolicy\x12\n" + "\n" + "\x06Always\x10\x00\x12\x12\n" + - "\x0eOnRootMismatch\x10\x01BVZTgithub.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata/agentpb;agentpbb\x06proto3" + "\x0eOnRootMismatch\x10\x01BfZdgithub.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpbb\x06proto3" var ( file_types_proto_rawDescOnce sync.Once diff --git a/cmd/ateom-microvm/internal/third_party/kata/agentpb/types.proto b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/types.proto similarity index 96% rename from cmd/ateom-microvm/internal/third_party/kata/agentpb/types.proto rename to cmd/ateworker-microvm/internal/third_party/kata/agentpb/types.proto index 0b1f626e9d..62c08fe7c2 100644 --- a/cmd/ateom-microvm/internal/third_party/kata/agentpb/types.proto +++ b/cmd/ateworker-microvm/internal/third_party/kata/agentpb/types.proto @@ -7,7 +7,7 @@ syntax = "proto3"; -option go_package = "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb;agentpb"; +option go_package = "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb;agentpb"; package types; diff --git a/cmd/ateom-microvm/main.go b/cmd/ateworker-microvm/main.go similarity index 80% rename from cmd/ateom-microvm/main.go rename to cmd/ateworker-microvm/main.go index f3e90b05c3..b577fc05d2 100644 --- a/cmd/ateom-microvm/main.go +++ b/cmd/ateworker-microvm/main.go @@ -14,8 +14,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Command ateom-microvm is the kata + cloud-hypervisor micro-VM -// implementation of the ateompb.Ateom service, a peer to cmd/ateom-gvisor. +// Command ateworker-microvm is the kata + cloud-hypervisor micro-VM +// implementation of the ateworkerpb.Worker service, a peer to cmd/ateworker-gvisor. // // It runs a substrate actor as a cloud-hypervisor micro-VM (launched via the // kata guest model) and supports full suspend/resume by driving CH's native @@ -41,18 +41,19 @@ import ( "github.com/agent-substrate/substrate/internal/actorlog" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateinterceptors" - "github.com/agent-substrate/substrate/internal/ateom" - "github.com/agent-substrate/substrate/internal/ateomcgroup" - "github.com/agent-substrate/substrate/internal/ateominit" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomstats" - "github.com/agent-substrate/substrate/internal/ateomtunnel" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" + "github.com/agent-substrate/substrate/internal/ateworker/cgroup" + "github.com/agent-substrate/substrate/internal/ateworker/pid1" + "github.com/agent-substrate/substrate/internal/ateworker/register" + "github.com/agent-substrate/substrate/internal/ateworker/tunnel" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/otlprelay" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/preview" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/version" + "github.com/agent-substrate/substrate/internal/workerstats" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "github.com/spf13/pflag" "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" @@ -69,15 +70,16 @@ var ( vmmMemReserve = pflag.Int("vmm-mem-reserve-mib", vmmMemReserveMiB, "Guest RAM (MiB) held back from the pod's memory limit for the cloud-hypervisor VMM + virtiofsd, which run as host processes in the pod cgroup alongside the guest RAM. Prevents the pod OOMing when the VM is sized to the pod's memory limit.") showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") + previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.") otlpRelaySocket = pflag.String("otlp-relay-socket", nodepath.AteletOTLPSocketPath(), "Unix socket of atelet's OTLP relay to export telemetry through, keeping it off the pod network. Empty, or absent at startup, exports directly to OTEL_EXPORTER_OTLP_ENDPOINT instead.") - tunnelConfig = ateomtunnel.RegisterFlags(pflag.CommandLine) + tunnelConfig = tunnel.RegisterFlags(pflag.CommandLine) readinessListenAddress = pflag.String("readiness-listen-address", "0.0.0.0:8080", "Address for the HTTP /readyz and /healthz probes") maxActors = pflag.Int("max-actors", 1000, "How many actors this worker will host at once") - usageSampleInterval = pflag.Duration("usage-sample-interval", time.Minute, "How often the ateom samples the resource usage of its actors. Each sample writes an ate.actor.usage_sampled record per actor, and GetActiveWorkloadStats serves the latest.") + usageSampleInterval = pflag.Duration("usage-sample-interval", time.Minute, "How often the ateworker samples the resource usage of its actors. Each sample writes an ate.actor.usage_sampled record per actor, and GetActiveWorkloadStats serves the latest.") ) // minUsageSampleInterval is the floor of --usage-sample-interval, the same as @@ -85,7 +87,7 @@ var ( const minUsageSampleInterval = 50 * time.Second func main() { - ateominit.RunIfPID1() + pid1.RunIfPID1() pflag.Parse() if *showVersion { fmt.Println(version.String()) @@ -111,7 +113,10 @@ func do(ctx context.Context) error { if err := serverboot.SetLogLevel(*logLevelFlag); err != nil { return err } - slog.InfoContext(ctx, "ateom-microvm booting", slog.String("version", version.Version)) + if err := preview.Init(*previewFlags...); err != nil { + return fmt.Errorf("invalid --preview: %w", err) + } + slog.InfoContext(ctx, "ateworker-microvm booting", slog.String("version", version.Version)) if *maxActors < 0 { return fmt.Errorf("--max-actors must not be negative, got %d", *maxActors) } @@ -119,7 +124,7 @@ func do(ctx context.Context) error { return fmt.Errorf("--usage-sample-interval must be at least %v, got %v", minUsageSampleInterval, *usageSampleInterval) } - const serviceName = "ateom-microvm" + const serviceName = "ateworker-microvm" // Export through atelet's node-local relay when it is there, so telemetry // never touches the worker pod's network. A nil conn means it is not, and // the providers fall back to dialing the collector directly. @@ -128,7 +133,7 @@ func do(ctx context.Context) error { // ends of the same decision: Dial already treats an absent socket as a // fallback rather than an error, and atelet logs and keeps going when it // cannot serve the relay at all. What is lost here is the node-local export - // path, not the ateom's ability to run actors, and failing the worker pod + // path, not the ateworker's ability to run actors, and failing the worker pod // over its telemetry route would turn a misconfigured flag into an outage. relayConn, err := otlprelay.Dial(ctx, *otlpRelaySocket) if err != nil { @@ -144,7 +149,7 @@ func do(ctx context.Context) error { Sampling: serverboot.ResolveTraceSampling(ctx, serverboot.ParentRatioSampling(serverboot.ControlPlaneTraceRatio)), ExporterConn: relayConn, // So the spans say which path they took, including when relayConn is nil - // because the dial above failed and this ateom is exporting directly. + // because the dial above failed and this ateworker is exporting directly. RelayCapable: true, }) if err != nil { @@ -172,15 +177,15 @@ func do(ctx context.Context) error { defer serverboot.ShutdownProvider("LoggerProvider", lp.Shutdown) } - // Create ateom dir. - ateomDir := nodepath.AteomPath(*podUID) + // Create ateworker dir. + ateworkerDir := nodepath.WorkerPath(*podUID) if err := resources.ValidateWorkerPodUID(*podUID); err != nil { return fmt.Errorf("in resources.ValidateWorkerPodUID: %w", err) } // atelet owns the directory and the sandbox versions file in it; only the // socket below is this process's. Both outlive a container restart. - if err := os.MkdirAll(ateomDir, 0o700); err != nil { - return fmt.Errorf("in os.MkdirAll(%q): %w", ateomDir, err) + if err := os.MkdirAll(ateworkerDir, 0o700); err != nil { + return fmt.Errorf("in os.MkdirAll(%q): %w", ateworkerDir, err) } // kata's virtio-fs sharing depends on mount propagation: it slave-binds @@ -196,7 +201,7 @@ func do(ctx context.Context) error { } // Clean up any old socket. - sockPath := nodepath.AteomSocketPath(*podUID) + sockPath := nodepath.WorkerSocketPath(*podUID) if err := os.RemoveAll(sockPath); err != nil { return fmt.Errorf("while removing %q: %w", sockPath, err) } @@ -207,43 +212,43 @@ func do(ctx context.Context) error { } defer func() { if err := os.Remove(sockPath); err != nil && !errors.Is(err, os.ErrNotExist) { - slog.ErrorContext(ctx, "Failed to remove the ateom socket on shutdown", slog.Any("err", err)) + slog.ErrorContext(ctx, "Failed to remove the ateworker socket on shutdown", slog.Any("err", err)) } }() // Forward the actor container's stdout/stderr to the worker pod's stdout as - // JSON with ate.dev/* labels (logging parity with ateom-gvisor). It shares + // JSON with ate.dev/* labels (logging parity with ateworker-gvisor). It shares // logWriter with the runtime logger so the two streams to os.Stdout are // serialized through one SyncedWriter and never interleave-corrupt lines. actorLogger := actorlog.NewActorLogger(logWriter, metadata.OnGCE()) // Give each actor's VMM and virtiofsd a cgroup of their own, so one busy // guest cannot starve the rest. - actorCgroups, err := ateomcgroup.Delegate(ctx) + actorCgroups, err := cgroup.Delegate(ctx) if err != nil { return fmt.Errorf("while delegating the worker cgroup: %w", err) } - tunnel, err := ateomtunnel.Start(ctx, *tunnelConfig, ateomnet.ActorHTTPUpstream) + tunnel, err := tunnel.Start(ctx, *tunnelConfig, actornet.ActorHTTPUpstream) if err != nil { return err } - ateomService := NewService(*podUID, *chBinary, *guestDebug, *vmmMemReserve, *maxActors, tunnel, actorLogger) - ateomService.actorCgroups = actorCgroups + ateworkerService := NewService(*podUID, *chBinary, *guestDebug, *vmmMemReserve, *maxActors, tunnel, actorLogger) + ateworkerService.actorCgroups = actorCgroups // The controller sets both from the downward API. - pool := ateomstats.Pool{Namespace: os.Getenv("POD_NAMESPACE"), Name: os.Getenv("WORKER_POOL_NAME")} + pool := workerstats.Pool{Namespace: os.Getenv("POD_NAMESPACE"), Name: os.Getenv("WORKER_POOL_NAME")} if pool.Namespace == "" || pool.Name == "" { slog.WarnContext(ctx, "Worker pool unknown; usage records will name no pool", slog.Any("pool", pool)) } - usageStdout := ateomstats.NewStdoutHandler(logWriter) + usageStdout := workerstats.NewStdoutHandler(logWriter) defer usageStdout.Close() - ateomService.usage = ateomstats.NewUsageEmitter(lp, usageStdout, pool) - defer ateomstats.StartSampler(ctx, *usageSampleInterval, func(ctx context.Context) { ateomService.sweepUsage(ctx) })() + ateworkerService.usage = workerstats.NewUsageEmitter(lp, usageStdout, pool) + defer workerstats.StartSampler(ctx, *usageSampleInterval, func(ctx context.Context) { ateworkerService.sweepUsage(ctx) })() svr := grpc.NewServer( grpc.StatsHandler(otelgrpc.NewServerHandler()), grpc.UnaryInterceptor(ateinterceptors.InternalServerUnaryInterceptor), ) - ateompb.RegisterAteomServer(svr, ateomService) + ateworkerpb.RegisterWorkerServer(svr, ateworkerService) reflection.Register(svr) readiness := &serverboot.Readiness{} @@ -261,7 +266,7 @@ func do(ctx context.Context) error { readiness.MarkNotReady() // Use a fresh context: the do() context is torn down on return, but the // shutdown must outlive it until the guest has stopped and the VM is down. - ateomService.gracefulShutdown(context.Background()) + ateworkerService.gracefulShutdown(context.Background()) // Only now stop the server, which blocks until any in-flight RPC (notably a // concurrent CheckpointWorkload) has completed, then unblocks svr.Serve below. svr.GracefulStop() @@ -273,8 +278,8 @@ func do(ctx context.Context) error { // outlast, including the window before the Worker record exists; anything // that reaches here is a misconfiguration no restart-in-place will fix. go func() { - err := ateom.Report(ctx, ateom.ReportConfig{ - SocketPath: nodepath.AteomSupportSocket, + err := register.Report(ctx, register.ReportConfig{ + SocketPath: nodepath.WorkerSupportSocket, CredentialBundlePath: tunnelConfig.CredentialBundle, TrustBundlePath: tunnelConfig.TrustBundle, AteletSPIFFEID: tunnelConfig.BrokerIdentity, @@ -288,7 +293,7 @@ func do(ctx context.Context) error { go serverboot.StartHealthServer(ctx, *readinessListenAddress, readiness) - slog.InfoContext(ctx, "ateom-microvm serving", slog.String("socket", sockPath)) + slog.InfoContext(ctx, "ateworker-microvm serving", slog.String("socket", sockPath)) if err := svr.Serve(lis); err != nil { return fmt.Errorf("while serving: %w", err) } @@ -329,9 +334,9 @@ const ( rpcCheckpointWorkload = "CheckpointWorkload" ) -// AteomService is the cloud-hypervisor implementation of ateompb.AteomServer. -type AteomService struct { - ateompb.UnimplementedAteomServer +// AteWorkerService is the cloud-hypervisor implementation of ateworkerpb.WorkerServer. +type AteWorkerService struct { + ateworkerpb.UnimplementedWorkerServer // Serializes lifecycle RPCs per actor. locks *actorlock.Locks @@ -357,11 +362,11 @@ type AteomService struct { // actorLogger forwards the actor container's stdout/stderr to the worker pod's // stdout as ate.dev/*-labeled JSON and emits actor lifecycle events (parity - // with ateom-gvisor). + // with ateworker-gvisor). actorLogger *actorlog.ActorLogger - tunnel *ateomtunnel.Tunnel + tunnel *tunnel.Tunnel // usage writes the usage records. Nil writes none. - usage *ateomstats.UsageEmitter + usage *workerstats.UsageEmitter // Guards actors, draining, and mutable hostedActor fields. actorsMu sync.RWMutex @@ -378,11 +383,11 @@ type AteomService struct { actorCgroups bool } -var _ ateompb.AteomServer = (*AteomService)(nil) +var _ ateworkerpb.WorkerServer = (*AteWorkerService)(nil) -// NewService creates a new AteomService. -func NewService(podUID, chBinary string, guestDebug bool, memReserveMiB, maxActors int, tunnel *ateomtunnel.Tunnel, actorLogger *actorlog.ActorLogger) *AteomService { - return &AteomService{ +// NewService creates a new AteWorkerService. +func NewService(podUID, chBinary string, guestDebug bool, memReserveMiB, maxActors int, tunnel *tunnel.Tunnel, actorLogger *actorlog.ActorLogger) *AteWorkerService { + return &AteWorkerService{ locks: actorlock.New(), inFlight: actorlock.NewInFlight(), actors: map[string]*hostedActor{}, @@ -398,7 +403,7 @@ func NewService(podUID, chBinary string, guestDebug bool, memReserveMiB, maxActo } // beginRPC registers before checking the drain flag so shutdown cannot miss it. -func (s *AteomService) beginRPC(actorUID, name string, cancel context.CancelFunc) (func(), error) { +func (s *AteWorkerService) beginRPC(actorUID, name string, cancel context.CancelFunc) (func(), error) { release := s.inFlight.Add(actorUID, name, cancel) if err := s.rejectIfDraining(); err != nil { release() @@ -408,7 +413,7 @@ func (s *AteomService) beginRPC(actorUID, name string, cancel context.CancelFunc } // validateActorDirs rejects a request whose actor directories are unusable. -func validateActorDirs(actorDirs *ateompb.ActorDirs) error { +func validateActorDirs(actorDirs *ateworkerpb.ActorDirs) error { if errs := resources.ValidateActorDirs(actorDirs, field.NewPath("actor_dirs")); len(errs) > 0 { return apierror.InvalidArgument("%v", errs.ToAggregate()) } @@ -417,15 +422,15 @@ func validateActorDirs(actorDirs *ateompb.ActorDirs) error { // supportedFidelities are the snapshot fidelities this runtime serves: the // rootfs upper is host-backed, so it ships with or without guest memory. -var supportedFidelities = []ateompb.SnapshotFidelity{ - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, +var supportedFidelities = []ateworkerpb.SnapshotFidelity{ + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, } // validateFidelity rejects a checkpoint or restore request whose fidelity // this runtime cannot serve. -func validateFidelity(fidelity ateompb.SnapshotFidelity) error { +func validateFidelity(fidelity ateworkerpb.SnapshotFidelity) error { if errs := resources.ValidateSnapshotFidelity(fidelity, supportedFidelities, field.NewPath("fidelity")); len(errs) > 0 { return apierror.InvalidArgument("%v", errs.ToAggregate()) } @@ -444,9 +449,9 @@ func validateRuntimeAssetPaths(paths map[string]string) error { return nil } -// rejectIfDraining returns a codes.Unavailable error if ateom has begun graceful +// rejectIfDraining returns a codes.Unavailable error if ateworker has begun graceful // shutdown, so the control plane reschedules the actor onto a live worker. -func (s *AteomService) rejectIfDraining() error { +func (s *AteWorkerService) rejectIfDraining() error { if s.shuttingDown.Load() { return apierror.Unavailable("worker draining: not accepting new workloads") } @@ -454,7 +459,7 @@ func (s *AteomService) rejectIfDraining() error { } // cancelStartups cancels all boots and restores, leaving checkpoints running. -func (s *AteomService) cancelStartups(ctx context.Context) { +func (s *AteWorkerService) cancelStartups(ctx context.Context) { for _, actorUID := range s.inFlight.CancelStartups() { slog.InfoContext(ctx, "Cancelling in-progress workload startup RPC due to graceful shutdown", slog.String("actorUID", actorUID)) diff --git a/cmd/ateom-microvm/main_unsupported.go b/cmd/ateworker-microvm/main_unsupported.go similarity index 90% rename from cmd/ateom-microvm/main_unsupported.go rename to cmd/ateworker-microvm/main_unsupported.go index a0416ac57c..c812dd7135 100644 --- a/cmd/ateom-microvm/main_unsupported.go +++ b/cmd/ateworker-microvm/main_unsupported.go @@ -22,6 +22,6 @@ import ( ) func main() { - fmt.Fprintln(os.Stderr, "ateom-microvm is only supported on Linux") + fmt.Fprintln(os.Stderr, "ateworker-microvm is only supported on Linux") os.Exit(1) } diff --git a/cmd/ateom-microvm/net.go b/cmd/ateworker-microvm/net.go similarity index 91% rename from cmd/ateom-microvm/net.go rename to cmd/ateworker-microvm/net.go index 56ac28ffe6..ef4703492e 100644 --- a/cmd/ateom-microvm/net.go +++ b/cmd/ateworker-microvm/net.go @@ -24,13 +24,13 @@ import ( "github.com/vishvananda/netlink" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" + "github.com/agent-substrate/substrate/internal/netns" ) const ( // gatewayMAC is deliberately FIXED (locally administered), unlike - // ateom-gvisor where a random veth MAC is fine. A CH snapshot freezes the + // ateworker-gvisor where a random veth MAC is fine. A CH snapshot freezes the // guest kernel's ARP cache, including the entry for the gateway // 169.254.17.1; restoring against a tap with a fresh random MAC would // blackhole guest egress until that entry expired. @@ -47,7 +47,7 @@ const ( actorTapMTU = 1500 ) -var gatewayHWAddr = ateomnet.MustParseMAC(gatewayMAC) +var gatewayHWAddr = actornet.MustParseMAC(gatewayMAC) // setupActorTap creates the guest's tap with a fixed gateway address and MAC. // Returns the FDs cloud-hypervisor adopts on boot or restore. @@ -80,7 +80,7 @@ func setupActorTap(ctx context.Context, actorNetNS netns.Handle, name string, qu if err := netlink.LinkSetHardwareAddr(link, gatewayHWAddr); err != nil { return fmt.Errorf("setting the gateway MAC on tap %q: %w", name, err) } - if err := netlink.AddrReplace(link, ateomnet.HostVethAddr); err != nil { + if err := netlink.AddrReplace(link, actornet.HostVethAddr); err != nil { return fmt.Errorf("assigning the gateway address to tap %q: %w", name, err) } if err := netlink.LinkSetUp(link); err != nil { diff --git a/cmd/ateom-microvm/net_linux_test.go b/cmd/ateworker-microvm/net_linux_test.go similarity index 92% rename from cmd/ateom-microvm/net_linux_test.go rename to cmd/ateworker-microvm/net_linux_test.go index 5a5fc803e1..0f6b220985 100644 --- a/cmd/ateom-microvm/net_linux_test.go +++ b/cmd/ateworker-microvm/net_linux_test.go @@ -22,10 +22,10 @@ import ( "github.com/vishvananda/netlink" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" - "github.com/agent-substrate/substrate/internal/ateomtunnel" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" + "github.com/agent-substrate/substrate/internal/ateworker/tunnel" "github.com/agent-substrate/substrate/internal/atunnel" + "github.com/agent-substrate/substrate/internal/netns" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/roottest" @@ -40,8 +40,8 @@ func TestHostActorReplacesSameActor(t *testing.T) { if err != nil { t.Fatal(err) } - service := &AteomService{ - tunnel: &ateomtunnel.Tunnel{Egress: egress, EgressPort: 15001}, + service := &AteWorkerService{ + tunnel: &tunnel.Tunnel{Egress: egress, EgressPort: 15001}, actors: map[string]*hostedActor{}, maxActors: 1, } @@ -121,12 +121,12 @@ func TestSetupActorTap(t *testing.T) { } var found bool for _, a := range addrs { - if a.IPNet.String() == ateomnet.HostVethAddr.IPNet.String() { + if a.IPNet.String() == actornet.HostVethAddr.IPNet.String() { found = true } } if !found { - t.Errorf("tap addresses = %v, want the gateway %s the guest routes to", addrs, ateomnet.HostVethAddr) + t.Errorf("tap addresses = %v, want the gateway %s the guest routes to", addrs, actornet.HostVethAddr) } return nil }); err != nil { diff --git a/cmd/ateom-microvm/phaselog.go b/cmd/ateworker-microvm/phaselog.go similarity index 95% rename from cmd/ateom-microvm/phaselog.go rename to cmd/ateworker-microvm/phaselog.go index bde1ee32ea..f277b26cd5 100644 --- a/cmd/ateom-microvm/phaselog.go +++ b/cmd/ateworker-microvm/phaselog.go @@ -18,7 +18,7 @@ package main import "github.com/agent-substrate/substrate/internal/ateattr" -// The phase names, suffixed onto the ateomphaselog duration keys. Kept out of +// The phase names, suffixed onto the phaselog duration keys. Kept out of // ateattr on purpose: that package's SnapshotPhase* values are the // ate.snapshot.phase metric enum, and these are not values of it. "total" is // shared so the two layers' records agree on the denominator. diff --git a/cmd/ateom-microvm/restore.go b/cmd/ateworker-microvm/restore.go similarity index 92% rename from cmd/ateom-microvm/restore.go rename to cmd/ateworker-microvm/restore.go index 560cae3eb1..02bf2971bf 100644 --- a/cmd/ateom-microvm/restore.go +++ b/cmd/ateworker-microvm/restore.go @@ -28,15 +28,15 @@ import ( "time" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/ateomphaselog" - "github.com/agent-substrate/substrate/internal/ateomstats" + "github.com/agent-substrate/substrate/internal/ateworker/phaselog" + "github.com/agent-substrate/substrate/internal/workerstats" - "github.com/agent-substrate/substrate/internal/ateomnet" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/ch" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/ch" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" "github.com/agent-substrate/substrate/internal/imagecache" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/sizing" "github.com/agent-substrate/substrate/internal/wakeupprobe" @@ -104,7 +104,7 @@ func newReseedNonce() ([]byte, error) { // // Contract with atelet: the snapshot's files are in ActorDirs.restore_dir, // and the durable-dir volume directories re-created (empty). -func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.RestoreWorkloadRequest) (resp *ateompb.RestoreWorkloadResponse, retErr error) { +func (s *AteWorkerService) RestoreWorkload(ctx context.Context, req *ateworkerpb.RestoreWorkloadRequest) (resp *ateworkerpb.RestoreWorkloadResponse, retErr error) { if err := validateActorDirs(req.GetActorDirs()); err != nil { return nil, err } @@ -128,7 +128,7 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore } defer release() - if err := s.tunnel.Deactivate(ctx, ateomstats.ActorAttributionFromRequest(req)); err != nil { + if err := s.tunnel.Deactivate(ctx, workerstats.ActorAttributionFromRequest(req)); err != nil { return nil, err } @@ -180,11 +180,11 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore } switch scope := req.GetFidelity(); scope { - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: if err := s.restoreMemoryFidelity(ctx, p, scope, restoreDir, req.GetPreserveRestoreDir(), tStart); err != nil { return nil, err } - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS: // A ROOTFS snapshot holds no guest state either, but it carries each // container's rootfs upper. The cold boot re-materializes the uppers // from the snapshot instead of starting from a pristine dir, overlapped @@ -196,9 +196,9 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore dTotal := time.Since(tStart) slog.InfoContext(ctx, "Actor restored (rootfs upper + durable-dir volumes, cold boot)", slog.String("id", p.actorUID), slog.Duration("total", dTotal)) - ateomphaselog.LogSnapshotPhases(ctx, "Restore timing breakdown", attribution, scope, - ateomphaselog.RestoreDurationKey, nil, []ateomphaselog.Phase{{Name: phaseTotal, D: dTotal}}) - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: + phaselog.LogSnapshotPhases(ctx, "Restore timing breakdown", attribution, scope, + phaselog.RestoreDurationKey, nil, []phaselog.Phase{{Name: phaseTotal, D: dTotal}}) + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: // A VOLUMES snapshot holds no guest state, so this is a cold boot that // happens to start with the volumes already populated. wakeup probe gating comes // with the cold-boot path, so the actor is serving when we return. @@ -210,14 +210,14 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore slog.String("id", p.actorUID), slog.Duration("total", dTotal)) // A cold boot has none of the memory-restore phases, so the total is the // only observation on its record. - ateomphaselog.LogSnapshotPhases(ctx, "Restore timing breakdown", attribution, scope, - ateomphaselog.RestoreDurationKey, nil, []ateomphaselog.Phase{{Name: phaseTotal, D: dTotal}}) + phaselog.LogSnapshotPhases(ctx, "Restore timing breakdown", attribution, scope, + phaselog.RestoreDurationKey, nil, []phaselog.Phase{{Name: phaseTotal, D: dTotal}}) default: return nil, apierror.InvalidArgument("unsupported snapshot fidelity: %v", scope) } s.actorLogger.EmitLifecycleLog(ctx, "Actor restored", attribution) - return &ateompb.RestoreWorkloadResponse{}, nil + return &ateworkerpb.RestoreWorkloadResponse{}, nil } // restoreMemoryFidelity restores a whole-guest snapshot: relaunch cloud-hypervisor @@ -232,7 +232,7 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore // and resume. Guest RAM — the actor's in-memory state and the frozen network config — // comes back from the memory snapshot; the durable-dir volumes were restored by the // caller from their tar. -func (s *AteomService) restoreMemoryFidelity(ctx context.Context, p actorBootParams, scope ateompb.SnapshotFidelity, restoreDir string, preserveRestoreDir bool, tStart time.Time) (retErr error) { +func (s *AteWorkerService) restoreMemoryFidelity(ctx context.Context, p actorBootParams, scope ateworkerpb.SnapshotFidelity, restoreDir string, preserveRestoreDir bool, tStart time.Time) (retErr error) { actorUID := p.actorUID rr := s.resolveRuntime(p.assetPaths) @@ -289,7 +289,7 @@ func (s *AteomService) restoreMemoryFidelity(ctx context.Context, p actorBootPar return apierror.InvalidArgument("actor spec has no containers") } if len(containers) > maxActorContainers { - return apierror.Unimplemented("ateom-microvm supports at most %d containers, got %d", maxActorContainers, len(containers)) + return apierror.Unimplemented("ateworker-microvm supports at most %d containers, got %d", maxActorContainers, len(containers)) } ctrs, err := s.buildActorContainers(p.actorUID, p.actorDirs, containers) if err != nil { @@ -451,7 +451,7 @@ func (s *AteomService) restoreMemoryFidelity(ctx context.Context, p actorBootPar } // Block until every wakeup-probe-enabled container reports 200. - if err := wakeupprobe.WaitAll(ctx, containers, ateomnet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(actorUID))); err != nil { + if err := wakeupprobe.WaitAll(ctx, containers, actornet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(actorUID))); err != nil { return fmt.Errorf("while waiting for container wakeup probe: %w", err) } @@ -476,8 +476,8 @@ func (s *AteomService) restoreMemoryFidelity(ctx context.Context, p actorBootPar // The joinable per-actor record the benchmarking tooling aggregates. The // durable delta is not carried: tDurable is pinned to tLowers today, so it // would always be the zero a record skips. - ateomphaselog.LogSnapshotPhases(ctx, "Restore timing breakdown", p.actorAttribution(), scope, - ateomphaselog.RestoreDurationKey, nil, []ateomphaselog.Phase{ + phaselog.LogSnapshotPhases(ctx, "Restore timing breakdown", p.actorAttribution(), scope, + phaselog.RestoreDurationKey, nil, []phaselog.Phase{ {Name: phasePrep, D: tPrep.Sub(tStart)}, {Name: phaseBundles, D: tBundles.Sub(tPrep)}, {Name: phaseUpperJoin, D: tUpper.Sub(tBundles)}, @@ -564,7 +564,7 @@ func rewriteSnapshotSocketPaths(snapshotDir, id string) error { changed = true } } - // ateom captures the guest console to a file under the source actor's VMDir + // ateworker captures the guest console to a file under the source actor's VMDir // (virtio-console normally, plus the UART in debug mode). On restore those paths // are stale (they point at the golden/source pod's VMDir), so CH's // CreateConsoleDevice fails (No such file or directory). Repoint them at this diff --git a/cmd/ateom-microvm/restore_test.go b/cmd/ateworker-microvm/restore_test.go similarity index 95% rename from cmd/ateom-microvm/restore_test.go rename to cmd/ateworker-microvm/restore_test.go index 0b371ba957..a7118617e0 100644 --- a/cmd/ateom-microvm/restore_test.go +++ b/cmd/ateworker-microvm/restore_test.go @@ -27,11 +27,11 @@ import ( "google.golang.org/grpc/codes" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/ch" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/ch" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" "github.com/agent-substrate/substrate/internal/actorlock" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // writeSnapshotConfig writes a config.json holding the given fs devices (plus the @@ -223,24 +223,24 @@ func TestRewriteSnapshotSocketPaths(t *testing.T) { } func TestRestoreWorkloadRejectsEmptyRestoreDir(t *testing.T) { - s := &AteomService{ + s := &AteWorkerService{ locks: actorlock.New(), inFlight: actorlock.NewInFlight(), } for _, tc := range []struct { name string - req *ateompb.RestoreWorkloadRequest + req *ateworkerpb.RestoreWorkloadRequest }{ { name: "nil actor_dirs", - req: &ateompb.RestoreWorkloadRequest{ActorUid: "actor-a"}, + req: &ateworkerpb.RestoreWorkloadRequest{ActorUid: "actor-a"}, }, { name: "empty restore_dir", - req: &ateompb.RestoreWorkloadRequest{ + req: &ateworkerpb.RestoreWorkloadRequest{ ActorUid: "actor-a", - ActorDirs: &ateompb.ActorDirs{ + ActorDirs: &ateworkerpb.ActorDirs{ RootDir: "/node/actors/actor-a", OciBundleDir: "/node/actors/actor-a/bundle", CheckpointDir: "/node/actors/actor-a/checkpoint-state", @@ -408,9 +408,9 @@ func TestNewReseedNonce(t *testing.T) { } func TestRPCsRejectUntrustedRuntimeAssetPaths(t *testing.T) { - s := &AteomService{} + s := &AteWorkerService{} ctx := context.Background() - dirs := &ateompb.ActorDirs{ + dirs := &ateworkerpb.ActorDirs{ RootDir: "/node/actors/actor-a", OciBundleDir: "/node/actors/actor-a/bundle", CheckpointDir: "/node/actors/actor-a/checkpoint-state", @@ -422,11 +422,11 @@ func TestRPCsRejectUntrustedRuntimeAssetPaths(t *testing.T) { assets := map[string]string{assetCH: "/bin/sh"} for name, call := range map[string]func() error{ "RunWorkload": func() error { - _, err := s.RunWorkload(ctx, &ateompb.RunWorkloadRequest{ActorDirs: dirs, RuntimeAssetPaths: assets}) + _, err := s.RunWorkload(ctx, &ateworkerpb.RunWorkloadRequest{ActorDirs: dirs, RuntimeAssetPaths: assets}) return err }, "RestoreWorkload": func() error { - _, err := s.RestoreWorkload(ctx, &ateompb.RestoreWorkloadRequest{ActorDirs: dirs, RuntimeAssetPaths: assets}) + _, err := s.RestoreWorkload(ctx, &ateworkerpb.RestoreWorkloadRequest{ActorDirs: dirs, RuntimeAssetPaths: assets}) return err }, } { diff --git a/cmd/ateom-microvm/rootfsupper.go b/cmd/ateworker-microvm/rootfsupper.go similarity index 89% rename from cmd/ateom-microvm/rootfsupper.go rename to cmd/ateworker-microvm/rootfsupper.go index 7a5c9bce2f..c8984ebc66 100644 --- a/cmd/ateom-microvm/rootfsupper.go +++ b/cmd/ateworker-microvm/rootfsupper.go @@ -31,7 +31,7 @@ package main // overlay on a virtio-fs upper needed three kernel workarounds. Snapshots // from the tmpfs era still restore, see below.) // -// The directory is owned entirely by ateom (atelet never touches it): created +// The directory is owned entirely by ateworker (atelet never touches it): created // pristine at cold boot, re-materialized from the snapshot at restore, and // removed at teardown — after CleanupSandboxState has dropped the overlay // mounts that use it. @@ -40,8 +40,8 @@ package main // snapshots ship it as one tar per container (rootfsUpperTarFile), taken while // the guest is paused (the share is write-through, so a paused guest's // completed writes are already in the upper). Each tar holds only the contents -// of that container's upperdir: ateom creates the directory layout itself on -// restore, so nothing in the snapshot decides a path ateom later mounts or +// of that container's upperdir: ateworker creates the directory layout itself on +// restore, so nothing in the snapshot decides a path ateworker later mounts or // wipes. A ROOTFS restore puts the upper back and cold-boots on it; a VOLUMES // snapshot deliberately excludes rootfs state, so the workload cold-starts on a // pristine upper. @@ -53,8 +53,8 @@ import ( "path/filepath" "strings" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/tarutil" ) @@ -72,7 +72,7 @@ func rootfsUpperTarFile(containerID string) (string, error) { } // containerNames returns the names of the actor's containers. -func containerNames(containers []*ateompb.Container) []string { +func containerNames(containers []*ateworkerpb.Container) []string { names := make([]string, len(containers)) for i, c := range containers { names[i] = c.GetName() @@ -85,7 +85,7 @@ func containerNames(containers []*ateompb.Container) []string { // a ROOTFS or MEMORY snapshot left in snapshotDir. Either way any previous // activation's contents are gone first. Must run after CleanupSandboxState // has dropped the overlay mounts that used the dir. -func stageRootfsUpper(actorDirs *ateompb.ActorDirs, snapshotDir string, containers []string) error { +func stageRootfsUpper(actorDirs *ateworkerpb.ActorDirs, snapshotDir string, containers []string) error { if snapshotDir == "" { return resetRootfsUpperDir(actorDirs) } @@ -94,10 +94,10 @@ func stageRootfsUpper(actorDirs *ateompb.ActorDirs, snapshotDir string, containe // resetRootfsUpperDir gives a cold boot a pristine upper directory: a cold // boot must start from the bare image, and atelet's actor-dir reset does not -// know about this directory, so ateom wipes any previous activation's contents +// know about this directory, so ateworker wipes any previous activation's contents // itself. The per-container fs/work subdirectories are created by the overlay // staging (kata.StageMergedRootfs). -func resetRootfsUpperDir(actorDirs *ateompb.ActorDirs) error { +func resetRootfsUpperDir(actorDirs *ateworkerpb.ActorDirs) error { dir := rootfsUpperDir(actorDirs) if err := os.RemoveAll(dir); err != nil { return fmt.Errorf("while clearing rootfs upper dir %q: %w", dir, err) diff --git a/cmd/ateom-microvm/rootfsupper_test.go b/cmd/ateworker-microvm/rootfsupper_test.go similarity index 97% rename from cmd/ateom-microvm/rootfsupper_test.go rename to cmd/ateworker-microvm/rootfsupper_test.go index c715711367..0181e1b251 100644 --- a/cmd/ateom-microvm/rootfsupper_test.go +++ b/cmd/ateworker-microvm/rootfsupper_test.go @@ -24,7 +24,7 @@ import ( "github.com/agent-substrate/substrate/internal/tarutil" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // upperDirWith returns a rootfs upper directory laid out the way the host @@ -111,7 +111,7 @@ func TestStageRootfsUpper(t *testing.T) { t.Run("pristine without a snapshot", func(t *testing.T) { stale := upperDirWith(t, map[string]string{"app_ovl/fs/stale.txt": "stale"}) - dirs := &ateompb.ActorDirs{RootDir: filepath.Dir(stale)} + dirs := &ateworkerpb.ActorDirs{RootDir: filepath.Dir(stale)} if err := os.Rename(stale, rootfsUpperDir(dirs)); err != nil { t.Fatal(err) } @@ -126,7 +126,7 @@ func TestStageRootfsUpper(t *testing.T) { t.Run("re-materialized from a snapshot", func(t *testing.T) { stale := upperDirWith(t, map[string]string{"app_ovl/fs/stale.txt": "stale"}) - dirs := &ateompb.ActorDirs{RootDir: filepath.Dir(stale)} + dirs := &ateworkerpb.ActorDirs{RootDir: filepath.Dir(stale)} if err := os.Rename(stale, rootfsUpperDir(dirs)); err != nil { t.Fatal(err) } diff --git a/cmd/ateom-microvm/run.go b/cmd/ateworker-microvm/run.go similarity index 89% rename from cmd/ateom-microvm/run.go rename to cmd/ateworker-microvm/run.go index 9dd7a68875..ea076e6888 100644 --- a/cmd/ateom-microvm/run.go +++ b/cmd/ateworker-microvm/run.go @@ -30,16 +30,16 @@ import ( "time" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/ateomstats" + "github.com/agent-substrate/substrate/internal/workerstats" - "github.com/agent-substrate/substrate/internal/ateomnet" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/ch" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/ch" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" "github.com/agent-substrate/substrate/internal/imagecache" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/sizing" "github.com/agent-substrate/substrate/internal/wakeupprobe" @@ -47,7 +47,7 @@ import ( "golang.org/x/sys/unix" ) -// runningActor holds the live state for one actor's micro-VM. ateom owns the +// runningActor holds the live state for one actor's micro-VM. ateworker owns the // cloud-hypervisor process directly (booted by RunWorkload or relaunched by // RestoreWorkload), so it tracks that process and its api-socket for teardown. type runningActor struct { @@ -58,13 +58,13 @@ type runningActor struct { // base-id file so the chain survives suspend->resume->suspend. baseID string - // ateom owns this CH process (booted at Run or relaunched at Restore). + // ateworker owns this CH process (booted at Run or relaunched at Restore). chCmd *exec.Cmd // vfsdCmd is the virtiofsd serving the unified share (merged rootfs overlay, - // durable-dir volumes, and CSI volumes). ateom owns it; teardownActor + // durable-dir volumes, and CSI volumes). ateworker owns it; teardownActor // kills it after the CH process. vfsdCmd *exec.Cmd - // apiSocket is the CH api-socket for this ateom-owned VMM. + // apiSocket is the CH api-socket for this ateworker-owned VMM. apiSocket string // restoreSourceDir is the snapshot dir this actor was OnDemand-restored from @@ -128,7 +128,7 @@ const kataAgentPath = "/usr/bin/kata-agent" // vmmMemReserveMiB is the DEFAULT guest RAM held back from the pod's memory limit // for the cloud-hypervisor VMM + virtiofsd, which run as host processes in the same // pod cgroup as the guest RAM; without a margin the pod OOMs. Overridable per -// deployment via --vmm-mem-reserve-mib (see AteomService.memReserveMiB). +// deployment via --vmm-mem-reserve-mib (see AteWorkerService.memReserveMiB). // // Measured on the worker pod's cgroup with one 256MiB-guest actor: the VMM stack's own // cost is ~12MiB (anon 8.1 + kernel 3.7; the rest of cloud-hypervisor's RSS is the guest @@ -188,7 +188,7 @@ type actorContainer struct { // spec is the container's OCI spec shaped for micro-VM execution. spec *specs.Spec // imageMounts are the image volumes this container mounts, and where. - imageMounts []*ateompb.ImageVolumeMount + imageMounts []*ateworkerpb.ImageVolumeMount } // resolvedRuntime holds the concrete binary paths for a request, taken from fetched @@ -210,7 +210,7 @@ func firstNonEmpty(vals ...string) string { // resolveRuntime resolves the cloud-hypervisor binary from fetched assets, falling // back to the flag. -func (s *AteomService) resolveRuntime(paths map[string]string) resolvedRuntime { +func (s *AteWorkerService) resolveRuntime(paths map[string]string) resolvedRuntime { return resolvedRuntime{ chBinary: firstNonEmpty(paths[assetCH], s.chBinary), virtiofsd: paths[assetVirtiofsd], @@ -219,7 +219,7 @@ func (s *AteomService) resolveRuntime(paths map[string]string) resolvedRuntime { // RunWorkload boots the actor as a cloud-hypervisor micro-VM and starts its containers. // -// ateom boots cloud-hypervisor directly (no kata shim) and gives each container a +// ateworker boots cloud-hypervisor directly (no kata shim) and gives each container a // rootfs merged ON THE HOST: overlay(image lower + host-disk upper), served over the // one kataShared virtio-fs share. It drives the kata clh boot (vm.create kernel+image+fs, // add-net, vm.boot) and the post-boot setup the shim would otherwise do (agent @@ -230,8 +230,8 @@ func (s *AteomService) resolveRuntime(paths map[string]string) resolvedRuntime { // - The runtime assets (guest kernel, guest OS image, cloud-hypervisor, virtiofsd) // are on disk and passed as runtime asset paths. // - Each container's bundle holds the overlay spec its rootfs/ is composed -// from. ateom builds the OCI spec itself. -func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkloadRequest) (resp *ateompb.RunWorkloadResponse, retErr error) { +// from. ateworker builds the OCI spec itself. +func (s *AteWorkerService) RunWorkload(ctx context.Context, req *ateworkerpb.RunWorkloadRequest) (resp *ateworkerpb.RunWorkloadResponse, retErr error) { if err := validateActorDirs(req.GetActorDirs()); err != nil { return nil, err } @@ -251,7 +251,7 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload } defer release() - if err := s.tunnel.Deactivate(ctx, ateomstats.ActorAttributionFromRequest(req)); err != nil { + if err := s.tunnel.Deactivate(ctx, workerstats.ActorAttributionFromRequest(req)); err != nil { return nil, err } @@ -291,7 +291,7 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload } s.actorLogger.EmitLifecycleLog(ctx, "Actor started", attribution) slog.InfoContext(ctx, "Actor started (overlay rootfs)", slog.String("id", p.actorUID)) - return &ateompb.RunWorkloadResponse{}, nil + return &ateworkerpb.RunWorkloadResponse{}, nil } // actorBootParams is what a cold boot needs about an actor. It comes from a Run @@ -300,17 +300,16 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload type actorBootParams struct { actorRef resources.ActorRef actorUID string - actorDirs *ateompb.ActorDirs + actorDirs *ateworkerpb.ActorDirs templateAtespace string templateName string - containers []*ateompb.Container + containers []*ateworkerpb.Container assetPaths map[string]string // egressGateway is nil unless actor TCP should be redirected through atunnel. - egressGateway *ateompb.EgressGateway + egressGateway *ateworkerpb.EgressGateway // size is the actor's declared limits (from the ActorTemplate), supplied on // the RunWorkload / RestoreWorkload RPC. It sizes the VM itself (vCPUs, // memory); a container's own cgroup limit comes from its declared resources. - // Zero fields keep the kata defaults. size sizing.SandboxSize // rootfsUpperSnapshotDir, when set, names a snapshot directory holding the // per-container rootfs upper tars; the cold boot re-materializes the uppers @@ -356,7 +355,7 @@ const coldBootAttempts = 2 // dead VM does not come back, so the alternative is failing the actor's resume. // Every retry is logged alongside the guest's boot diagnostics, so a guest that // dies at boot is never silent. -func (s *AteomService) coldBootActorRetrying(ctx context.Context, p actorBootParams) error { +func (s *AteWorkerService) coldBootActorRetrying(ctx context.Context, p actorBootParams) error { for attempt := 1; ; attempt++ { err := s.coldBootActor(ctx, p) if err == nil || attempt >= coldBootAttempts || !errors.Is(err, errGuestStopped) { @@ -375,7 +374,7 @@ func (s *AteomService) coldBootActorRetrying(ctx context.Context, p actorBootPar // coldBootActor boots the actor's micro-VM from scratch and starts its // containers, publishing the VM on the actor's hosted record. The caller holds // the actor's lifecycle lock and owns the lifecycle logging. -func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (retErr error) { +func (s *AteWorkerService) coldBootActor(ctx context.Context, p actorBootParams) (retErr error) { actorUID := p.actorUID // All of the actor's containers share the one micro-VM (which is the pod @@ -387,15 +386,15 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re return apierror.InvalidArgument("actor spec has no containers") } if len(containers) > maxActorContainers { - return apierror.Unimplemented("ateom-microvm supports at most %d containers, got %d", maxActorContainers, len(containers)) + return apierror.Unimplemented("ateworker-microvm supports at most %d containers, got %d", maxActorContainers, len(containers)) } - // ateom builds the CH vm.create itself, so it needs the guest kernel + image + // ateworker builds the CH vm.create itself, so it needs the guest kernel + image // paths directly. paths := p.assetPaths kernel, image := paths[assetKernel], paths[assetImage] if kernel == "" || image == "" { - return fmt.Errorf("ateom-microvm requires %q and %q asset paths", assetKernel, assetImage) + return fmt.Errorf("ateworker-microvm requires %q and %q asset paths", assetKernel, assetImage) } rr := s.resolveRuntime(paths) egress, err := s.tunnel.PrepareEgress(ctx, p.attribution(), p.egressGateway) @@ -444,22 +443,19 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re } }() - // Guest sizing + agent kernel params. - memMiB, vcpus, kparams := s.guestConfig() - - // Right-size the VM to the actor's declared limits (see internal/sizing), - // keeping the defaults above as the fallback when a limit is unset. - // vCPUs round up; VM RAM reserves a fixed margin for the VMM + virtiofsd, which - // share the pod cgroup with the guest RAM. A declared memory limit the reserve - // leaves too small to boot is rejected (resolveGuestMemMiB) rather than silently - // falling back to the larger kata default. NB: a MEMORY snapshot restore - // reuses the size baked into the snapshot (restoreMemoryFidelity), so resizing an - // existing actor takes effect on its next cold boot. - sz := p.size - if v := sz.VCPUs(); v > 0 { - vcpus = v - } - memMiB, err = resolveGuestMemMiB(sz.MemoryBytes, s.memReserveMiB, memMiB) + kparams := s.kernelParams() + + // Size the VM to the actor's declared limits (see internal/sizing), which the + // API requires. vCPUs round up; VM RAM reserves a fixed margin for the VMM + + // virtiofsd, which share the pod cgroup with the guest RAM. NB: a MEMORY + // snapshot restore reuses the size baked into the snapshot + // (restoreMemoryFidelity), so resizing an existing actor takes effect on its + // next cold boot. + vcpus := p.size.VCPUs() + if vcpus == 0 { + return errors.New("actor has no cpu limit to size the micro-VM's vCPUs") + } + memMiB, err := resolveGuestMemMiB(p.size.MemoryBytes, s.memReserveMiB) if err != nil { return err } @@ -478,7 +474,7 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re if err := checkResourceEnvelope(ctrs, guestEnvelope{ memMiB: memMiB, vcpus: vcpus, - declaredBytes: sz.MemoryBytes, + declaredBytes: p.size.MemoryBytes, reserveMiB: s.memReserveMiB, }); err != nil { return err @@ -493,7 +489,7 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re // Assemble each container's merged rootfs on the host (overlay of image lower + // host upper, mounted into the shared dir) + durable-dir and CSI volumes (if any), // and start the ONE virtiofsd that serves them all. CH connects to it at vm.create - // and demand-pages for the actor's lifetime, so ateom owns the process (killed in teardownActor). + // and demand-pages for the actor's lifetime, so ateworker owns the process (killed in teardownActor). leaf, err := s.actorLeaf(actorUID, p.size) if err != nil { return err @@ -510,7 +506,7 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re } }() - // Launch a bare VMM (CH + api-socket); ateom owns this process for teardown. + // Launch a bare VMM (CH + api-socket); ateworker owns this process for teardown. apiSocket := filepath.Join(kata.VMDir(actorUID), "clh-api.sock") chCmd, client, err := ch.LaunchVMM(ctx, ch.LaunchVMMOptions{ Binary: rr.chBinary, @@ -598,11 +594,11 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re tContainers := time.Now() // Block until every wakeup-probe-enabled container reports 200. - if err := wakeupprobe.WaitAll(ctx, containers, ateomnet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(actorUID))); err != nil { + if err := wakeupprobe.WaitAll(ctx, containers, actornet.ActorVethIP, wakeupprobe.DialFunc(s.sandboxDialer(actorUID))); err != nil { return fmt.Errorf("while waiting for container wakeup probe: %w", err) } - // Everything from BootVM onward, split. ateom used to log only the total, which + // Everything from BootVM onward, split. ateworker used to log only the total, which // hid where a cold boot actually goes: it is not the guest booting. slog.InfoContext(ctx, "Actor boot phases", slog.String("id", actorUID), slog.Duration("vsock_wait", tVsock.Sub(tBooted)), @@ -648,7 +644,7 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re // and records the bundle rootfs that backs the overlay's RO lower. No host disk is // mounted here — the merged overlays are assembled in stageMergedRootfs after the // sandbox state is clean. Both RunWorkload and RestoreWorkload go through here. -func (s *AteomService) buildActorContainers(actorUID string, actorDirs *ateompb.ActorDirs, containers []*ateompb.Container) ([]actorContainer, error) { +func (s *AteWorkerService) buildActorContainers(actorUID string, actorDirs *ateworkerpb.ActorDirs, containers []*ateworkerpb.Container) ([]actorContainer, error) { ctrs := make([]actorContainer, len(containers)) for i, c := range containers { cn := c.GetName() @@ -691,7 +687,7 @@ func (s *AteomService) buildActorContainers(actorUID string, actorDirs *ateompb. // wipes SharedDir) and stageRootfsUpper (which owns the upper contents). The returned virtiofsd cmd outlives this call (CH // demand-pages from it); the caller owns it (tracked on runningActor, killed // in teardownActor). -func (s *AteomService) stageMergedRootfs(ctx context.Context, rr resolvedRuntime, id string, actorDirs *ateompb.ActorDirs, ctrs []actorContainer, containers []*ateompb.Container, procAttr *syscall.SysProcAttr) (*exec.Cmd, error) { +func (s *AteWorkerService) stageMergedRootfs(ctx context.Context, rr resolvedRuntime, id string, actorDirs *ateworkerpb.ActorDirs, ctrs []actorContainer, containers []*ateworkerpb.Container, procAttr *syscall.SysProcAttr) (*exec.Cmd, error) { upperBase := rootfsUpperDir(actorDirs) for _, c := range ctrs { if err := kata.StageMergedRootfs(ctx, c.bundleRootfs, upperBase, id, c.name); err != nil { @@ -733,25 +729,22 @@ func (s *AteomService) stageMergedRootfs(ctx context.Context, rr resolvedRuntime return vfsdCmd, nil } -// guestConfig returns the default guest sizing and kernel params -func (s *AteomService) guestConfig() (memMiB, vcpus int, kparams string) { - kparams = kata.BaseKernelParams +// kernelParams returns the guest kernel params +func (s *AteWorkerService) kernelParams() string { + kparams := kata.BaseKernelParams if s.guestDebug { kparams = kata.WithAgentDebug(kparams) } - return kata.DefaultMemoryMiB, kata.DefaultVCPUs, kparams + return kparams } // resolveGuestMemMiB returns the micro-VM guest RAM (MiB) for an actor's declared -// memory limit. declaredBytes == 0 means "unset" and returns fallbackMiB -// (kata.DefaultMemoryMiB). Otherwise the guest gets the declared memory minus the VMM -// reserve; if that leaves less than a bootable minimum it errors — naming the limit, -// the reserve, and the minimum — instead of silently reverting to the (larger) -// fallback, which would boot the actor bigger than the worker was sized for and OOM -// the pod (see vmmMemReserveMiB, minGuestMemMiB, and internal/sizing). -func resolveGuestMemMiB(declaredBytes int64, reserveMiB, fallbackMiB int) (int, error) { +// memory limit: the declared memory minus the VMM reserve. It errors if the limit is +// unset or leaves less than a bootable minimum (see vmmMemReserveMiB, minGuestMemMiB, +// and internal/sizing). +func resolveGuestMemMiB(declaredBytes int64, reserveMiB int) (int, error) { if declaredBytes <= 0 { - return fallbackMiB, nil + return 0, errors.New("actor has no memory limit to size the micro-VM's RAM") } declaredMiB := int(declaredBytes / (1024 * 1024)) m := declaredMiB - reserveMiB @@ -859,7 +852,7 @@ func buildFsConfigs(id string) []ch.FsConfig { // does at boot: establish the sandbox once (mounting the kataShared virtio-fs base), // configure guest networking (eth0 IP/MAC/MTU + routes) once, then start each // container on its own overlay rootfs. On failure it dumps guest diagnostics. -func (s *AteomService) startActorContainers(ctx context.Context, ac *kata.AgentClient, id, vsockPath string, ctrs []actorContainer) error { +func (s *AteWorkerService) startActorContainers(ctx context.Context, ac *kata.AgentClient, id, vsockPath string, ctrs []actorContainer) error { // Establish the agent sandbox + the kataShared virtio-fs mount (every // container's merged rootfs, durable volumes, CSI volumes, and system-info // volumes). All containers share it, so use the first container's hostname. @@ -908,7 +901,7 @@ func (s *AteomService) startActorContainers(ctx context.Context, ac *kata.AgentC // dumps the guest's view of the shared tree. // // Its spec binds every declared volume at its mount path. -func (s *AteomService) startRootfsContainer(ctx context.Context, ac *kata.AgentClient, vsockPath string, c actorContainer) error { +func (s *AteWorkerService) startRootfsContainer(ctx context.Context, ac *kata.AgentClient, vsockPath string, c actorContainer) error { cCtx, cancel := context.WithTimeout(ctx, 30*time.Second) err := ac.StartRootfsContainer(cCtx, c.name, c.spec) cancel() @@ -939,7 +932,7 @@ func (s *AteomService) startRootfsContainer(ctx context.Context, ac *kata.AgentC // ending WrapContainerLogs. This keeps the agent connection (which ttrpc allows // concurrent Calls on) alive for forwarding while guaranteeing no goroutine outlives // the connection. -func (s *AteomService) startActorLogForwarding(ac *kata.AgentClient, a resources.ActorAttribution, streamID, containerName string) { +func (s *AteWorkerService) startActorLogForwarding(ac *kata.AgentClient, a resources.ActorAttribution, streamID, containerName string) { go s.actorLogger.WrapContainerLogs(kata.NewStdioReader(context.Background(), ac, streamID, streamID, false), a, containerName) go s.actorLogger.WrapContainerLogs(kata.NewStdioReader(context.Background(), ac, streamID, streamID, true), a, containerName) } @@ -1044,27 +1037,27 @@ func tailString(s string, n int) string { // agent: configure eth0 (IP/MAC/MTU), install the connected + default routes, and // pin the gateway's ARP entry to its fixed MAC (so a restored guest's frozen // neighbor entry stays valid). -func (s *AteomService) configureGuestNetwork(ctx context.Context, ac *kata.AgentClient, mtu uint64) error { +func (s *AteWorkerService) configureGuestNetwork(ctx context.Context, ac *kata.AgentClient, mtu uint64) error { if err := ac.UpdateInterface(ctx, &agentpb.Interface{ - Device: ateomnet.ActorVethName, - Name: ateomnet.ActorVethName, + Device: actornet.ActorVethName, + Name: actornet.ActorVethName, HwAddr: actorGuestMAC, Mtu: mtu, IPAddresses: []*agentpb.IPAddress{ - {Family: agentpb.IPFamily_v4, Address: ateomnet.ActorVethIP, Mask: "30"}, + {Family: agentpb.IPFamily_v4, Address: actornet.ActorVethIP, Mask: "30"}, }, }); err != nil { return err } if err := ac.UpdateRoutes(ctx, []*agentpb.Route{ - {Dest: ateomnet.ActorVethSubnet, Device: ateomnet.ActorVethName, Scope: uint32(unix.RT_SCOPE_LINK), Family: agentpb.IPFamily_v4}, - {Dest: "", Gateway: ateomnet.ActorVethGateway, Device: ateomnet.ActorVethName, Family: agentpb.IPFamily_v4}, + {Dest: actornet.ActorVethSubnet, Device: actornet.ActorVethName, Scope: uint32(unix.RT_SCOPE_LINK), Family: agentpb.IPFamily_v4}, + {Dest: "", Gateway: actornet.ActorVethGateway, Device: actornet.ActorVethName, Family: agentpb.IPFamily_v4}, }); err != nil { return err } return ac.AddARPNeighbors(ctx, []*agentpb.ARPNeighbor{{ - ToIPAddress: &agentpb.IPAddress{Family: agentpb.IPFamily_v4, Address: ateomnet.ActorVethGateway}, - Device: ateomnet.ActorVethName, + ToIPAddress: &agentpb.IPAddress{Family: agentpb.IPFamily_v4, Address: actornet.ActorVethGateway}, + Device: actornet.ActorVethName, Lladdr: gatewayMAC, State: 0x80, // NUD_PERMANENT }}) diff --git a/cmd/ateom-microvm/run_test.go b/cmd/ateworker-microvm/run_test.go similarity index 89% rename from cmd/ateom-microvm/run_test.go rename to cmd/ateworker-microvm/run_test.go index 2d7bb92e7e..bc23538f3e 100644 --- a/cmd/ateom-microvm/run_test.go +++ b/cmd/ateworker-microvm/run_test.go @@ -27,7 +27,7 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" ) // A vsock socket that has gone missing means cloud-hypervisor stopped the VM @@ -100,14 +100,12 @@ func TestDialAgentRetryContextCanceled(t *testing.T) { } } -// resolveGuestMemMiB must honor a declared limit (minus the VMM reserve), fall back -// to the kata default only when the limit is unset, and error — never silently boot -// bigger than declared — when the reserve leaves too little to boot a guest. +// resolveGuestMemMiB must honor a declared limit (minus the VMM reserve) and error +// when the limit is unset or the reserve leaves too little to boot a guest. func TestResolveGuestMemMiB(t *testing.T) { const ( - mib = 1024 * 1024 - reserve = 256 // vmmMemReserveMiB - fallback = 2048 // kata.DefaultMemoryMiB + mib = 1024 * 1024 + reserve = 256 // vmmMemReserveMiB ) tests := []struct { name string @@ -115,7 +113,7 @@ func TestResolveGuestMemMiB(t *testing.T) { wantMiB int wantErr bool }{ - {name: "unset falls back to kata default", declaredMiB: 0, wantMiB: fallback}, + {name: "unset", declaredMiB: 0, wantErr: true}, {name: "declared honored minus reserve", declaredMiB: 1536, wantMiB: 1536 - reserve}, {name: "just above minimum", declaredMiB: reserve + minGuestMemMiB, wantMiB: minGuestMemMiB}, {name: "reserve exactly swallows limit", declaredMiB: reserve, wantErr: true}, @@ -124,7 +122,7 @@ func TestResolveGuestMemMiB(t *testing.T) { } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - got, err := resolveGuestMemMiB(tc.declaredMiB*mib, reserve, fallback) + got, err := resolveGuestMemMiB(tc.declaredMiB*mib, reserve) if tc.wantErr { if err == nil { t.Fatalf("resolveGuestMemMiB(%dMiB) = %d, nil; want an error", tc.declaredMiB, got) @@ -209,18 +207,18 @@ func TestWorkloadIDs(t *testing.T) { } } -func TestGuestConfigDebugConsole(t *testing.T) { - _, _, prodParams := (&AteomService{guestDebug: false}).guestConfig() +func TestKernelParamsDebugConsole(t *testing.T) { + prodParams := (&AteWorkerService{guestDebug: false}).kernelParams() for _, forbidden := range []string{"agent.debug_console", "agent.debug_console_vport", "1026"} { if strings.Contains(prodParams, forbidden) { - t.Errorf("expected guestConfig() with guestDebug=false not to contain %q, but got %q", forbidden, prodParams) + t.Errorf("expected kernelParams() with guestDebug=false not to contain %q, but got %q", forbidden, prodParams) } } - _, _, dbgParams := (&AteomService{guestDebug: true}).guestConfig() + dbgParams := (&AteWorkerService{guestDebug: true}).kernelParams() for _, want := range []string{"agent.log=debug", "agent.debug_console", "agent.debug_console_vport=1026"} { if !strings.Contains(dbgParams, want) { - t.Errorf("expected guestConfig() with guestDebug=true to contain %q, but got %q", want, dbgParams) + t.Errorf("expected kernelParams() with guestDebug=true to contain %q, but got %q", want, dbgParams) } } } diff --git a/cmd/ateom-microvm/sandboxnet.go b/cmd/ateworker-microvm/sandboxnet.go similarity index 84% rename from cmd/ateom-microvm/sandboxnet.go rename to cmd/ateworker-microvm/sandboxnet.go index cb841209fe..3749205ce5 100644 --- a/cmd/ateom-microvm/sandboxnet.go +++ b/cmd/ateworker-microvm/sandboxnet.go @@ -20,8 +20,8 @@ import ( "fmt" "os" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomnet/dns" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns" ) // writeActorResolvConf points the guest resolver at its fixed gateway address. @@ -30,5 +30,5 @@ func writeActorResolvConf(rootfs string) error { if err != nil { return fmt.Errorf("reading the worker pod resolv.conf: %w", err) } - return dns.WriteRootfsResolvConf(rootfs, dns.SandboxResolvConf(ateomnet.ActorVethGateway, pod)) + return dns.WriteRootfsResolvConf(rootfs, dns.SandboxResolvConf(actornet.ActorVethGateway, pod)) } diff --git a/cmd/ateom-microvm/shutdown.go b/cmd/ateworker-microvm/shutdown.go similarity index 95% rename from cmd/ateom-microvm/shutdown.go rename to cmd/ateworker-microvm/shutdown.go index 0094ec567f..2a4e0c427c 100644 --- a/cmd/ateom-microvm/shutdown.go +++ b/cmd/ateworker-microvm/shutdown.go @@ -31,16 +31,16 @@ import ( "syscall" "time" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" ) // workloadGracePeriod is the whole budget for draining the worker on shutdown: // waiting for an in-flight RPC to release the lock and letting the guest -// workloads handle SIGTERM both draw on it, and ateom escalates to SIGKILL once -// it is gone. Matches ateom-gvisor, and is deliberately shorter than the pod's +// workloads handle SIGTERM both draw on it, and ateworker escalates to SIGKILL once +// it is gone. Matches ateworker-gvisor, and is deliberately shorter than the pod's // own termination grace period — 3600s, set by // workerTerminationGracePeriodSeconds in cmd/atecontroller — so the escalation -// happens here rather than as a kubelet SIGKILL of ateom itself. +// happens here rather than as a kubelet SIGKILL of ateworker itself. const workloadGracePeriod = 30 * time.Minute // workloadKillTimeout bounds the post-SIGKILL wait. The VM teardown that @@ -65,7 +65,7 @@ const ( // gracefulShutdown propagates SIGTERM into every running actor's guest and waits // for the workloads to exit, so the caller can exit cleanly. It takes no actor's // lifecycle lock, so a suspend can still land mid-drain. -func (s *AteomService) gracefulShutdown(ctx context.Context) { +func (s *AteWorkerService) gracefulShutdown(ctx context.Context) { // Set this first so an RPC that arrives while we drain is turned away. s.shuttingDown.Store(true) @@ -231,7 +231,7 @@ func stopGuestWorkload(ctx context.Context, agent guestAgent, id, wid string, de // // Liveness is therefore unknown, so report it rather than claiming the workload // exited. Do not escalate: without a working agent there is no way to reach the - // process anyway, and ateom is already on its way out, so the container goes + // process anyway, and ateworker is already on its way out, so the container goes // down with the pod. if !errors.Is(err, context.DeadlineExceeded) && !errors.Is(err, context.Canceled) { return fmt.Errorf("while waiting for workload %q to exit: %w", wid, err) @@ -262,7 +262,7 @@ func stopGuestWorkload(ctx context.Context, agent guestAgent, id, wid string, de } // signalWorkload delivers one signal to a guest workload's init process, bounded -// by signalDeliveryTimeout. ateom sets ExecId equal to ContainerId, so passing wid +// by signalDeliveryTimeout. ateworker sets ExecId equal to ContainerId, so passing wid // for both targets that init process. // // The bound is the point: the shutdown context has no deadline of its own, and diff --git a/cmd/ateom-microvm/shutdown_test.go b/cmd/ateworker-microvm/shutdown_test.go similarity index 98% rename from cmd/ateom-microvm/shutdown_test.go rename to cmd/ateworker-microvm/shutdown_test.go index 7a0b673ba6..43d23e3c22 100644 --- a/cmd/ateom-microvm/shutdown_test.go +++ b/cmd/ateworker-microvm/shutdown_test.go @@ -170,7 +170,7 @@ func TestStopGuestWorkload(t *testing.T) { } // TestStopGuestWorkloadHonorsParentCancellation asserts that a cancelled shutdown -// context stops the drain instead of escalating: ateom is going away anyway, and +// context stops the drain instead of escalating: ateworker is going away anyway, and // the guest goes down with the VM. func TestStopGuestWorkloadHonorsParentCancellation(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) diff --git a/cmd/ateom-microvm/snapshotcache.go b/cmd/ateworker-microvm/snapshotcache.go similarity index 100% rename from cmd/ateom-microvm/snapshotcache.go rename to cmd/ateworker-microvm/snapshotcache.go diff --git a/cmd/ateom-microvm/snapshotcache_test.go b/cmd/ateworker-microvm/snapshotcache_test.go similarity index 98% rename from cmd/ateom-microvm/snapshotcache_test.go rename to cmd/ateworker-microvm/snapshotcache_test.go index d262fafdfc..f6ab7aaa8c 100644 --- a/cmd/ateom-microvm/snapshotcache_test.go +++ b/cmd/ateworker-microvm/snapshotcache_test.go @@ -22,7 +22,7 @@ import ( "path/filepath" "testing" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/ch" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/ch" "github.com/agent-substrate/substrate/internal/snapshotcache" ) diff --git a/cmd/ateom-microvm/spec.go b/cmd/ateworker-microvm/spec.go similarity index 84% rename from cmd/ateom-microvm/spec.go rename to cmd/ateworker-microvm/spec.go index e62c731442..b0a2a88030 100644 --- a/cmd/ateom-microvm/spec.go +++ b/cmd/ateworker-microvm/spec.go @@ -22,20 +22,19 @@ import ( specs "github.com/opencontainers/runtime-spec/specs-go" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" "github.com/agent-substrate/substrate/internal/apierror" ) // guestEnvelope is the ceiling an actor's container limits must fit inside. -// declaredBytes is the actor-level memory limit (0 when the template declares -// none), and reserveMiB the guest RAM held back for the VMM; together they -// explain where memMiB came from, so the error can name the field the user can -// actually raise. +// declaredBytes is the actor-level memory limit and reserveMiB the guest RAM +// held back for the VMM; together they explain where memMiB came from, so the +// error can name the field the user can actually raise. // // memMiB is already net of reserveMiB, while vcpus is not. The asymmetry is // deliberate: an unreduced memory limit would push the worker pod past its own // and get it OOM-killed, whereas CPU is compressible, so the shortfall only -// slows the workload down. cloud-hypervisor's vCPU threads, virtiofsd and ateom +// slows the workload down. cloud-hypervisor's vCPU threads, virtiofsd and ateworker // are host processes drawing on the same worker-pod CPU quota, and the // scheduler's capacity check sets none of it aside, so a container gets somewhat // less CPU than it declared and the in-guest quota is throttled by the host @@ -49,11 +48,8 @@ type guestEnvelope struct { // remedy names the field that raises the memory ceiling. func (e guestEnvelope) remedy() string { - if e.declaredBytes > 0 { - return fmt.Sprintf("raise spec.resources.limits.memory (declared %dMiB, less the %dMiB VMM reserve) or lower the container limits", - e.declaredBytes/(1024*1024), e.reserveMiB) - } - return "lower the limits or use a SandboxConfig with a larger guest" + return fmt.Sprintf("raise spec.resources.limits.memory (declared %dMiB, less the %dMiB VMM reserve) or lower the container limits", + e.declaredBytes/(1024*1024), e.reserveMiB) } // cpuRemedy names the field that raises the vCPU ceiling. Unlike remedy, it @@ -64,10 +60,9 @@ func (e guestEnvelope) cpuRemedy() string { } // checkResourceEnvelope rejects limits the guest can never satisfy. The guest is -// sized from the actor's own declared limits, or from the pool's SandboxConfig -// when the template declares none, so a limit above the guest can never bind: -// the container would hit the guest's own ceiling instead, with an error -// pointing nowhere useful. +// sized from the actor's own declared limits, so a container limit above the +// guest can never bind: the container would hit the guest's own ceiling +// instead, with an error pointing nowhere useful. // // Limits are summed across the actor's containers rather than checked one at a // time, because they share one guest. Errors carry codes.InvalidArgument: the diff --git a/cmd/ateom-microvm/spec_test.go b/cmd/ateworker-microvm/spec_test.go similarity index 91% rename from cmd/ateom-microvm/spec_test.go rename to cmd/ateworker-microvm/spec_test.go index d4aa2a2a91..126dabe2b7 100644 --- a/cmd/ateom-microvm/spec_test.go +++ b/cmd/ateworker-microvm/spec_test.go @@ -26,7 +26,7 @@ import ( "google.golang.org/grpc/codes" "k8s.io/utils/ptr" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ocispec" ) @@ -101,7 +101,7 @@ func TestCheckResourceEnvelope(t *testing.T) { for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { - err := checkResourceEnvelope(tc.ctrs, guestEnvelope{memMiB: 2048, vcpus: 1}) + err := checkResourceEnvelope(tc.ctrs, guestEnvelope{memMiB: 2048, vcpus: 1, declaredBytes: (2048 + 256) * 1024 * 1024, reserveMiB: 256}) if tc.wantErr == "" { if err != nil { t.Fatalf("checkResourceEnvelope() = %v, want nil", err) @@ -182,7 +182,7 @@ func TestCheckResourceEnvelope_MeasuresAgainstPostReserveGuest(t *testing.T) { const mib = 1024 * 1024 const declaredMiB, reserveMiB = 1024, 256 - guestMiB, err := resolveGuestMemMiB(int64(declaredMiB)*mib, reserveMiB, 2048) + guestMiB, err := resolveGuestMemMiB(int64(declaredMiB)*mib, reserveMiB) if err != nil { t.Fatalf("resolveGuestMemMiB() = %v", err) } @@ -201,23 +201,6 @@ func TestCheckResourceEnvelope_MeasuresAgainstPostReserveGuest(t *testing.T) { } } -// With no actor-level limit the guest is the SandboxConfig default, so that -// remains the right thing to point at. -func TestCheckResourceEnvelope_ErrorNamesSandboxConfigWhenUndeclared(t *testing.T) { - const mib = 1024 * 1024 - ctr := actorContainer{name: "hog", spec: &specs.Spec{Linux: &specs.Linux{ - Resources: &specs.LinuxResources{Memory: &specs.LinuxMemory{Limit: ptr.To(int64(4096 * mib))}}, - }}} - - err := checkResourceEnvelope([]actorContainer{ctr}, guestEnvelope{memMiB: 2048, vcpus: 1}) - if err == nil { - t.Fatal("checkResourceEnvelope() = nil, want an error") - } - if !strings.Contains(err.Error(), "SandboxConfig") { - t.Errorf("error %q does not mention SandboxConfig", err.Error()) - } -} - // A volume's path relative to the share is the same on the host and in the // guest. func TestVolumeSubtreePathsMatchTheGuestShare(t *testing.T) { diff --git a/cmd/ateom-microvm/stats.go b/cmd/ateworker-microvm/stats.go similarity index 86% rename from cmd/ateom-microvm/stats.go rename to cmd/ateworker-microvm/stats.go index 6c0f784fb5..d7e8c61d19 100644 --- a/cmd/ateom-microvm/stats.go +++ b/cmd/ateworker-microvm/stats.go @@ -25,11 +25,11 @@ import ( "sync" "time" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/agentstats" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/agentstats" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" ) @@ -64,7 +64,7 @@ type containerStatsReader interface { // guestStatsTarget is everything GetWorkloadStats needs to sample a live guest. // -// It exists so the handler never reads AteomService.running, which lock guards: +// It exists so the handler never reads AteWorkerService.running, which lock guards: // the handler must not take lock (see below), and a map read racing a lifecycle // RPC's write is a data race whatever the read is for. The fields it holds are // the ones RunWorkload and RestoreWorkload already produce. @@ -90,7 +90,7 @@ type guestStatsTarget struct { workloadIDs []string } -// GetWorkloadStats implements ateompb.Ateom/GetWorkloadStats. +// GetWorkloadStats implements ateworkerpb.Worker/GetWorkloadStats. // // The sample comes from inside the guest, not from the host cgroup. On this // runtime the host cgroup holds cloud-hypervisor, whose memory is the guest RAM @@ -101,7 +101,7 @@ type guestStatsTarget struct { // It must not take the actor's lifecycle lock, which is held across a whole // cold boot, snapshot, or restore; blocking there would silence the poller // through the phases whose usage matters most. -func (s *AteomService) GetWorkloadStats(ctx context.Context, req *ateompb.GetWorkloadStatsRequest) (*ateompb.GetWorkloadStatsResponse, error) { +func (s *AteWorkerService) GetWorkloadStats(ctx context.Context, req *ateworkerpb.GetWorkloadStatsRequest) (*ateworkerpb.GetWorkloadStatsResponse, error) { if req.GetActorUid() == "" { return nil, apierror.InvalidArgument("actor_uid is required") } @@ -111,7 +111,7 @@ func (s *AteomService) GetWorkloadStats(ctx context.Context, req *ateompb.GetWor // worker-to-actor mapping wants re-resolving. hosted := s.lookupActor(req.GetActorUid()) if hosted == nil { - return nil, apierror.NotFound("ateom is not executing actor %q", req.GetActorUid()) + return nil, apierror.NotFound("worker is not executing actor %q", req.GetActorUid()) } sample, err := s.measureGuest(ctx, hosted) if err != nil { @@ -132,21 +132,21 @@ func (s *AteomService) GetWorkloadStats(ctx context.Context, req *ateompb.GetWor // underneath them. Pointer identity catches that: hostActor stores a new // record every time. if s.lookupActor(req.GetActorUid()) != hosted { - return nil, apierror.NotFound("ateom stopped executing actor %q while the sample was being taken", req.GetActorUid()) + return nil, apierror.NotFound("worker stopped executing actor %q while the sample was being taken", req.GetActorUid()) } // The reading advanced the activation's CPU, so the cache keeps up with it. hosted.usage.Store(sample) - return &ateompb.GetWorkloadStatsResponse{Sample: sample}, nil + return &ateworkerpb.GetWorkloadStatsResponse{Sample: sample}, nil } -// GetActiveWorkloadStats implements ateompb.Ateom/GetActiveWorkloadStats: the +// GetActiveWorkloadStats implements ateworkerpb.Worker/GetActiveWorkloadStats: the // discovery read. It serves each actor's latest sample from the sampler, so a // poll puts no load of its own on the guests; an actor not sampled yet is // pending. -func (s *AteomService) GetActiveWorkloadStats(ctx context.Context, req *ateompb.GetActiveWorkloadStatsRequest) (*ateompb.GetActiveWorkloadStatsResponse, error) { +func (s *AteWorkerService) GetActiveWorkloadStats(ctx context.Context, req *ateworkerpb.GetActiveWorkloadStatsRequest) (*ateworkerpb.GetActiveWorkloadStatsResponse, error) { hosted := s.hostedActors() - samples := make([]*ateompb.WorkloadStatsSample, 0, len(hosted)) + samples := make([]*ateworkerpb.WorkloadStatsSample, 0, len(hosted)) for _, h := range hosted { sample := h.usage.Latest() if sample == nil { @@ -157,13 +157,13 @@ func (s *AteomService) GetActiveWorkloadStats(ctx context.Context, req *ateompb. // An empty list is "available", per the proto: a normal answer for a // scraper to get, not an error. - return &ateompb.GetActiveWorkloadStatsResponse{Samples: samples}, nil + return &ateworkerpb.GetActiveWorkloadStatsResponse{Samples: samples}, nil } // sweepUsage samples every hosted actor between its initial reading and its // final record, and stores each sample and writes its periodic record. Same // lock discipline as GetWorkloadStats, for the same reasons. -func (s *AteomService) sweepUsage(ctx context.Context) { +func (s *AteWorkerService) sweepUsage(ctx context.Context) { sweepCtx, cancel := context.WithTimeout(ctx, statsSweepBudget) defer cancel() @@ -189,12 +189,12 @@ func (s *AteomService) sweepUsage(ctx context.Context) { // writes its periodic record unless the actor is outside its sampling window or // no longer hosted. A guest not reached before ctx is done, or that does not // answer, is pending, so it stays attributable. -func (s *AteomService) sampleHostedGuest(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) sampleHostedGuest(ctx context.Context, h *hostedActor) { if !h.usage.Sampling() { return } sample := h.usage.WithEpoch(pendingSample(&h.attribution)) - measured, err := h.usage.Measure(ctx, func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { + measured, err := h.usage.Measure(ctx, func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { return s.sampleGuestInSlot(ctx, h) }) if errors.Is(err, errStaleGuestTarget) { @@ -215,8 +215,8 @@ func (s *AteomService) sampleHostedGuest(ctx context.Context, h *hostedActor) { } // measureGuest reads h's guest as a reading of its activation. -func (s *AteomService) measureGuest(ctx context.Context, h *hostedActor) (*ateompb.WorkloadStatsSample, error) { - return h.usage.Measure(ctx, func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { +func (s *AteWorkerService) measureGuest(ctx context.Context, h *hostedActor) (*ateworkerpb.WorkloadStatsSample, error) { + return h.usage.Measure(ctx, func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { return s.sampleGuest(ctx, h) }) } @@ -224,7 +224,7 @@ func (s *AteomService) measureGuest(ctx context.Context, h *hostedActor) (*ateom // sampleGuestInSlot reads h's guest while holding one of s.guestSlots. The slot // is taken inside the reading, so a sweep waiting for another reading of the // same actor holds no slot. -func (s *AteomService) sampleGuestInSlot(ctx context.Context, h *hostedActor) (*ateompb.WorkloadStatsSample, map[string]uint64, error) { +func (s *AteWorkerService) sampleGuestInSlot(ctx context.Context, h *hostedActor) (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { select { case s.guestSlots <- struct{}{}: case <-ctx.Done(): @@ -237,9 +237,9 @@ func (s *AteomService) sampleGuestInSlot(ctx context.Context, h *hostedActor) (* // recordInitial samples a new activation and writes its initial record. It // runs off the resume path, since a guest read can take many seconds, and // bounds that read like one sweep's. It recovers from panics, as a sweep does: -// it runs on its own goroutine, where a panic would take the ateom down. The +// it runs on its own goroutine, where a panic would take the ateworker down. The // activation is then sampled without an initial record. -func (s *AteomService) recordInitial(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) recordInitial(ctx context.Context, h *hostedActor) { actorUID := h.attribution.UID defer func() { if r := recover(); r != nil { @@ -250,7 +250,7 @@ func (s *AteomService) recordInitial(ctx context.Context, h *hostedActor) { }() ctx, cancel := context.WithTimeout(ctx, statsSweepBudget) defer cancel() - sample, err := h.usage.Measure(ctx, func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { + sample, err := h.usage.Measure(ctx, func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { return s.sampleGuestInSlot(ctx, h) }) if err != nil { @@ -263,7 +263,7 @@ func (s *AteomService) recordInitial(ctx context.Context, h *hostedActor) { // recordFinalIfEnded writes h's final record if a checkpoint or terminate tore // its activation down, which unhosts the actor even when a later step fails. // The caller holds the actor's lock. -func (s *AteomService) recordFinalIfEnded(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) recordFinalIfEnded(ctx context.Context, h *hostedActor) { if h != nil && s.lookupActor(h.attribution.UID) != h { s.recordFinal(ctx, h) } @@ -273,7 +273,7 @@ func (s *AteomService) recordFinalIfEnded(ctx context.Context, h *hostedActor) { // A failed read leaves the newest measured sample to stand in. If even that // bound is too much checkpoint latency, drop the read and the final record // uses the newest cached sample instead. -func (s *AteomService) readFinal(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) readFinal(ctx context.Context, h *hostedActor) { readCtx, cancel := context.WithTimeout(ctx, finalReadTimeout) defer cancel() sample, err := s.measureGuest(readCtx, h) @@ -287,9 +287,9 @@ func (s *AteomService) readFinal(ctx context.Context, h *hostedActor) { // recordFinal writes the final record of an activation that a checkpoint or a // terminate ended, from its newest measured sample. -func (s *AteomService) recordFinal(ctx context.Context, h *hostedActor) { +func (s *AteWorkerService) recordFinal(ctx context.Context, h *hostedActor) { pending := h.usage.WithEpoch(pendingSample(&h.attribution)) - h.usage.Final(func(measured *ateompb.WorkloadStatsSample) { + h.usage.Final(func(measured *ateworkerpb.WorkloadStatsSample) { if measured == nil { measured = pending } @@ -301,8 +301,8 @@ func (s *AteomService) recordFinal(ctx context.Context, h *hostedActor) { // read reports it: attribution and the runtime family, measurements absent -- // source stays STATS_SOURCE_UNSPECIFIED, which the sample's contract defines // as "not measured" rather than "measured as zero". -func pendingSample(active *resources.ActorAttribution) *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{ +func pendingSample(active *resources.ActorAttribution) *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{ Atespace: active.Ref.Atespace, ActorName: active.Ref.Name, ActorUid: active.UID, @@ -331,10 +331,10 @@ var errStaleGuestTarget = errors.New("guest agent connection belongs to a differ // callers express the routine ones differently: an error code for the keyed // read, a pending sample for the sweep. The read holds no lifecycle lock, so // the keyed caller re-checks the actor record it loaded after this returns. -func (s *AteomService) sampleGuest(ctx context.Context, h *hostedActor) (*ateompb.WorkloadStatsSample, map[string]uint64, error) { +func (s *AteWorkerService) sampleGuest(ctx context.Context, h *hostedActor) (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { active := &h.attribution // The actor is the one here, but there is no guest to ask yet. Usually that - // is a poll landing in the boot or the restore: the ateom retains the + // is a poll landing in the boot or the restore: the ateworker retains the // attribution from the moment it accepts the actor, and the target is only // published once the containers are up. It is also what a teardown looks // like from here, since teardownActor clears the target before it closes @@ -360,7 +360,7 @@ func (s *AteomService) sampleGuest(ctx context.Context, h *hostedActor) (*ateomp return nil, nil, fmt.Errorf("no container stats from the guest agent: %w", err) } - return &ateompb.WorkloadStatsSample{ + return &ateworkerpb.WorkloadStatsSample{ Atespace: active.Ref.Atespace, ActorName: active.Ref.Name, ActorUid: active.UID, @@ -368,7 +368,7 @@ func (s *AteomService) sampleGuest(ctx context.Context, h *hostedActor) (*ateomp ActorTemplateName: active.TemplateName, SandboxClass: string(atev1alpha1.SandboxClassMicroVM), - Source: ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT, + Source: ateworkerpb.StatsSource_STATS_SOURCE_GUEST_AGENT, MemoryCurrentBytes: sample.MemoryCurrentBytes, MemoryPeakBytes: sample.MemoryPeakBytes, diff --git a/cmd/ateom-microvm/stats_test.go b/cmd/ateworker-microvm/stats_test.go similarity index 86% rename from cmd/ateom-microvm/stats_test.go rename to cmd/ateworker-microvm/stats_test.go index fe22fd8179..6340144dd0 100644 --- a/cmd/ateom-microvm/stats_test.go +++ b/cmd/ateworker-microvm/stats_test.go @@ -30,12 +30,12 @@ import ( "google.golang.org/grpc/codes" "google.golang.org/protobuf/testing/protocmp" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" "github.com/agent-substrate/substrate/internal/actorlock" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/ateomstats" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/internal/workerstats" ) // TestActorBootParamsAttribution pins the mapping from actorBootParams onto the @@ -85,7 +85,7 @@ func TestActorBootParamsAttribution(t *testing.T) { // what the caller sent. The two hops are written in different files, so this is // the assertion that catches them drifting apart. func TestActorBootParamsAttributionMatchesRequest(t *testing.T) { - req := &ateompb.RunWorkloadRequest{ + req := &ateworkerpb.RunWorkloadRequest{ Atespace: "atespace-a", ActorName: "actor-b", ActorUid: "uid-c", @@ -101,7 +101,7 @@ func TestActorBootParamsAttributionMatchesRequest(t *testing.T) { templateName: req.GetActorTemplateName(), } - if diff := cmp.Diff(ateomstats.ActorAttributionFromRequest(req), p.actorAttribution()); diff != "" { + if diff := cmp.Diff(workerstats.ActorAttributionFromRequest(req), p.actorAttribution()); diff != "" { t.Errorf("attribution via actorBootParams differs from attribution via request (-request +params):\n%s", diff) } } @@ -194,14 +194,14 @@ func containerStats(usage, peak, inactiveFile, cpuNanos uint64) *agentpb.CgroupS // containers published to GetWorkloadStats. lock is constructed like NewService // does, since it is a pointer with no usable zero value and // TestGetWorkloadStatsDoesNotTakeLock holds it. -func newStatsService(agent containerStatsReader, workloadIDs ...string) *AteomService { - s := &AteomService{locks: actorlock.New(), actors: map[string]*hostedActor{}, guestSlots: make(chan struct{}, statsFanOut)} +func newStatsService(agent containerStatsReader, workloadIDs ...string) *AteWorkerService { + s := &AteWorkerService{locks: actorlock.New(), actors: map[string]*hostedActor{}, guestSlots: make(chan struct{}, statsFanOut)} hostTestActor(s, testActor, &guestStatsTarget{actorUID: testActor.UID, agent: agent, workloadIDs: workloadIDs}) return s } // hostTestActor registers an actor; a nil target represents an actor still booting. -func hostTestActor(s *AteomService, attribution resources.ActorAttribution, target *guestStatsTarget) *hostedActor { +func hostTestActor(s *AteWorkerService, attribution resources.ActorAttribution, target *guestStatsTarget) *hostedActor { s.actorsMu.Lock() defer s.actorsMu.Unlock() if s.actors == nil { @@ -213,7 +213,7 @@ func hostTestActor(s *AteomService, attribution resources.ActorAttribution, targ } // unhostTestActor removes one actor, standing in for CheckpointWorkload. -func unhostTestActor(s *AteomService, actorUID string) { +func unhostTestActor(s *AteWorkerService, actorUID string) { s.actorsMu.Lock() defer s.actorsMu.Unlock() delete(s.actors, actorUID) @@ -226,7 +226,7 @@ func TestGetWorkloadStats(t *testing.T) { s := newStatsService(agent, "app_ovl") before := time.Now().UnixNano() - got, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + got, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) after := time.Now().UnixNano() if err != nil { t.Fatalf("GetWorkloadStats() error = %v, want nil", err) @@ -238,14 +238,14 @@ func TestGetWorkloadStats(t *testing.T) { // Checked above; zeroed so the rest can be compared as a whole. got.GetSample().ObservedAtUnixNano = 0 - want := &ateompb.GetWorkloadStatsResponse{Sample: &ateompb.WorkloadStatsSample{ + want := &ateworkerpb.GetWorkloadStatsResponse{Sample: &ateworkerpb.WorkloadStatsSample{ Atespace: "space-a", ActorName: "actor-a", ActorUid: "uid-a", ActorTemplateAtespace: "ns-a", ActorTemplateName: "template-a", SandboxClass: "microvm", - Source: ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT, + Source: ateworkerpb.StatsSource_STATS_SOURCE_GUEST_AGENT, MemoryCurrentBytes: 157286400, MemoryPeakBytes: 209715200, MemoryWorkingSetBytes: 136314880, @@ -273,7 +273,7 @@ func TestGetWorkloadStatsSumsContainers(t *testing.T) { }} s := newStatsService(agent, "app_ovl", "sidecar_ovl") - got, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + got, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) if err != nil { t.Fatalf("GetWorkloadStats() error = %v, want nil", err) } @@ -310,7 +310,7 @@ func TestGetWorkloadStatsSkipsUnreadableContainer(t *testing.T) { } s := newStatsService(agent, "app_ovl", "exited_ovl") - got, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + got, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) if err != nil { t.Fatalf("GetWorkloadStats() error = %v, want nil", err) } @@ -329,7 +329,7 @@ func TestGetWorkloadStatsCountsAnsweredContainer(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": nil}} s := newStatsService(agent, "app_ovl") - got, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + got, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) if err != nil { t.Fatalf("GetWorkloadStats() error = %v, want nil", err) } @@ -345,7 +345,7 @@ func TestGetWorkloadStatsErrors(t *testing.T) { name string // service builds the service under test, so each case can put the two // atomics in exactly the state it means to exercise. - service func() *AteomService + service func() *AteWorkerService actorUID string want codes.Code }{ @@ -353,15 +353,15 @@ func TestGetWorkloadStatsErrors(t *testing.T) { // A required field the caller left off: a client bug, distinct from // the races below, so it gets a distinct code. name: "empty actor_uid", - service: func() *AteomService { return newStatsService(healthy, "app_ovl") }, + service: func() *AteWorkerService { return newStatsService(healthy, "app_ovl") }, actorUID: "", want: codes.InvalidArgument, }, { // Not here at all. NOT_FOUND rather than FAILED_PRECONDITION, because // what the caller should do about it is re-resolve, not retry. - name: "ateom is available", - service: func() *AteomService { return &AteomService{} }, + name: "ateworker is available", + service: func() *AteWorkerService { return &AteWorkerService{} }, actorUID: "uid-a", want: codes.NotFound, }, @@ -370,7 +370,7 @@ func TestGetWorkloadStatsErrors(t *testing.T) { // this call. Reporting anyway would file one actor's numbers under // another's name, and it is the same "not here" as the case above. name: "actor_uid does not match the executing workload", - service: func() *AteomService { return newStatsService(healthy, "app_ovl") }, + service: func() *AteWorkerService { return newStatsService(healthy, "app_ovl") }, actorUID: "uid-b", want: codes.NotFound, }, @@ -379,8 +379,8 @@ func TestGetWorkloadStatsErrors(t *testing.T) { // a poll landing in the boot or the restore, or one landing after // teardownActor cleared the target. The transient case. name: "no guest agent connection yet", - service: func() *AteomService { - s := &AteomService{} + service: func() *AteWorkerService { + s := &AteWorkerService{} hostTestActor(s, testActor, nil) return s }, @@ -392,7 +392,7 @@ func TestGetWorkloadStatsErrors(t *testing.T) { // lock — so a disagreement is an invariant violation, not a routine // state: Internal, unlike every other way sampleGuest declines. name: "guest agent connection belongs to another actor", - service: func() *AteomService { + service: func() *AteWorkerService { s := newStatsService(healthy, "app_ovl") hostTestActor(s, testActor, &guestStatsTarget{actorUID: "uid-b", agent: healthy, workloadIDs: []string{"app_ovl"}}) return s @@ -405,9 +405,9 @@ func TestGetWorkloadStatsErrors(t *testing.T) { // sandbox is going away, which the next CheckpointWorkload turns into // the NOT_FOUND above, or the agent is briefly unreachable. Either way // it is "no numbers right now" rather than Internal — the guest not - // answering is a routine state here, not a bug in this ateom. + // answering is a routine state here, not a bug in this ateworker. name: "no container answers", - service: func() *AteomService { + service: func() *AteWorkerService { agent := &fakeAgent{errs: map[string]error{ "app_ovl": errors.New("ttrpc: closed"), "sidecar_ovl": errors.New("ttrpc: closed"), @@ -421,13 +421,13 @@ func TestGetWorkloadStatsErrors(t *testing.T) { // The boot path does not produce an actor with no containers, so a // confident zero would be reporting a state we do not understand. name: "no containers to measure", - service: func() *AteomService { return newStatsService(healthy) }, + service: func() *AteWorkerService { return newStatsService(healthy) }, actorUID: "uid-a", want: codes.FailedPrecondition, }, } { t.Run(tc.name, func(t *testing.T) { - resp, err := tc.service().GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: tc.actorUID}) + resp, err := tc.service().GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: tc.actorUID}) if resp != nil { t.Errorf("GetWorkloadStats() returned response %v, want nil", resp) } @@ -451,23 +451,23 @@ func TestGetWorkloadStatsDoesNotTakeLock(t *testing.T) { } defer s.locks.Unlock(testActor.UID) - if _, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}); err != nil { + if _, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}); err != nil { t.Errorf("GetWorkloadStats() error = %v, want nil", err) } } -// TestAteomServiceStartsAvailable checks that a freshly constructed service -// retains no attribution and offers no guest, mirroring the gVisor ateom's test +// TestAteWorkerServiceStartsAvailable checks that a freshly constructed service +// retains no attribution and offers no guest, mirroring the gVisor ateworker's test // of the same name. GetWorkloadStats's NOT_FOUND-when-available behavior is -// built on the first: a non-nil zero value would make an idle ateom report an +// built on the first: a non-nil zero value would make an idle ateworker report an // empty actor's usage instead of refusing. -func TestAteomServiceStartsAvailable(t *testing.T) { - s := &AteomService{} +func TestAteWorkerServiceStartsAvailable(t *testing.T) { + s := &AteWorkerService{} if got := s.hostedActors(); len(got) != 0 { - t.Errorf("new AteomService hosts %d actors, want none", len(got)) + t.Errorf("new AteWorkerService hosts %d actors, want none", len(got)) } if got := s.lookupActor(testActor.UID); got != nil { - t.Errorf("new AteomService.lookupActor(%q) = %v, want nil", testActor.UID, got) + t.Errorf("new AteWorkerService.lookupActor(%q) = %v, want nil", testActor.UID, got) } } @@ -488,7 +488,7 @@ func TestGetActiveWorkloadStats(t *testing.T) { // The keyed read against the same fake is the reference: the discovery read // must produce the identical sample, since both are the same measurement // with a different addressing mode. - want, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + want, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) if err != nil { t.Fatalf("GetWorkloadStats() error = %v, want nil", err) } @@ -501,24 +501,24 @@ func TestGetActiveWorkloadStats(t *testing.T) { } // TestGetActiveWorkloadStatsAvailable pins the contract that makes the -// discovery read scrapeable: an idle ateom is an empty list, never an error. +// discovery read scrapeable: an idle ateworker is an empty list, never an error. func TestGetActiveWorkloadStatsAvailable(t *testing.T) { - s := &AteomService{} + s := &AteWorkerService{} got, err := sweepAndList(s) if err != nil { - t.Fatalf("GetActiveWorkloadStats() on an available ateom: error = %v, want nil", err) + t.Fatalf("GetActiveWorkloadStats() on an available ateworker: error = %v, want nil", err) } if n := len(got.GetSamples()); n != 0 { - t.Errorf("GetActiveWorkloadStats() on an available ateom = %v, want no samples", got) + t.Errorf("GetActiveWorkloadStats() on an available ateworker = %v, want no samples", got) } } // pendingFor is the entry the discovery read answers for a workload with no // numbers yet: attribution and the runtime family, source UNSPECIFIED, // measurements absent. observed_at is zeroed by the caller before comparing. -func pendingFor(attr resources.ActorAttribution) *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{ +func pendingFor(attr resources.ActorAttribution) *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{ Atespace: attr.Ref.Atespace, ActorName: attr.Ref.Name, ActorUid: attr.UID, @@ -534,7 +534,7 @@ func pendingFor(attr resources.ActorAttribution) *ateompb.WorkloadStatsSample { // routinely as idle workers, and the entry keeps a workload that dies during // boot attributable. func TestGetActiveWorkloadStatsBooting(t *testing.T) { - s := &AteomService{guestSlots: make(chan struct{}, statsFanOut)} + s := &AteWorkerService{guestSlots: make(chan struct{}, statsFanOut)} hostTestActor(s, testActor, nil) // attribution retained, target not published got, err := sweepAndList(s) @@ -573,7 +573,7 @@ func TestGetActiveWorkloadStatsSeveralActors(t *testing.T) { if len(got.GetSamples()) != 2 { t.Fatalf("GetActiveWorkloadStats() returned %d samples, want 2: %v", len(got.GetSamples()), got) } - byUID := map[string]*ateompb.WorkloadStatsSample{} + byUID := map[string]*ateworkerpb.WorkloadStatsSample{} for _, sample := range got.GetSamples() { byUID[sample.GetActorUid()] = sample } @@ -607,7 +607,7 @@ func TestGetActiveWorkloadStatsOneBooting(t *testing.T) { t.Fatalf("GetActiveWorkloadStats() returned %d samples, want 2: %v", len(got.GetSamples()), got) } for _, sample := range got.GetSamples() { - measured := sample.GetSource() != ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED + measured := sample.GetSource() != ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED if want := sample.GetActorUid() == testActor.UID; measured != want { t.Errorf("actor %q measured = %v, want %v", sample.GetActorUid(), measured, want) } @@ -624,7 +624,7 @@ func TestGetWorkloadStatsTransition(t *testing.T) { s := newStatsService(agent, "app_ovl") agent.onCall = func() { unhostTestActor(s, testActor.UID) } - _, err := s.GetWorkloadStats(context.Background(), &ateompb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) + _, err := s.GetWorkloadStats(context.Background(), &ateworkerpb.GetWorkloadStatsRequest{ActorUid: "uid-a"}) if got := apierror.Code(err); got != codes.NotFound { t.Errorf("GetWorkloadStats() during transition: code = %v, want %v (err: %v)", got, codes.NotFound, err) } @@ -647,7 +647,7 @@ func TestGetActiveWorkloadStatsStaleTarget(t *testing.T) { if n := len(got.GetSamples()); n != 1 { t.Fatalf("GetActiveWorkloadStats() with stale target returned %d samples, want 1", n) } - if src := got.GetSamples()[0].GetSource(); src != ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED { + if src := got.GetSamples()[0].GetSource(); src != ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED { t.Errorf("sample source = %v, want pending", src) } } @@ -677,7 +677,7 @@ func TestGetActiveWorkloadStatsSamplesGuestsConcurrently(t *testing.T) { } }, } - s := &AteomService{locks: actorlock.New(), actors: map[string]*hostedActor{}, guestSlots: make(chan struct{}, statsFanOut)} + s := &AteWorkerService{locks: actorlock.New(), actors: map[string]*hostedActor{}, guestSlots: make(chan struct{}, statsFanOut)} for i := range actors { attr := testActor attr.UID = fmt.Sprintf("uid-%d", i) @@ -708,24 +708,24 @@ func TestGetActiveWorkloadStatsTransition(t *testing.T) { tests := []struct { name string // during changes the hosted set inside the guest read. - during func(s *AteomService) - want []*ateompb.WorkloadStatsSample + during func(s *AteWorkerService) + want []*ateworkerpb.WorkloadStatsSample }{ { name: "re-hosted on another template", - during: func(s *AteomService) { hostTestActor(s, otherTemplate, nil) }, - want: []*ateompb.WorkloadStatsSample{pendingFor(otherTemplate)}, + during: func(s *AteWorkerService) { hostTestActor(s, otherTemplate, nil) }, + want: []*ateworkerpb.WorkloadStatsSample{pendingFor(otherTemplate)}, }, { name: "to another actor", - during: func(s *AteomService) { + during: func(s *AteWorkerService) { unhostTestActor(s, testActor.UID) hostTestActor(s, otherActor, nil) }, }, { name: "to available", - during: func(s *AteomService) { unhostTestActor(s, testActor.UID) }, + during: func(s *AteWorkerService) { unhostTestActor(s, testActor.UID) }, }, } for _, tc := range tests { @@ -739,7 +739,7 @@ func TestGetActiveWorkloadStatsTransition(t *testing.T) { if err != nil { t.Fatalf("GetActiveWorkloadStats() during transition: error = %v, want nil", err) } - var samples []*ateompb.WorkloadStatsSample + var samples []*ateworkerpb.WorkloadStatsSample for _, sample := range got.GetSamples() { // otherActor was not snapshotted by this read. if sample.GetActorUid() == otherActor.UID { @@ -759,14 +759,14 @@ func TestGetActiveWorkloadStatsTransition(t *testing.T) { // so the sweep samples it. It starts at the unix epoch, so a sample's epoch is // zero and the expected samples here need not name it; the epoch has tests of // its own. -func testActivation() *ateomstats.Activation { - a := ateomstats.NewActivation(time.Unix(0, 0), false) +func testActivation() *workerstats.Activation { + a := workerstats.NewActivation(time.Unix(0, 0), false) a.Initial(nil, nil) return a } // sweepAndList runs one sampler sweep, then the discovery read that serves it. -func sweepAndList(s *AteomService) (*ateompb.GetActiveWorkloadStatsResponse, error) { +func sweepAndList(s *AteWorkerService) (*ateworkerpb.GetActiveWorkloadStatsResponse, error) { s.sweepUsage(context.Background()) - return s.GetActiveWorkloadStats(context.Background(), &ateompb.GetActiveWorkloadStatsRequest{}) + return s.GetActiveWorkloadStats(context.Background(), &ateworkerpb.GetActiveWorkloadStatsRequest{}) } diff --git a/cmd/ateom-microvm/systeminfo.go b/cmd/ateworker-microvm/systeminfo.go similarity index 89% rename from cmd/ateom-microvm/systeminfo.go rename to cmd/ateworker-microvm/systeminfo.go index d237b9e262..32157e07b0 100644 --- a/cmd/ateom-microvm/systeminfo.go +++ b/cmd/ateworker-microvm/systeminfo.go @@ -24,7 +24,7 @@ // ActorDirs.system_info_volume_roots_dir and wipes/rebuilds them when // the actor's directories are reset. // -// ateom exposes that host directory to the guest under the single kataShared +// ateworker exposes that host directory to the guest under the single kataShared // virtio-fs share at SharedDir(actorUID)/system-info, like durable-dir and CSI // volumes — no extra virtio-fs device. Read-only is enforced twice: the host bind is // remounted read-only (so nothing in the guest can write through the share), @@ -47,15 +47,15 @@ import ( "os" "path/filepath" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/kata" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/kata" "github.com/agent-substrate/substrate/internal/ocispec" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "golang.org/x/sys/unix" ) // hasSystemInfoVolumes reports whether any container mounts a system-info // volume. -func hasSystemInfoVolumes(containers []*ateompb.Container) bool { +func hasSystemInfoVolumes(containers []*ateworkerpb.Container) bool { for _, c := range containers { if len(c.GetSystemInfoVolumeMounts()) > 0 { return true @@ -68,7 +68,7 @@ func hasSystemInfoVolumes(containers []*ateompb.Container) bool { // directory, into the sandbox's shared virtio-fs tree at SharedDir(actorUID)/system-info, // then remounts the bind read-only: atelet is the only writer, and it writes // the host source directly, never through the share. -func (s *AteomService) stageSystemInfoVolumes(ctx context.Context, actorUID, src string) error { +func (s *AteWorkerService) stageSystemInfoVolumes(ctx context.Context, actorUID, src string) error { if _, err := os.Stat(src); err != nil { return fmt.Errorf("while checking system-info volumes dir %q: %w", src, err) } diff --git a/cmd/ateom-microvm/usage_test.go b/cmd/ateworker-microvm/usage_test.go similarity index 86% rename from cmd/ateom-microvm/usage_test.go rename to cmd/ateworker-microvm/usage_test.go index b62a8f62e6..b78f71482c 100644 --- a/cmd/ateom-microvm/usage_test.go +++ b/cmd/ateworker-microvm/usage_test.go @@ -23,27 +23,27 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/cmd/ateom-microvm/internal/third_party/kata/agentpb" + "github.com/agent-substrate/substrate/cmd/ateworker-microvm/internal/third_party/kata/agentpb" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/ateomstats" - "github.com/agent-substrate/substrate/internal/ateomstats/ateomstatstest" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" + "github.com/agent-substrate/substrate/internal/workerstats" + "github.com/agent-substrate/substrate/internal/workerstats/workerstatstest" ) -func withUsageRecorder(s *AteomService) *ateomstatstest.Recorder { - e, rec := ateomstatstest.NewEmitter() +func withUsageRecorder(s *AteWorkerService) *workerstatstest.Recorder { + e, rec := workerstatstest.NewEmitter() s.usage = e return rec } // running is a after its initial reading, so the sweep samples it. -func running(a *ateomstats.Activation) *ateomstats.Activation { +func running(a *workerstats.Activation) *workerstats.Activation { a.Initial(nil, nil) return a } // setActivation replaces the test actor's activation. -func setActivation(s *AteomService, a *ateomstats.Activation) *hostedActor { +func setActivation(s *AteWorkerService, a *workerstats.Activation) *hostedActor { h := s.lookupActor(testActor.UID) s.actorsMu.Lock() h.usage = a @@ -55,13 +55,13 @@ func TestSweepStampsEpochAndDiscoveryServesIt(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") epoch := time.Unix(1700, 0) - setActivation(s, running(ateomstats.NewActivation(epoch, false))) + setActivation(s, running(workerstats.NewActivation(epoch, false))) - before, err := s.GetActiveWorkloadStats(context.Background(), &ateompb.GetActiveWorkloadStatsRequest{}) + before, err := s.GetActiveWorkloadStats(context.Background(), &ateworkerpb.GetActiveWorkloadStatsRequest{}) if err != nil { t.Fatal(err) } - if got := before.GetSamples()[0]; got.GetSource() != ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED || got.GetEpochUnixNano() != epoch.UnixNano() { + if got := before.GetSamples()[0]; got.GetSource() != ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED || got.GetEpochUnixNano() != epoch.UnixNano() { t.Errorf("before a sweep: source %v epoch %d, want pending with epoch %d", got.GetSource(), got.GetEpochUnixNano(), epoch.UnixNano()) } if len(agent.calls) != 0 { @@ -69,7 +69,7 @@ func TestSweepStampsEpochAndDiscoveryServesIt(t *testing.T) { } s.sweepUsage(context.Background()) - after, err := s.GetActiveWorkloadStats(context.Background(), &ateompb.GetActiveWorkloadStatsRequest{}) + after, err := s.GetActiveWorkloadStats(context.Background(), &ateworkerpb.GetActiveWorkloadStatsRequest{}) if err != nil { t.Fatal(err) } @@ -83,7 +83,7 @@ func TestSweepStampsEpochAndDiscoveryServesIt(t *testing.T) { func TestResumedGuestCPUIsActivationRelative(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 9_000_000)}} s := newStatsService(agent, "app_ovl") - h := setActivation(s, running(ateomstats.NewActivation(time.Now(), true))) + h := setActivation(s, running(workerstats.NewActivation(time.Now(), true))) s.sweepUsage(context.Background()) if got := h.usage.Latest().GetCpuUsageUsec(); got != 0 { @@ -102,7 +102,7 @@ func TestRecordInitialAndFinal(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") rec := withUsageRecorder(s) - h := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + h := setActivation(s, workerstats.NewActivation(time.Now(), false)) s.recordInitial(context.Background(), h) if h.usage.Latest() == nil { @@ -125,7 +125,7 @@ func TestRecordInitialAndFinal(t *testing.T) { func TestRecordFinalIfEnded(t *testing.T) { s := newStatsService(&fakeAgent{}, "app_ovl") rec := withUsageRecorder(s) - h := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + h := setActivation(s, workerstats.NewActivation(time.Now(), false)) s.recordFinalIfEnded(context.Background(), h) if got := rec.Kinds(); len(got) != 0 { @@ -143,7 +143,7 @@ func TestRecordFinalIfEnded(t *testing.T) { func TestReadFinal(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") - h := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + h := setActivation(s, workerstats.NewActivation(time.Now(), false)) s.recordInitial(context.Background(), h) agent.stats["app_ovl"] = containerStats(1000, 2000, 100, 9_000_000) @@ -166,7 +166,7 @@ func TestEndPreviousActivation(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") rec := withUsageRecorder(s) - old := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + old := setActivation(s, workerstats.NewActivation(time.Now(), false)) if err := s.endPreviousActivation(context.Background(), testActor.UID, nil); err != nil { t.Fatal(err) @@ -180,7 +180,7 @@ func TestEndPreviousActivation(t *testing.T) { func TestRecordFinalWithNoSampleIsPending(t *testing.T) { s := newStatsService(&fakeAgent{}, "app_ovl") rec := withUsageRecorder(s) - h := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + h := setActivation(s, workerstats.NewActivation(time.Now(), false)) s.recordFinal(context.Background(), h) if len(rec.Sources()) != 1 || rec.Sources()[0] != ateattr.StatsSourceUnspecified { t.Errorf("final record sources = %v, want one pending", rec.Sources()) @@ -193,7 +193,7 @@ func TestRecordInitialAfterFinal(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") rec := withUsageRecorder(s) - h := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + h := setActivation(s, workerstats.NewActivation(time.Now(), false)) agent.onCall = func() { s.recordFinal(context.Background(), h) } s.recordInitial(context.Background(), h) @@ -208,7 +208,7 @@ func TestRecordInitialRecoversFromPanic(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") rec := withUsageRecorder(s) - h := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + h := setActivation(s, workerstats.NewActivation(time.Now(), false)) agent.onCall = func() { panic("guest read") } s.recordInitial(context.Background(), h) @@ -226,7 +226,7 @@ func TestSweepRecoversFromPanic(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") rec := withUsageRecorder(s) - setActivation(s, running(ateomstats.NewActivation(time.Now(), false))) + setActivation(s, running(workerstats.NewActivation(time.Now(), false))) agent.onCall = func() { panic("guest read") } s.sweepUsage(context.Background()) @@ -241,7 +241,7 @@ func TestRecordInitialTakesASlot(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") withUsageRecorder(s) - h := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + h := setActivation(s, workerstats.NewActivation(time.Now(), false)) for range statsFanOut { s.guestSlots <- struct{}{} } @@ -260,7 +260,7 @@ func TestRecordInitialAfterRehost(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") withUsageRecorder(s) - old := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + old := setActivation(s, workerstats.NewActivation(time.Now(), false)) target := s.guestOf(old) agent.onCall = func() { hostTestActor(s, testActor, target) } @@ -282,7 +282,7 @@ func TestPartialGuestReadDoesNotDoubleCharge(t *testing.T) { "b_ovl": containerStats(1000, 2000, 100, 400_000_000), }} s := newStatsService(agent, "a_ovl", "b_ovl") - h := setActivation(s, running(ateomstats.NewActivation(time.Now(), true))) + h := setActivation(s, running(workerstats.NewActivation(time.Now(), true))) s.sweepUsage(context.Background()) agent.mu.Lock() @@ -311,7 +311,7 @@ func TestSweepWaitsForInitialAndStopsAtFinal(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") rec := withUsageRecorder(s) - h := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + h := setActivation(s, workerstats.NewActivation(time.Now(), false)) s.sweepUsage(context.Background()) if len(agent.calls) != 0 { @@ -339,9 +339,9 @@ func TestRecordInitialReadsItsOwnGuest(t *testing.T) { newAgent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 9_000_000)}} s := newStatsService(oldAgent, "app_ovl") withUsageRecorder(s) - old := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + old := setActivation(s, workerstats.NewActivation(time.Now(), false)) fresh := hostTestActor(s, testActor, &guestStatsTarget{actorUID: testActor.UID, agent: newAgent, workloadIDs: []string{"app_ovl"}}) - fresh.usage = ateomstats.NewActivation(time.Now(), false) + fresh.usage = workerstats.NewActivation(time.Now(), false) s.recordInitial(context.Background(), old) if got := old.usage.Latest().GetCpuUsageUsec(); got != 5000 { @@ -362,10 +362,10 @@ func TestLateInitialLeavesNewActivationAlone(t *testing.T) { agent := &fakeAgent{stats: map[string]*agentpb.CgroupStats{"app_ovl": containerStats(1000, 2000, 100, 5_000_000)}} s := newStatsService(agent, "app_ovl") rec := withUsageRecorder(s) - old := setActivation(s, ateomstats.NewActivation(time.Now(), false)) + old := setActivation(s, workerstats.NewActivation(time.Now(), false)) s.recordFinal(context.Background(), old) fresh := hostTestActor(s, testActor, nil) - fresh.usage = ateomstats.NewActivation(time.Now(), false) + fresh.usage = workerstats.NewActivation(time.Now(), false) s.recordInitial(context.Background(), old) if got := rec.Kinds(); len(got) != 1 || got[0] != ateattr.StatsKindFinal { diff --git a/cmd/benchmarking/isolate/fake-atelet/herder.go b/cmd/benchmarking/isolate/fake-atelet/herder.go index 48a34b4fa7..ec25740f43 100644 --- a/cmd/benchmarking/isolate/fake-atelet/herder.go +++ b/cmd/benchmarking/isolate/fake-atelet/herder.go @@ -44,16 +44,16 @@ type objectWriter interface { // actorCertMinter is the part of ateapipb.WorkerServiceClient the herder uses. type actorCertMinter interface { - MintAteomActorCertificate(ctx context.Context, in *ateapipb.MintAteomActorCertificateRequest, opts ...grpc.CallOption) (*ateapipb.MintAteomActorCertificateResponse, error) + MintWorkerActorCertificate(ctx context.Context, in *ateapipb.MintWorkerActorCertificateRequest, opts ...grpc.CallOption) (*ateapipb.MintWorkerActorCertificateResponse, error) } -// delays is how long each AteomHerder call takes before it succeeds: the +// delays is how long each Atelet call takes before it succeeds: the // data plane's share of it, not counting an actor certificate mint. type delays struct { run, restore, checkpoint, promoteSnapshot, terminate time.Duration } -// herder answers every AteomHerder call with success after its delay, without +// herder answers every Atelet call with success after its delay, without // running or saving any workload. Actors it "runs" do not exist. // // The one side effect it keeps is a placeholder object under each snapshot @@ -61,12 +61,12 @@ type delays struct { // snapshot when it creates a tag, golden tags included, and refuses to copy // an empty one. // -// The one call back it keeps is the actor certificate mint: before ateom +// The one call back it keeps is the actor certificate mint: before worker // starts a workload with tunneled egress it has atelet mint the actor's // certificate from ate-api-server, so in production every Run and Restore // that names an egress gateway costs ate-api-server a mint. type herder struct { - ateletpb.UnimplementedAteomHerderServer + ateletpb.UnimplementedAteletServer delays delays storage objectWriter @@ -77,7 +77,7 @@ type herder struct { } // newActorCSR returns a CSR for a fresh key. ate-api-server takes only the -// public key from it, so one serves every mint; ateom makes a key per +// public key from it, so one serves every mint; worker makes a key per // activation, but that cost is the data plane's, not ate-api-server's. func newActorCSR() ([]byte, error) { key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) @@ -92,14 +92,14 @@ func newActorCSR() ([]byte, error) { } // mintActorCertificate mints the actor's certificate when the call names an -// egress gateway, as ateom does before it starts the workload. Its time is +// egress gateway, as worker does before it starts the workload. Its time is // ate-api-server's, so it adds to the call's delay rather than running inside -// it; a failure fails the call, as tunneled egress fails closed in ateom. +// it; a failure fails the call, as tunneled egress fails closed in worker. func (h *herder) mintActorCertificate(ctx context.Context, atespace, name, uid string, gateway *ateletpb.EgressGateway) error { if h.minter == nil || gateway == nil { return nil } - if _, err := h.minter.MintAteomActorCertificate(ctx, &ateapipb.MintAteomActorCertificateRequest{ + if _, err := h.minter.MintWorkerActorCertificate(ctx, &ateapipb.MintWorkerActorCertificateRequest{ Actor: &ateapipb.ObjectRef{Atespace: atespace, Name: name}, ActorUid: uid, CertificateSigningRequest: h.csr, diff --git a/cmd/benchmarking/isolate/fake-atelet/herder_test.go b/cmd/benchmarking/isolate/fake-atelet/herder_test.go index 8e8ac9018d..0268c0f6c6 100644 --- a/cmd/benchmarking/isolate/fake-atelet/herder_test.go +++ b/cmd/benchmarking/isolate/fake-atelet/herder_test.go @@ -69,7 +69,7 @@ func uniformDelays(d time.Duration) delays { return delays{run: d, restore: d, checkpoint: d, promoteSnapshot: d, terminate: d} } -// herderCalls invokes every AteomHerder method the herder implements. The +// herderCalls invokes every Atelet method the herder implements. The // Checkpoint and PromoteSnapshot calls upload a snapshot, so each writes a // placeholder. func herderCalls(h *herder) map[string]func(context.Context) error { @@ -261,12 +261,12 @@ func TestHerderWarnsOnlyWhenWriteOutlastsANonzeroDelay(t *testing.T) { // recordingMinter records every mint, as ate-api-server's WorkerService. type recordingMinter struct { mu sync.Mutex - reqs []*ateapipb.MintAteomActorCertificateRequest + reqs []*ateapipb.MintWorkerActorCertificateRequest err error delay time.Duration } -func (m *recordingMinter) MintAteomActorCertificate(_ context.Context, in *ateapipb.MintAteomActorCertificateRequest, _ ...grpc.CallOption) (*ateapipb.MintAteomActorCertificateResponse, error) { +func (m *recordingMinter) MintWorkerActorCertificate(_ context.Context, in *ateapipb.MintWorkerActorCertificateRequest, _ ...grpc.CallOption) (*ateapipb.MintWorkerActorCertificateResponse, error) { if m.delay > 0 { time.Sleep(m.delay) } @@ -276,10 +276,10 @@ func (m *recordingMinter) MintAteomActorCertificate(_ context.Context, in *ateap if m.err != nil { return nil, m.err } - return &ateapipb.MintAteomActorCertificateResponse{ActorCertificates: [][]byte{[]byte("leaf")}}, nil + return &ateapipb.MintWorkerActorCertificateResponse{ActorCertificates: [][]byte{[]byte("leaf")}}, nil } -func (m *recordingMinter) mints() []*ateapipb.MintAteomActorCertificateRequest { +func (m *recordingMinter) mints() []*ateapipb.MintWorkerActorCertificateRequest { m.mu.Lock() defer m.mu.Unlock() return slices.Clone(m.reqs) @@ -312,7 +312,7 @@ func newMintingHerder(t *testing.T, m *recordingMinter, d delays) *herder { return &herder{delays: d, storage: &recordingStorage{}, minter: m, csr: csr} } -// ateom mints the actor's certificate before starting any workload with +// worker mints the actor's certificate before starting any workload with // tunneled egress, so each Run and Restore that names a gateway costs // ate-api-server one mint, for that actor, with a CSR it accepts. func TestHerderMintsActorCertificateOnActivation(t *testing.T) { @@ -342,7 +342,7 @@ func TestHerderMintsActorCertificateOnActivation(t *testing.T) { } } -// ateom mints only for tunneled egress, which ate-api-server asks for by +// worker mints only for tunneled egress, which ate-api-server asks for by // naming a gateway; and --mint-actor-certificate=false leaves the mint out. func TestHerderSkipsMint(t *testing.T) { m := &recordingMinter{} @@ -361,7 +361,7 @@ func TestHerderSkipsMint(t *testing.T) { } } -// A failed mint fails the activation, as tunneled egress fails closed in ateom. +// A failed mint fails the activation, as tunneled egress fails closed in worker. func TestHerderFailsWhenMintFails(t *testing.T) { m := &recordingMinter{err: errors.New("actor not found")} for name, call := range activations(newMintingHerder(t, m, delays{}), testGateway) { diff --git a/cmd/benchmarking/isolate/fake-atelet/main.go b/cmd/benchmarking/isolate/fake-atelet/main.go index 1a1cda1631..3f3e7dc381 100644 --- a/cmd/benchmarking/isolate/fake-atelet/main.go +++ b/cmd/benchmarking/isolate/fake-atelet/main.go @@ -14,10 +14,10 @@ // Command fake-atelet stands in for atelet in control-plane benchmarks. It // runs in atelet's DaemonSet, installed by ate-setup --fake-data-plane, so -// ate-api-server dials it as the node's atelet. It answers every AteomHerder +// ate-api-server dials it as the node's atelet. It answers every Atelet // call with success after a delay, without running or saving any workload, // and relays fake-workersync's capacity reports for the fake Workers on its -// node, as atelet relays ateom's. +// node, as atelet relays worker's. // // Actors it "runs" do not exist. Never run it on a cluster that serves real // actors. @@ -55,7 +55,7 @@ import ( ) var ( - port = pflag.Int("port", atelet.DefaultPort, "Port to serve AteomHerder on. ate-api-server dials atelet.DefaultPort.") + port = pflag.Int("port", atelet.DefaultPort, "Port to serve Atelet on. ate-api-server dials atelet.DefaultPort.") relayPort = pflag.Int("relay-port", fakeworker.DefaultRelayPort, "Port to serve fake-workersync's capacity reports on.") relayAllowedID = pflag.String("relay-allowed-spiffe-id", installdefaults.SPIFFEID("ate-system", "fake-workersync"), "The only SPIFFE ID allowed to send capacity reports.") healthListenAddr = pflag.String("health-listen-addr", ":9091", "Address to serve /healthz and /readyz on.") @@ -66,13 +66,13 @@ var ( ateapiCAFile = pflag.String("ateapi-ca-file", "/run/servicedns.podcert.ate.dev/trust-bundle.pem", "CA bundle used to verify ate-api-server.") ateapiServerName = pflag.String("ateapi-server-name", "api.ate-system.svc", "DNS name expected on the ate-api-server certificate.") - delay = pflag.Duration("delay", 0, "How long each AteomHerder call takes before it succeeds, not counting the actor certificate mint on Run and Restore.") + delay = pflag.Duration("delay", 0, "How long each Atelet call takes before it succeeds, not counting the actor certificate mint on Run and Restore.") delayRun = pflag.Duration("delay-run", -1, "Override --delay for Run. Negative uses --delay.") delayRestore = pflag.Duration("delay-restore", -1, "Override --delay for Restore. Negative uses --delay.") delayCheckpoint = pflag.Duration("delay-checkpoint", -1, "Override --delay for Checkpoint. Negative uses --delay.") delayPromoteSnapshot = pflag.Duration("delay-promote-snapshot", -1, "Override --delay for PromoteSnapshot. Negative uses --delay.") delayTerminate = pflag.Duration("delay-terminate", -1, "Override --delay for Terminate. Negative uses --delay.") - mintActorCertificate = pflag.Bool("mint-actor-certificate", true, "On each Run and Restore that names an egress gateway, mint the actor's certificate from ate-api-server before the delay, as ateom does before starting the workload.") + mintActorCertificate = pflag.Bool("mint-actor-certificate", true, "On each Run and Restore that names an egress gateway, mint the actor's certificate from ate-api-server before the delay, as worker does before starting the workload.") showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") @@ -136,7 +136,7 @@ func main() { h.minter = workers } srv := grpc.NewServer(grpc.Creds(credentials.NewTLS(herderTLS))) - ateletpb.RegisterAteomHerderServer(srv, h) + ateletpb.RegisterAteletServer(srv, h) relayTLS, err := relayTLSConfig(*grpcServerCredBundle, *clientCACerts, *relayAllowedID) if err != nil { @@ -158,7 +158,7 @@ func main() { go func() { if err := srv.Serve(lis); err != nil { - serverboot.Fatal(ctx, "Failed to serve AteomHerder", err) + serverboot.Fatal(ctx, "Failed to serve Atelet", err) } }() go func() { diff --git a/cmd/benchmarking/isolate/fake-atelet/relay.go b/cmd/benchmarking/isolate/fake-atelet/relay.go index 96ca89f0a7..c25d80a678 100644 --- a/cmd/benchmarking/isolate/fake-atelet/relay.go +++ b/cmd/benchmarking/isolate/fake-atelet/relay.go @@ -30,8 +30,8 @@ type capacityClient interface { } // relay forwards fake-workersync's capacity reports to ate-api-server under -// this pod's atelet identity. It stands where atelet's AteomSupport stands in -// production: ateom knows its capacity and atelet relays it, because +// this pod's atelet identity. It stands where atelet's WorkerSupport stands in +// production: worker knows its capacity and atelet relays it, because // ate-api-server accepts a report only from the atelet on the Worker's node. // The relay decides nothing; fake-workersync names the Worker and its // capacity, and ate-api-server checks the Worker is on this node. diff --git a/cmd/benchmarking/isolate/fake-atelet/relay_test.go b/cmd/benchmarking/isolate/fake-atelet/relay_test.go index 2d5afcace6..88f70d7304 100644 --- a/cmd/benchmarking/isolate/fake-atelet/relay_test.go +++ b/cmd/benchmarking/isolate/fake-atelet/relay_test.go @@ -87,9 +87,9 @@ func TestRelayPassesThroughAteAPIServersStatus(t *testing.T) { // nothing but capacity reports: a minted actor certificate, say, would hand // fake-workersync an identity it does not hold. func TestRelayServesOnlyCapacityReports(t *testing.T) { - _, err := (&relay{client: &recordingWorkerService{}}).MintAteomActorCertificate(context.Background(), &ateapipb.MintAteomActorCertificateRequest{}) + _, err := (&relay{client: &recordingWorkerService{}}).MintWorkerActorCertificate(context.Background(), &ateapipb.MintWorkerActorCertificateRequest{}) if got := status.Code(err); got != codes.Unimplemented { - t.Errorf("MintAteomActorCertificate code = %v, want Unimplemented", got) + t.Errorf("MintWorkerActorCertificate code = %v, want Unimplemented", got) } } diff --git a/cmd/benchmarking/isolate/fake-workersync/controller.go b/cmd/benchmarking/isolate/fake-workersync/controller.go index fe40a5211a..320ecfc7cf 100644 --- a/cmd/benchmarking/isolate/fake-workersync/controller.go +++ b/cmd/benchmarking/isolate/fake-workersync/controller.go @@ -103,7 +103,7 @@ type fakeWorker struct { // tmpl is the pool template the Worker was made from; nil for a Worker // adopted after a restart until reconcile matches it to its pool. tmpl *template - // capacity is fixed when the Worker is made, as ateom reads its limits + // capacity is fixed when the Worker is made, as worker reads its limits // once at startup; reported is whether ate-api-server has accepted it. capacity *ateapipb.WorkerResources reported bool @@ -351,7 +351,7 @@ func (c *controller) reconcile(ctx context.Context) error { // ensure registers a Worker for the desired pod if w, its current Worker, is // nil, brings the Worker's labels in line with the pool's, then has its // capacity reported until ate-api-server accepts it. The capacity is fixed -// when the Worker is made, as ateom reads its limits once at startup, and is +// when the Worker is made, as worker reads its limits once at startup, and is // reported once. The Worker is tracked before CreateWorker is called and // marked created after, so a failed create is retried by the next pass under // the same name, and one the server committed anyway is never lost: retire @@ -493,9 +493,9 @@ func (c *controller) updateLabels(ctx context.Context, name string, w *fakeWorke // retire removes a Worker no pool wants, the way a real one goes when its pod // does: drained first so nothing new lands on it, deleted once no Actor is -// assigned to it. In production ateom stops the pod's Actors within its +// assigned to it. In production worker stops the pod's Actors within its // termination grace period, and the Worker is deleted when the pod is gone, -// which releases any Actor still bound. No ateom stops them here, so after +// which releases any Actor still bound. No worker stops them here, so after // drainGrace the Worker is deleted with its Actors still assigned, and // DeleteWorker releases them the same way. func (c *controller) retire(ctx context.Context, name string) error { @@ -627,7 +627,7 @@ func templateOf(wp *atev1alpha1.WorkerPool) (template, error) { // capacity is what a worker made from t reports on a node with allocatable: // cpu and memory from its limits, or, with no limit set, the node's -// allocatable, which is what the downward API projects into ateom for an +// allocatable, which is what the downward API projects into worker for an // unset limit. func (t template) capacity(allocatable corev1.ResourceList) *ateapipb.WorkerResources { cpu, memory := t.cpuMilli, t.memoryBytes @@ -664,7 +664,7 @@ func fits(w *fakeWorker, wp *atev1alpha1.WorkerPool, allocatable map[string]core return proto.Equal(w.capacity, t.capacity(alloc)) } -// maxActors is the pool's actor capacity per Worker: ateom's default, unless +// maxActors is the pool's actor capacity per Worker: worker's default, unless // the pool's annotation overrides it. func maxActors(wp *atev1alpha1.WorkerPool) (int, error) { v, ok := wp.Annotations[fakeworker.MaxActorsAnnotation] diff --git a/cmd/benchmarking/isolate/fake-workersync/controller_test.go b/cmd/benchmarking/isolate/fake-workersync/controller_test.go index 276cb90ce4..43c4a24024 100644 --- a/cmd/benchmarking/isolate/fake-workersync/controller_test.go +++ b/cmd/benchmarking/isolate/fake-workersync/controller_test.go @@ -638,7 +638,7 @@ func TestReconcileIsIdempotent(t *testing.T) { } // A limits edit replaces the pool's Workers, as it rolls a real pool's pods: -// ateom reads its limits once at startup, so a Worker's capacity never +// worker reads its limits once at startup, so a Worker's capacity never // changes. The old Worker is not reported again. func TestPoolLimitsChangeReplacesWorkers(t *testing.T) { limits := corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("2"), corev1.ResourceMemory: resource.MustParse("4Gi")} @@ -678,7 +678,7 @@ func TestPoolLimitsChangeReplacesWorkers(t *testing.T) { } } -// ateom's actor count is a flag, so changing it rolls a real pool's pods too. +// worker's actor count is a flag, so changing it rolls a real pool's pods too. func TestMaxActorsAnnotationChangeReplacesWorkers(t *testing.T) { c, ctl, rel, cl := newTestController(pool("bench", 1, nil)) if err := reconcile(t, c); err != nil { @@ -704,7 +704,7 @@ func TestMaxActorsAnnotationChangeReplacesWorkers(t *testing.T) { } } -// A node's allocatable reaches ateom once, at startup, so a later change +// A node's allocatable reaches worker once, at startup, so a later change // leaves a pool with no limits as it is: no new Worker and no report. func TestAllocatableChangeKeepsWorkerCapacity(t *testing.T) { c, ctl, rel, cl := newTestController(pool("bench", 2, nil)) @@ -758,7 +758,7 @@ func TestScaleDownDrainsThenDeletes(t *testing.T) { } } -// Nothing stops a fake Worker's Actors the way ateom stops a terminating +// Nothing stops a fake Worker's Actors the way worker stops a terminating // pod's, so a Worker still held after the drain grace is deleted anyway, // which releases its Actors, rather than left draining for good. func TestDrainGraceDeletesAWorkerStillHeld(t *testing.T) { @@ -1168,7 +1168,7 @@ func TestAdoptReplacesWorkersOfAPoolEditedWhileDown(t *testing.T) { } // An adopted Worker whose capacity ate-api-server never accepted is reported -// after the restart, as ateom keeps retrying its one report. +// after the restart, as worker keeps retrying its one report. func TestAdoptReportsAWorkerNeverAccepted(t *testing.T) { c, ctl, rel, cl := newTestController(pool("bench", 1, nil)) const first = "worker-1" // the first name newTestController's controller mints @@ -1208,7 +1208,7 @@ func TestDeleteAll(t *testing.T) { } } -func TestCapacityFormatsLikeAteom(t *testing.T) { +func TestCapacityFormatsLikeWorker(t *testing.T) { capacity := func(wp *atev1alpha1.WorkerPool) *ateapipb.WorkerResources { t.Helper() tmpl, err := templateOf(wp) @@ -1218,7 +1218,7 @@ func TestCapacityFormatsLikeAteom(t *testing.T) { return tmpl.capacity(nil) } got := capacity(pool("p", 1, corev1.ResourceList{corev1.ResourceCPU: resource.MustParse("0.5"), corev1.ResourceMemory: resource.MustParse("1G")})) - // ateom spells cpu as decimal milli-cores and memory as a binary-SI byte + // worker spells cpu as decimal milli-cores and memory as a binary-SI byte // count; 1G is not a power of two, so 1e9. if want := wantCapacity(1000, "500m", "1e9"); !proto.Equal(got, want) { t.Errorf("capacity = %v, want %v", got, want) diff --git a/cmd/benchmarking/isolate/fake-workersync/main.go b/cmd/benchmarking/isolate/fake-workersync/main.go index 915b0e3503..d5596f7a07 100644 --- a/cmd/benchmarking/isolate/fake-workersync/main.go +++ b/cmd/benchmarking/isolate/fake-workersync/main.go @@ -16,7 +16,7 @@ // controller and worker syncer in control-plane benchmarks. It honors every // WorkerPool's spec.replicas with fake Workers that no pod backs, placed // round robin on the nodes that run a fake-atelet pod, and has each Worker's -// capacity reported through the fake-atelet on its node, as ateom reports +// capacity reported through the fake-atelet on its node, as worker reports // through atelet. It writes each pool's status, so the scale subresource and // `kubectl get workerpool` work. It leaves its Workers registered when it // shuts down and adopts them when it starts, so a restart keeps every Actor @@ -76,7 +76,7 @@ var ( cleanup = pflag.Bool("cleanup", false, "Delete this run's registered fake Workers, then exit. Stop the serving fake-workersync first, as by scaling its Deployment to zero: it would go on counting the deleted Workers until it restarts.") nodeGrace = pflag.Duration("node-grace", 5*time.Minute, "How long a node keeps its fake Workers with no fake-atelet pod, as while its fake-atelet is replaced.") healthAddr = pflag.String("health-listen-addr", ":9091", "Address to serve /healthz and /readyz on.") - drainGrace = pflag.Duration("drain-grace", time.Minute, "How long a draining fake Worker waits for its Actors to leave before it is deleted, releasing them. Stands in for the time ateom takes to stop a terminating pod's Actors.") + drainGrace = pflag.Duration("drain-grace", time.Minute, "How long a draining fake Worker waits for its Actors to leave before it is deleted, releasing them. Stands in for the time worker takes to stop a terminating pod's Actors.") showVersion = pflag.Bool("version", false, "Print version and exit.") logLevelFlag = pflag.String("log-level", "info", "Minimum log level: debug, info, warn, or error.") diff --git a/cmd/benchmarking/isolate/fake-workersync/relayclient.go b/cmd/benchmarking/isolate/fake-workersync/relayclient.go index b53703bfa1..3b27c64d27 100644 --- a/cmd/benchmarking/isolate/fake-workersync/relayclient.go +++ b/cmd/benchmarking/isolate/fake-workersync/relayclient.go @@ -34,7 +34,7 @@ import ( const reportTimeout = 10 * time.Second // grpcRelay sends capacity reports to the fake-atelet relay on a Worker's -// node, as ateom sends them to the atelet on its own node. A rejected report +// node, as worker sends them to the atelet on its own node. A rejected report // carries ate-api-server's status unchanged. type grpcRelay struct { creds credentials.TransportCredentials diff --git a/cmd/benchmarking/isolate/internal/fakeworker/fakeworker.go b/cmd/benchmarking/isolate/internal/fakeworker/fakeworker.go index c1244940de..bd16e6b8cd 100644 --- a/cmd/benchmarking/isolate/internal/fakeworker/fakeworker.go +++ b/cmd/benchmarking/isolate/internal/fakeworker/fakeworker.go @@ -33,7 +33,7 @@ import ( const MaxRunLength = 8 // DefaultMaxActors is how many actors a fake Worker holds, the default of -// ateom's --max-actors, which no WorkerPool field sets for a real worker. +// worker's --max-actors, which no WorkerPool field sets for a real worker. const DefaultMaxActors = 1000 // MaxActorsAnnotation, on a WorkerPool, overrides DefaultMaxActors for that diff --git a/cmd/benchmarking/isolate/internal/fakeworker/fakeworker_test.go b/cmd/benchmarking/isolate/internal/fakeworker/fakeworker_test.go index 5df7beff47..ce98024116 100644 --- a/cmd/benchmarking/isolate/internal/fakeworker/fakeworker_test.go +++ b/cmd/benchmarking/isolate/internal/fakeworker/fakeworker_test.go @@ -27,7 +27,7 @@ import ( // Names are pinned: a change renames every fake Worker's pod, which strands // the Workers a previous fake-workersync registered. func TestNamePinned(t *testing.T) { - if got, want := Name("r1", "benchmark-workloads", "benchmark-ateom", 0), "fake-r1-c52b9bd3-0"; got != want { + if got, want := Name("r1", "benchmark-workloads", "benchmark-worker", 0), "fake-r1-4ce21861-0"; got != want { t.Errorf("Name = %q, want %q", got, want) } } @@ -63,7 +63,7 @@ func TestIndexRoundTrips(t *testing.T) { Name("r2", "ns", "pool", 3), // another run Name("r1", "ns", "pool", 3) + "x", // not a number strings.TrimSuffix(Name("r1", "ns", "pool", 0), "0") + "01", // not canonical - "benchmark-ateom-5d8f-xyz", + "benchmark-worker-5d8f-xyz", } { if _, ok := Index("r1", "ns", "pool", name); ok { t.Errorf("Index(%q) matched the pool", name) diff --git a/cmd/credential-provider/kubernetes-secrets/kubeprovider_test.go b/cmd/credential-provider/kubernetes-secrets/kubeprovider_test.go index 9def93ed76..1f2758aa81 100644 --- a/cmd/credential-provider/kubernetes-secrets/kubeprovider_test.go +++ b/cmd/credential-provider/kubernetes-secrets/kubeprovider_test.go @@ -172,8 +172,8 @@ func TestFetchSecretAuthorization(t *testing.T) { wantCode: codes.PermissionDenied, }, { - name: "ateom identity", - actorSpiffeID: "spiffe://substrate-actor.local/ateom-for-actor/team-a/my-actor", + name: "worker identity", + actorSpiffeID: "spiffe://substrate-actor.local/worker-for-actor/team-a/my-actor", uri: "ate-secret://k8s.io/default/ns1/example-api/token", wantCode: codes.PermissionDenied, }, diff --git a/cmd/credential-provider/kubernetes-secrets/main.go b/cmd/credential-provider/kubernetes-secrets/main.go index 53c7416d88..6ce789e698 100644 --- a/cmd/credential-provider/kubernetes-secrets/main.go +++ b/cmd/credential-provider/kubernetes-secrets/main.go @@ -43,6 +43,7 @@ import ( "github.com/agent-substrate/substrate/internal/credbundle" "github.com/agent-substrate/substrate/internal/installdefaults" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/internal/version" "github.com/agent-substrate/substrate/pkg/proto/credproviderpb" @@ -66,6 +67,7 @@ var ( drainGrace = pflag.Duration("drain-grace", 5*time.Second, "how long to wait for in-flight RPCs on shutdown before a hard stop") kubeAPIQPS = pflag.Float32("kube-api-qps", 50, "Sustained queries per second allowed against the Kubernetes API.") kubeAPIBurst = pflag.Int("kube-api-burst", 100, "Burst queries allowed against the Kubernetes API.") + previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.") ) func main() { @@ -76,6 +78,9 @@ func main() { if err := serverboot.SetLogLevel(*logLevel); err != nil { serverboot.Fatal(ctx, "invalid --log-level", err) } + if err := preview.Init(*previewFlags...); err != nil { + serverboot.Fatal(ctx, "invalid --preview", err) + } slog.InfoContext(ctx, "starting credprovider", slog.String("version", version.String())) diff --git a/cmd/dataplane/envoy/Dockerfile b/cmd/dataplane/envoy/Dockerfile index f133873e9e..b179a711d4 100644 --- a/cmd/dataplane/envoy/Dockerfile +++ b/cmd/dataplane/envoy/Dockerfile @@ -12,10 +12,10 @@ # See the License for the specific language governing permissions and # limitations under the License. -# Stage 1: Build the release version of the Rust dynamic module +# Stage 1: Build the release version of the Rust dynamic modules FROM rust:bookworm@sha256:93ce27a88655056a51dbdd8f5f2d7ddc071c7b0070fb288a37b5a285fc83971e AS builder -WORKDIR /usr/src/dynamic-modules/egress-policy +WORKDIR /usr/src/dynamic-modules # Install build dependencies required by bindgen for generating Envoy ABI bindings RUN apt-get update && apt-get install -y --no-install-recommends \ @@ -23,13 +23,13 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ libclang-dev \ && rm -rf /var/lib/apt/lists/* -# Copy rust module source code -COPY dynamic-modules/egress-policy/ . +# Copy rust workspace source code +COPY dynamic-modules/ . -# Build release version of the dynamic module shared library +# Build release version of the dynamic module shared libraries RUN cargo build --release --locked -# Stage 2: Create the runtime image containing Envoy and the built dynamic module shared library +# Stage 2: Create the runtime image containing Envoy and the built dynamic module shared libraries # When changing Envoy version, the version of SDK in the Cargo.toml files SHOULD match. # Pin the multi-platform image index, not one platform's manifest, so that the # --platform the installer passes (the host architecture on kind, linux/amd64 @@ -37,7 +37,7 @@ RUN cargo build --release --locked FROM envoyproxy/envoy:v1.39.4@sha256:74bd4eb5629708d3fadca1a5e6b85d208f338d7bb54e06b69565905c5294cc01 # Copy the built dynamic module shared library from the builder stage -COPY --from=builder /usr/src/dynamic-modules/egress-policy/target/release/libenvoy_substrate_egress_policy.so /usr/local/lib/libenvoy_substrate_egress_policy.so +COPY --from=builder /usr/src/dynamic-modules/target/release/libenvoy_substrate_egress_policy.so /usr/local/lib/libenvoy_substrate_egress_policy.so # Set default search path for Envoy dynamic modules ENV ENVOY_DYNAMIC_MODULES_SEARCH_PATH=/usr/local/lib diff --git a/cmd/dataplane/envoy/README.md b/cmd/dataplane/envoy/README.md index a3ffb8602e..e86f6f2db7 100644 --- a/cmd/dataplane/envoy/README.md +++ b/cmd/dataplane/envoy/README.md @@ -8,6 +8,7 @@ This directory contains the build definition and custom extensions for the Envoy cmd/dataplane/envoy/ ├── Dockerfile # Multi-stage build for the Envoy dataplane image └── dynamic-modules/ + ├── Cargo.toml # Rust Cargo workspace manifest └── egress-policy/ # Rust Envoy Dynamic Module for egress policy enforcement ``` diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/.gitignore b/cmd/dataplane/envoy/dynamic-modules/.gitignore similarity index 100% rename from cmd/dataplane/envoy/dynamic-modules/egress-policy/.gitignore rename to cmd/dataplane/envoy/dynamic-modules/.gitignore diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/Cargo.lock b/cmd/dataplane/envoy/dynamic-modules/Cargo.lock similarity index 93% rename from cmd/dataplane/envoy/dynamic-modules/egress-policy/Cargo.lock rename to cmd/dataplane/envoy/dynamic-modules/Cargo.lock index d5aa3de254..088375f4d7 100644 --- a/cmd/dataplane/envoy/dynamic-modules/egress-policy/Cargo.lock +++ b/cmd/dataplane/envoy/dynamic-modules/Cargo.lock @@ -250,6 +250,15 @@ dependencies = [ "proc-macro2", ] +[[package]] +name = "rapidhash" +version = "4.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5da7e78a036ce858e8d55b7e7dc8ba3a88b78350fd2155d3591bbd966b58589e" +dependencies = [ + "rustversion", +] + [[package]] name = "regex" version = "1.13.1" @@ -285,6 +294,12 @@ version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" +[[package]] +name = "rustversion" +version = "1.0.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f" + [[package]] name = "serde" version = "1.0.229" @@ -339,8 +354,10 @@ name = "substrate-envoy-egress-policy" version = "0.1.0" dependencies = [ "envoy-proxy-dynamic-modules-rust-sdk", + "rapidhash", "serde", "serde_json", + "thread_local", ] [[package]] @@ -371,6 +388,15 @@ version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f50febec83f5ee1df3015341d8bd429f2d1cc62bcba7ea2076759d315084683" +[[package]] +name = "thread_local" +version = "1.1.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070" +dependencies = [ + "cfg-if", +] + [[package]] name = "unicode-ident" version = "1.0.24" diff --git a/cmd/dataplane/envoy/dynamic-modules/Cargo.toml b/cmd/dataplane/envoy/dynamic-modules/Cargo.toml new file mode 100644 index 0000000000..d8e278e895 --- /dev/null +++ b/cmd/dataplane/envoy/dynamic-modules/Cargo.toml @@ -0,0 +1,34 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +[workspace] +resolver = "3" +members = [ + "egress-policy", +] + +[workspace.package] +version = "0.1.0" +edition = "2024" + +[workspace.dependencies] +# Pin to v1.39.1 commit SHA +# When changing the SDK version, the version of Envoy in the cmd/dataplane/envoy/Dockerfile SHOULD match. +# SDK version SHOULD NOT be newer than Envoy's in the dataplane image, as extensions will not load if they use +# APIs not supported by the Envoy binary in the image. +envoy-proxy-dynamic-modules-rust-sdk = { git = "https://github.com/envoyproxy/envoy", rev = "b579d07d3ad7ee11d32b105e91a5a39ad24718d7" } +rapidhash = "4.5.1" +serde = { version = "1.0", features = ["derive"] } +serde_json = "1.0" +thread_local = "1.1" diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/Cargo.toml b/cmd/dataplane/envoy/dynamic-modules/egress-policy/Cargo.toml index a4e881a7ec..6c699cbee0 100644 --- a/cmd/dataplane/envoy/dynamic-modules/egress-policy/Cargo.toml +++ b/cmd/dataplane/envoy/dynamic-modules/egress-policy/Cargo.toml @@ -14,18 +14,16 @@ [package] name = "substrate-envoy-egress-policy" -version = "0.1.0" -edition = "2024" +version.workspace = true +edition.workspace = true [lib] name = "envoy_substrate_egress_policy" crate-type = ["cdylib"] [dependencies] -# Pin to v1.39.1 commit SHA -# When changing the SDK version, the version of Envoy in the cmd/dataplane/envoy/Dockerfile SHOULD match. -# SDK version SHOULD NOT be newer than Envoy's in the dataplane image, as extensions will not load if they use -# APIs not supported by the Envoy binary in the image. -envoy-proxy-dynamic-modules-rust-sdk = { git = "https://github.com/envoyproxy/envoy", rev = "b579d07d3ad7ee11d32b105e91a5a39ad24718d7" } -serde = { version = "1.0", features = ["derive"] } -serde_json = "1.0" +envoy-proxy-dynamic-modules-rust-sdk.workspace = true +rapidhash.workspace = true +serde.workspace = true +serde_json.workspace = true +thread_local.workspace = true diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/README.md b/cmd/dataplane/envoy/dynamic-modules/egress-policy/README.md index 52bc99227f..91daa78aa9 100644 --- a/cmd/dataplane/envoy/dynamic-modules/egress-policy/README.md +++ b/cmd/dataplane/envoy/dynamic-modules/egress-policy/README.md @@ -1,89 +1,61 @@ -# Envoy Substrate Egress Policy Implementation - Rust Dynamic Module +# Envoy Substrate Egress Policy Implementation -An Envoy dynamic module, written in Rust, that runs as a listener filter on -the egress gateway's inner listener and names the filter chain each -tunneled connection belongs on. +This Envoy dynamic module implements two extensions that work together to minimize +memory usage by Substrate egress policies. -## What it decides +To minimize memory usage egress policies are hashed using [Rapidhash](https://docs.rs/rapidhash/latest/rapidhash/) +which produces 64 bit hashes with low probability of collision. Policies are then stored in the worker local +store using computed hash and the hash is exchanged between outer CONNECT request and inner +listener that terminates TCP connection encapsulated in the CONNECT request. -The gateway's ext_proc sidecar answers every allowed CONNECT with the rules -that decide the tunnel's TLS: the policy's `https` rules for the port the -actor dialed, most specific first (a pattern without a wildcard before one -with, then a rule naming its ports before one naming all of them). The outer -chain copies that answer into the `dev.ate.policy.egress` filter state as -JSON, shared with the inner listener: +## Details -```json -{"rules": [{"pattern": "api.example.com", "mode": "mitm"}, - {"pattern": "*.example.com", "mode": "mitm"}]} -``` +In the outer CONNECT filter chain the policy store HTTP filter is instantiated after +the ext_proc filter. After ext_proc successfully looks up egress policy for the given actor and +destinaion port, the subsequent policy store filter: -After `tls_inspector` and `http_inspector` have looked at the first bytes, -this filter writes one of these verdicts to the `dev.ate.egress.filter_chain` -filter state, and the listener's `filter_chain_matcher` selects the chain by -it: +1. Creates EngressPolicy object by parsing JSON from metadata sent by the ext_proc server and computes its hash. +1. Checks if the policy with this hash already exists in the worker local policy store. +1. If it does it adds a reference to the existing policy. +1. Otherwise it stores the new policy in the store. +1. The policy hash is written to the filter state shared with the upstream. -| First bytes | Verdict | Chain | +In the inner listener filter chain, the policy enforcement listener filter: + +1. reads policy hash from the filter state, shared by the outer CONNECT filter chain. +1. Looks up the policy object by hash in the worker local policy store. +1. Evaluates and enforces egress policy. + +With this approch, egress policy objects are shared by many CONNECT requests. Per request state is +a 64 bit hash. For example if all actors have the same policy, Envoy will keep only one policy object in memory +and all requests will share the same object. + +## Policy Lifetime + +Ownership of EgressPolicy objects is shared by policy store HTTP filters and the worker local policy store. As such +policy object is guaranteed to outlive all CONNECT requests that share the same policy. Lifetime of the CONNECT request +is at least as long as the lifetime of the inner request, so it is guaratedd that inner request will always +find a policy with hash it received from the outer CONNECT request. + +To avoid leaking policy object in the local policy store, policy store HTTP filter, during its cleanup, checks the +reference count value of the EgressPolicy. If this cound vaoue is 2, then only this filter and policy store have outstanding +references, and it this point it is safe to remove policy object from the store. This guaratees that policy is detroyed, +when no more CONNECT requests are referencing it. + +## Policy Hashing + +[Rapidhash](https://docs.rs/rapidhash/latest/rapidhash/) is considered one of the best noon-cryptographic hashing funections +wth near [ideal collision probability](https://chromium.googlesource.com/chromium/src/+/HEAD/third_party/rapidhash/README.md). +Other options are xxhash which can produce 128 bit hashes or cryptographic hash such as SHA-256. 64 bit Rapidhash offers the +best balance between quality and performance, with approximately 1 in 10^19 probability of collision. + +## Envoy stats + +The HTTP filter (`envoy_substrate_egress_policy_store`) defines the following +metrics: + +| Metric | Type | Description | |---|---|---| -| A ClientHello whose SNI matches an https rule (first match wins; `*` matches every name, `*.suffix` exactly one label, anything else the whole name, ASCII case folded) | `mitm` | `egress_tls_mitm`: terminated with a minted leaf, decided per request | -| A ClientHello whose first matching rule is tls_passthrough | `passthrough` | `egress_passthrough`: relayed unread to the resolved SNI on the dialed port | -| Any other ClientHello: no SNI, no match, no rules, unparseable rules | `denied` | none: the connection is closed | -| Not TLS | `cleartext` | `egress_cleartext`: decided per request | -| A transport protocol other than `tls` or `raw_buffer` | `denied` | none | - -A `tls_passthrough` rule yields the `passthrough` verdict and the -`egress_passthrough` chain, which resolves the SNI itself and relays the bytes -to it unread, on the port the actor dialed. The address the actor dialed is -never used. A connection that sends nothing before the listener filter timeout -never reaches this filter, sets no verdict, and is closed. - -The Go side of the contract is `cmd/atenet/internal/router/extproc` -(`EgressPolicyMetadataNamespace`, `EgressFilterChainFilterStateKey`) and -`internal/egresspolicy` (`SNIRules`, whose pattern grammar this filter -mirrors). The manifest tests in `cmd/atenet/internal/router` hold the -listener configuration to it. - -## Building - -Prerequisites: Rust toolchain (Cargo, rustc 1.75+), plus `clang` and -`libclang-dev` for the SDK's bindgen step. - -```bash -cargo build --release -``` - -The compiled shared object will be located at -`target/release/libenvoy_substrate_egress_policy.so`. `cmd/dataplane/envoy/Dockerfile` -builds it the same way and packages it into the Envoy image. - -## Testing - -```bash -cargo test -``` - -## Envoy configuration - -Add the filter after the inspectors in the listener's `listener_filters`: - -```yaml -listener_filters: -- name: envoy.filters.listener.tls_inspector - typed_config: - "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector -- name: envoy.filters.listener.http_inspector - typed_config: - "@type": type.googleapis.com/envoy.extensions.filters.listener.http_inspector.v3.HttpInspector -- name: envoy.filters.listener.dynamic_modules - typed_config: - "@type": type.googleapis.com/envoy.extensions.filters.listener.dynamic_modules.v3.DynamicModuleListenerFilter - dynamic_module_config: - name: envoy_substrate_egress_policy - filter_name: envoy_substrate_egress_policy -``` - -Set the environment variable `ENVOY_DYNAMIC_MODULES_SEARCH_PATH` to the -directory containing `libenvoy_substrate_egress_policy.so` (e.g. -`export ENVOY_DYNAMIC_MODULES_SEARCH_PATH=/path/to/target/release`). -`manifests/ate-install/atenet-egress.yaml` is the complete -configuration, matcher and chains included. +| `ate_egress.policy_store_size` | Gauge | Current number of unique `EgressPolicy` entries stored in the worker thread's local store (summed across workers by Envoy). | +| `ate_egress.parse_errors` | Counter | Number of times `EgressPolicy::new_from_metadata` returned `None` due to missing or malformed policy metadata. | +| `ate_egress.policy_shared` | Counter | Number of times a `CONNECT` request reused an `EgressPolicy` already present in the worker thread's local store. | diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/egress_policy.rs b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/egress_policy.rs new file mode 100644 index 0000000000..f9613f3948 --- /dev/null +++ b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/egress_policy.rs @@ -0,0 +1,183 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Shared egress policy types for Substrate Envoy dynamic modules. + +use std::hash::{Hash, Hasher}; +use std::rc::Rc; + +use envoy_proxy_dynamic_modules_rust_sdk::{ + abi::envoy_dynamic_module_type_metadata_source, EnvoyHttpFilter, +}; +// Use quiality flavor of Rapidhash to trade collision resistance over performance. +use rapidhash::quality::RapidHasher; +use serde::Deserialize; + +/// Dynamic metadata namespace holding the egress policy JSON. +pub const ATE_POLICY_EGRESS_NAMESPACE: &str = "dev.ate.policy.egress"; + +/// Dynamic metadata key under [`ATE_POLICY_EGRESS_NAMESPACE`] holding the serialized policy JSON. +pub const ATE_POLICY_JSON_KEY: &str = "policy_json"; + +/// The SNI rules for a connection. +#[derive(Debug, Default, Deserialize, PartialEq, Eq, Hash)] +pub struct EgressPolicy { + /// Most specific first; the first match wins. + pub rules: Vec, + #[serde(skip)] + pub hash: u64, +} + +#[derive(Debug, Deserialize, PartialEq, Eq, Hash)] +pub struct SniRule { + pub pattern: String, + // TODO(yanavlasov): optimize this to use enum to reduce memory usage. + // Mode determines which filter chain (MITM or passthrough) is selected if the patter matches. + pub mode: String, +} + +impl EgressPolicy { + /// Reads the egress policy JSON from dynamic metadata (`dev.ate.policy.egress:policy_json`) + /// and parses it into an [`Rc`]. + pub fn new_from_metadata(envoy_filter: &mut EHF) -> Option> { + let metadata = envoy_filter.get_metadata_string( + envoy_dynamic_module_type_metadata_source::Dynamic, + ATE_POLICY_EGRESS_NAMESPACE, + ATE_POLICY_JSON_KEY, + )?; + let mut policy = serde_json::from_slice::(metadata.as_slice()).ok()?; + // Once the policy was parsed overwrite JSON string in the metadata with empty value so it does + // not consume memory. + envoy_filter.set_dynamic_metadata_string(ATE_POLICY_EGRESS_NAMESPACE, ATE_POLICY_JSON_KEY, ""); + let mut hasher = RapidHasher::default(); + Hash::hash(&policy, &mut hasher); + policy.hash = hasher.finish(); + Some(Rc::new(policy)) + } +} + +// ========================================== +// Tests +// ========================================== + +#[cfg(test)] +mod tests { + use super::*; + use envoy_proxy_dynamic_modules_rust_sdk::{EnvoyBuffer, MockEnvoyHttpFilter}; + + #[test] + fn test_policy_json_shape() { + let parsed: EgressPolicy = serde_json::from_str( + r#"{"rules":[{"pattern":"api.example.com","mode":"mitm"},{"pattern":"*","mode":"mitm"}]}"#, + ) + .unwrap(); + assert_eq!( + parsed, + EgressPolicy { + rules: vec![ + SniRule { + pattern: "api.example.com".to_string(), + mode: "mitm".to_string(), + }, + SniRule { + pattern: "*".to_string(), + mode: "mitm".to_string(), + }, + ], + hash: 0, + } + ); + assert!( + serde_json::from_str::(r#"{"allowed_snis":["api.example.com"]}"#).is_err() + ); + } + + #[test] + fn test_new_from_metadata() { + let mut mock_filter = MockEnvoyHttpFilter::default(); + mock_filter + .expect_get_metadata_string() + .withf(|source, namespace, key| { + *source == envoy_dynamic_module_type_metadata_source::Dynamic + && namespace == ATE_POLICY_EGRESS_NAMESPACE + && key == ATE_POLICY_JSON_KEY + }) + .returning(|_, _, _| { + Some(EnvoyBuffer::new( + br#"{"rules":[{"pattern":"api.example.com","mode":"mitm"}]}"#, + )) + }) + .times(2); + mock_filter + .expect_set_dynamic_metadata_string() + .withf(|namespace, key, value| { + namespace == ATE_POLICY_EGRESS_NAMESPACE && key == ATE_POLICY_JSON_KEY && value.is_empty() + }) + .return_const(()) + .times(2); + + let p1 = EgressPolicy::new_from_metadata(&mut mock_filter).unwrap(); + let p2 = EgressPolicy::new_from_metadata(&mut mock_filter).unwrap(); + assert_eq!( + p1.rules, + vec![SniRule { + pattern: "api.example.com".to_string(), + mode: "mitm".to_string(), + }] + ); + assert_ne!(p1.hash, 0); + assert_eq!(p1.hash, p2.hash); + assert_eq!(p1, p2); + + mock_filter + .expect_get_metadata_string() + .withf(|source, namespace, key| { + *source == envoy_dynamic_module_type_metadata_source::Dynamic + && namespace == ATE_POLICY_EGRESS_NAMESPACE + && key == ATE_POLICY_JSON_KEY + }) + .returning(|_, _, _| { + Some(EnvoyBuffer::new( + br#"{"rules":[{"pattern":"api.example.com","mode":"passthrough"}]}"#, + )) + }) + .once(); + mock_filter + .expect_set_dynamic_metadata_string() + .withf(|namespace, key, value| { + namespace == ATE_POLICY_EGRESS_NAMESPACE && key == ATE_POLICY_JSON_KEY && value.is_empty() + }) + .return_const(()) + .once(); + + let p3 = EgressPolicy::new_from_metadata(&mut mock_filter).unwrap(); + assert_ne!(p1.hash, p3.hash); + } + + #[test] + fn test_new_from_metadata_missing_or_invalid() { + let mut mock_filter = MockEnvoyHttpFilter::default(); + mock_filter + .expect_get_metadata_string() + .returning(|_, _, _| None) + .once(); + assert_eq!(EgressPolicy::new_from_metadata(&mut mock_filter), None); + + mock_filter + .expect_get_metadata_string() + .returning(|_, _, _| Some(EnvoyBuffer::new(b"not-json"))) + .once(); + assert_eq!(EgressPolicy::new_from_metadata(&mut mock_filter), None); + } +} diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/egress_policy_store.rs b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/egress_policy_store.rs new file mode 100644 index 0000000000..f9bdfe33b7 --- /dev/null +++ b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/egress_policy_store.rs @@ -0,0 +1,121 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Global store for egress policies shared across filters in the dynamic module. + +use std::cell::{RefCell, RefMut}; +use std::collections::HashMap; +use std::rc::Rc; + +use thread_local::ThreadLocal; + +use crate::egress_policy::EgressPolicy; + +/// Per-thread store for egress policies. +#[derive(Debug, Default)] +pub struct EgressPolicyStore { + // The 64 bit key is computed by Rapidhash and is shared via filter state + // between the HTTP filter that populates policy store and inner listener + // filter that looks up the policy by this key and enforces it. + pub policies: HashMap>, +} + +// This is needed to mollify Rust's borrows checker, as otherwise it will not +// allow retruning a mutable reference of EgressPolicyStore from the ThreadLocal store. +// It is safe to mollify borrows checker, because EgressPolicyStore object are creted and accessed +// on the same worker thread, and never cross thread boundaries. They are also never dropped. +unsafe impl Send for EgressPolicyStore {} + +impl EgressPolicyStore { + /// Creates a new [`EgressPolicyStore`]. + pub fn new() -> Self { + Self { + policies: HashMap::new(), + } + } + + /// Looks up a policy by key and returns a cloned [`Rc`] if present. + pub fn lookup(&self, key: u64) -> Option> { + self.policies.get(&key).cloned() + } + + /// Stores a cloned [`Rc`] under `key`. + pub fn store(&mut self, key: u64, policy: &Rc) { + self.policies.insert(key, Rc::clone(policy)); + } + + /// Removes the policy under `key` and returns it, or `None` if not present. + pub fn remove(&mut self, key: u64) -> Option> { + self.policies.remove(&key) + } +} + +/// Global store for egress policies holding a thread-local [`EgressPolicyStore`]. +#[derive(Debug, Default)] +pub struct GlobalEgressPolicyStore { + pub store: ThreadLocal>, +} + +impl GlobalEgressPolicyStore { + /// Creates a new [`GlobalEgressPolicyStore`]. + pub const fn new() -> Self { + Self { + store: ThreadLocal::new(), + } + } + + /// Returns a mutable reference to the current thread's [`EgressPolicyStore`], + /// initializing it on first access. + pub fn get_local_store() -> RefMut<'static, EgressPolicyStore> { + GLOBAL_EGRESS_POLICY_STORE + .store + .get_or(|| RefCell::new(EgressPolicyStore::new())) + .borrow_mut() + } +} + +/// Process-wide [`GlobalEgressPolicyStore`] instance. +pub static GLOBAL_EGRESS_POLICY_STORE: GlobalEgressPolicyStore = GlobalEgressPolicyStore::new(); + +// ========================================== +// Tests +// ========================================== + +#[cfg(test)] +mod tests { + use super::*; + use crate::egress_policy::SniRule; + + #[test] + fn test_global_egress_policy_store() { + let policy = Rc::new(EgressPolicy { + rules: vec![SniRule { + pattern: "api.example.com".to_string(), + mode: "mitm".to_string(), + }], + hash: 1, + }); + GlobalEgressPolicyStore::get_local_store().store(policy.hash, &policy); + let looked_up = GlobalEgressPolicyStore::get_local_store().lookup(policy.hash); + assert_eq!(looked_up, Some(Rc::clone(&policy))); + assert!(Rc::ptr_eq(&looked_up.unwrap(), &policy)); + assert_eq!(GlobalEgressPolicyStore::get_local_store().lookup(2), None); + + let removed = GlobalEgressPolicyStore::get_local_store().remove(policy.hash); + assert_eq!(removed, Some(Rc::clone(&policy))); + assert!(Rc::ptr_eq(&removed.unwrap(), &policy)); + assert_eq!(GlobalEgressPolicyStore::get_local_store().lookup(policy.hash), None); + assert_eq!(GlobalEgressPolicyStore::get_local_store().remove(policy.hash), None); + } +} diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/lib.rs b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/lib.rs index 271d087a6d..4397cbe6bd 100644 --- a/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/lib.rs +++ b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/lib.rs @@ -12,445 +12,37 @@ // See the License for the specific language governing permissions and // limitations under the License. -//! The egress-policy listener filter picks the filter chain for a tunneled -//! connection. See the README in this directory. - -use envoy_proxy_dynamic_modules_rust_sdk::{ - abi::envoy_dynamic_module_type_on_listener_filter_status, - declare_listener_filter_init_functions, envoy_log_trace, EnvoyListenerFilter, - EnvoyListenerFilterConfig, ListenerFilter, ListenerFilterConfig, -}; -use serde::Deserialize; - -/// Filter state holding the SNI rules as JSON. See -/// EgressPolicyMetadataNamespace in cmd/atenet/internal/router/extproc. -pub const ATE_POLICY_EGRESS: &[u8] = b"dev.ate.policy.egress"; - -/// Filter state this filter writes the chosen chain name to. -pub const ATE_EGRESS_FILTER_CHAIN: &[u8] = b"dev.ate.egress.filter_chain"; - -/// Verdict for TLS an https rule allows. -pub const ATE_EGRESS_FILTER_CHAIN_MITM: &str = "mitm"; - -/// Verdict for TLS a tls_passthrough rule allows. -pub const ATE_EGRESS_FILTER_CHAIN_PASSTHROUGH: &str = "passthrough"; - -/// Verdict for anything that is not TLS. -pub const ATE_EGRESS_FILTER_CHAIN_CLEARTEXT: &str = "cleartext"; - -/// Verdict for a denied connection. No chain has this name. -pub const ATE_EGRESS_FILTER_CHAIN_DENIED: &str = "denied"; - -/// Mode of a rule whose match terminates the connection. -pub const SNI_MODE_MITM: &str = "mitm"; - -/// Mode of a rule whose match forwards the connection without decryption. -pub const SNI_MODE_PASSTHROUGH: &str = "passthrough"; - -/// Transport protocol tls_inspector sets for TLS. -const TRANSPORT_TLS: &str = "tls"; - -/// Transport protocol Envoy assumes when no inspector detected one. -const TRANSPORT_RAW_BUFFER: &str = "raw_buffer"; - -/// The SNI rules for a connection. -#[derive(Debug, Deserialize, PartialEq)] -pub struct EgressPolicy { - /// Most specific first; the first match wins. - pub rules: Vec, -} - -/// A pattern and the mode applied when it matches. -#[derive(Debug, Deserialize, PartialEq)] -pub struct SniRule { - pub pattern: String, - pub mode: String, -} - -/// Reports whether a normalized `hostname` matches `pattern`. Must agree with -/// HostnamePattern.Matches in internal/egresspolicy. -pub fn pattern_matches(pattern: &str, hostname: &str) -> bool { - if hostname.is_empty() { - return false; - } - if pattern == "*" { - return true; - } - if let Some(suffix) = pattern.strip_prefix("*.") { - return match hostname - .strip_suffix(suffix) - .and_then(|rest| rest.strip_suffix('.')) - { - Some(label) => !label.is_empty() && !label.contains('.'), - None => false, - }; - } - pattern == hostname -} - -/// ASCII-lowercases an SNI and strips one trailing dot, as the gateway does. -fn normalize_sni(sni: &str) -> String { - let lower = sni.to_ascii_lowercase(); - match lower.strip_suffix('.') { - Some(stripped) => stripped.to_owned(), - None => lower, - } -} - -/// Returns the verdict from the transport alone, or None for TLS, which is -/// decided by SNI. Unknown transports are denied. -pub fn transport_verdict(transport: Option<&str>) -> Option<&'static str> { - match transport { - Some(TRANSPORT_TLS) => None, - None | Some(TRANSPORT_RAW_BUFFER) => Some(ATE_EGRESS_FILTER_CHAIN_CLEARTEXT), - Some(_) => Some(ATE_EGRESS_FILTER_CHAIN_DENIED), - } -} - -/// Returns the mode of the first rule matching the SNI, or denied. -pub fn tls_verdict(policy: Option<&EgressPolicy>, sni: Option<&str>) -> &'static str { - let (Some(policy), Some(sni)) = (policy, sni) else { - return ATE_EGRESS_FILTER_CHAIN_DENIED; - }; - let hostname = normalize_sni(sni); - match policy - .rules - .iter() - .find(|rule| pattern_matches(&rule.pattern, &hostname)) - { - Some(rule) if rule.mode == SNI_MODE_MITM => ATE_EGRESS_FILTER_CHAIN_MITM, - Some(rule) if rule.mode == SNI_MODE_PASSTHROUGH => ATE_EGRESS_FILTER_CHAIN_PASSTHROUGH, - _ => ATE_EGRESS_FILTER_CHAIN_DENIED, - } -} - -/// The filter takes no configuration. -pub struct EgressPolicyFilterConfig; - -impl ListenerFilterConfig for EgressPolicyFilterConfig { - fn new_listener_filter(&self, _envoy: &mut ELF) -> Box> { - Box::new(EgressPolicyFilter) - } -} - -/// Runs after tls_inspector and http_inspector, once per connection. -pub struct EgressPolicyFilter; - -impl ListenerFilter for EgressPolicyFilter { - fn on_accept( - &mut self, - envoy_filter: &mut ELF, - ) -> envoy_dynamic_module_type_on_listener_filter_status { - let transport = envoy_filter - .get_detected_transport_protocol() - .map(|value| String::from_utf8_lossy(value.as_slice()).into_owned()); - - let verdict = match transport_verdict(transport.as_deref()) { - Some(verdict) => verdict, - None => { - let sni = envoy_filter - .get_requested_server_name() - .map(|value| String::from_utf8_lossy(value.as_slice()).into_owned()); - // Unparseable rules deny, like absent ones. - let policy = envoy_filter - .get_filter_state_bytes(ATE_POLICY_EGRESS) - .and_then(|value| serde_json::from_slice::(value.as_slice()).ok()); - let verdict = tls_verdict(policy.as_ref(), sni.as_deref()); - envoy_log_trace!("egress policy: sni={:?} chain={}", sni, verdict); - verdict - } - }; - - envoy_filter.set_filter_state_bytes(ATE_EGRESS_FILTER_CHAIN, verdict.as_bytes()); - envoy_log_trace!( - "egress policy: transport={:?} chain={}", - transport, - verdict - ); - envoy_dynamic_module_type_on_listener_filter_status::Continue - } -} - -declare_listener_filter_init_functions!(init, new_listener_filter_config_fn); +//! The egress-policy dynamic module provides listener and HTTP filters for +//! egress policy evaluation and storage. See the README in this directory. + +pub mod egress_policy; +pub mod egress_policy_store; +pub mod listener_filter; +pub mod policy_store_filter; + +use envoy_proxy_dynamic_modules_rust_sdk::declare_all_init_functions; +pub use egress_policy::*; +pub use egress_policy_store::*; +pub use listener_filter::*; +pub use policy_store_filter::*; + +declare_all_init_functions!( + init, + http: policy_store_filter::new_http_filter_config_fn, + listener: listener_filter::new_listener_filter_config_fn, +); /// Called when the dynamic module is loaded into Envoy. fn init() -> bool { true } -/// Called when a new listener filter configuration is created. -fn new_listener_filter_config_fn< - EC: EnvoyListenerFilterConfig, - ELF: EnvoyListenerFilter, ->( - _envoy_filter_config: &mut EC, - _name: &str, - _config: &[u8], -) -> Option>> { - Some(Box::new(EgressPolicyFilterConfig)) -} - #[cfg(test)] mod tests { use super::*; - use envoy_proxy_dynamic_modules_rust_sdk::{ - EnvoyBuffer, MockEnvoyListenerFilter, MockEnvoyListenerFilterConfig, - }; - - fn policy(rules: &[(&str, &str)]) -> EgressPolicy { - EgressPolicy { - rules: rules - .iter() - .map(|(pattern, mode)| SniRule { - pattern: pattern.to_string(), - mode: mode.to_string(), - }) - .collect(), - } - } - - #[test] - fn test_pattern_matches() { - let cases: &[(&str, &str, bool)] = &[ - // "*" matches every name and only names. - ("*", "anything.example.com", true), - ("*", "localhost", true), - ("*", "", false), - // "*.suffix" matches exactly one non-empty leftmost label. - ("*.example.com", "api.example.com", true), - ("*.example.com", "example.com", false), - ("*.example.com", "a.b.example.com", false), - ("*.example.com", ".example.com", false), - ("*.example.com", "xexample.com", false), - ("*.example.com", "api.example.co", false), - ("*.example.com", "api.example.com.evil", false), - // Anything else matches the whole name. - ("api.example.com", "api.example.com", true), - ("api.example.com", "www.api.example.com", false), - ("api.example.com", "api.example.co", false), - ("example.com", "api.example.com", false), - // Normalization is the caller's job. - ("api.example.com", "API.example.com", false), - ("api.example.com", "api.example.com.", false), - ]; - for (pattern, hostname, want) in cases { - assert_eq!( - pattern_matches(pattern, hostname), - *want, - "pattern_matches({pattern:?}, {hostname:?})" - ); - } - } - - #[test] - fn test_normalize_sni() { - assert_eq!(normalize_sni("API.Example.COM."), "api.example.com"); - assert_eq!(normalize_sni("api.example.com"), "api.example.com"); - // One trailing dot, not every one. - assert_eq!(normalize_sni("api.example.com.."), "api.example.com."); - // Only ASCII folds: U+212A KELVIN SIGN stays itself and cannot become "k". - assert_eq!(normalize_sni("\u{212A}.example.com"), "\u{212A}.example.com"); - } - - #[test] - fn test_transport_verdict() { - assert_eq!(transport_verdict(Some("tls")), None); - assert_eq!(transport_verdict(None), Some(ATE_EGRESS_FILTER_CHAIN_CLEARTEXT)); - assert_eq!( - transport_verdict(Some("raw_buffer")), - Some(ATE_EGRESS_FILTER_CHAIN_CLEARTEXT) - ); - assert_eq!(transport_verdict(Some("quic")), Some(ATE_EGRESS_FILTER_CHAIN_DENIED)); - assert_eq!(transport_verdict(Some("")), Some(ATE_EGRESS_FILTER_CHAIN_DENIED)); - } - - #[test] - fn test_tls_verdict_denies_without_inputs() { - let p = policy(&[("*", SNI_MODE_MITM)]); - assert_eq!(tls_verdict(None, Some("api.example.com")), ATE_EGRESS_FILTER_CHAIN_DENIED); - assert_eq!(tls_verdict(Some(&p), None), ATE_EGRESS_FILTER_CHAIN_DENIED); - assert_eq!(tls_verdict(Some(&p), Some("")), ATE_EGRESS_FILTER_CHAIN_DENIED); - assert_eq!( - tls_verdict(Some(&policy(&[])), Some("api.example.com")), - ATE_EGRESS_FILTER_CHAIN_DENIED - ); - } - - #[test] - fn test_tls_verdict_matches_wildcards() { - let p = policy(&[("api.example.com", SNI_MODE_MITM), ("*.example.org", SNI_MODE_MITM)]); - assert_eq!(tls_verdict(Some(&p), Some("api.example.com")), ATE_EGRESS_FILTER_CHAIN_MITM); - assert_eq!(tls_verdict(Some(&p), Some("API.EXAMPLE.COM.")), ATE_EGRESS_FILTER_CHAIN_MITM); - assert_eq!(tls_verdict(Some(&p), Some("www.example.org")), ATE_EGRESS_FILTER_CHAIN_MITM); - assert_eq!(tls_verdict(Some(&p), Some("example.org")), ATE_EGRESS_FILTER_CHAIN_DENIED); - assert_eq!(tls_verdict(Some(&p), Some("a.b.example.org")), ATE_EGRESS_FILTER_CHAIN_DENIED); - assert_eq!(tls_verdict(Some(&p), Some("www.example.com")), ATE_EGRESS_FILTER_CHAIN_DENIED); - - let any = policy(&[("*", SNI_MODE_MITM)]); - assert_eq!(tls_verdict(Some(&any), Some("whatever.test")), ATE_EGRESS_FILTER_CHAIN_MITM); - } - - #[test] - fn test_tls_verdict_first_match_decides() { - // The first match is final, even with an unknown mode. - let unknown_first = policy(&[("*.example.com", "not-a-mode"), ("api.example.com", SNI_MODE_MITM)]); - assert_eq!( - tls_verdict(Some(&unknown_first), Some("api.example.com")), - ATE_EGRESS_FILTER_CHAIN_DENIED - ); - let known_first = policy(&[ - ("api.example.com", SNI_MODE_MITM), - ("pinned.example.com", SNI_MODE_PASSTHROUGH), - ("*.example.com", "not-a-mode"), - ]); - assert_eq!( - tls_verdict(Some(&known_first), Some("api.example.com")), - ATE_EGRESS_FILTER_CHAIN_MITM - ); - assert_eq!( - tls_verdict(Some(&known_first), Some("pinned.example.com")), - ATE_EGRESS_FILTER_CHAIN_PASSTHROUGH - ); - assert_eq!( - tls_verdict(Some(&known_first), Some("www.example.com")), - ATE_EGRESS_FILTER_CHAIN_DENIED - ); - } - - #[test] - fn test_policy_json_shape() { - let parsed: EgressPolicy = serde_json::from_str( - r#"{"rules":[{"pattern":"api.example.com","mode":"mitm"},{"pattern":"*","mode":"mitm"}]}"#, - ) - .unwrap(); - assert_eq!(parsed, policy(&[("api.example.com", "mitm"), ("*", "mitm")])); - assert!(serde_json::from_str::(r#"{"allowed_snis":["api.example.com"]}"#).is_err()); - } - - fn new_filter_config() -> Box> { - let mut mock_config = MockEnvoyListenerFilterConfig::new(); - new_listener_filter_config_fn::( - &mut mock_config, - "envoy_substrate_egress_policy", - b"", - ) - .expect("a filter config") - } - - /// Runs on_accept for a TLS connection with the given SNI and policy filter - /// state, asserting the verdict written. - fn assert_tls_verdict(sni: Option<&'static [u8]>, policy: Option<&'static [u8]>, want: &'static str) { - let config = new_filter_config(); - let mut mock_filter = MockEnvoyListenerFilter::new(); - mock_filter - .expect_get_detected_transport_protocol() - .returning(|| Some(EnvoyBuffer::new(b"tls"))); - mock_filter - .expect_get_requested_server_name() - .returning(move || sni.map(EnvoyBuffer::new)); - mock_filter - .expect_get_filter_state_bytes() - .withf(|key| key == ATE_POLICY_EGRESS) - .returning(move |_| policy.map(EnvoyBuffer::new)); - mock_filter - .expect_set_filter_state_bytes() - .withf(move |key, value| key == ATE_EGRESS_FILTER_CHAIN && value == want.as_bytes()) - .times(1) - .returning(|_, _| true); - - let mut filter = config.new_listener_filter(&mut mock_filter); - assert_eq!( - filter.on_accept(&mut mock_filter), - envoy_dynamic_module_type_on_listener_filter_status::Continue - ); - } - - /// Runs on_accept for a connection with the given detected transport, - /// asserting the verdict written. - fn assert_transport_verdict(transport: Option<&'static [u8]>, want: &'static str) { - let config = new_filter_config(); - let mut mock_filter = MockEnvoyListenerFilter::new(); - mock_filter - .expect_get_detected_transport_protocol() - .returning(move || transport.map(EnvoyBuffer::new)); - mock_filter - .expect_set_filter_state_bytes() - .withf(move |key, value| key == ATE_EGRESS_FILTER_CHAIN && value == want.as_bytes()) - .times(1) - .returning(|_, _| true); - - let mut filter = config.new_listener_filter(&mut mock_filter); - assert_eq!( - filter.on_accept(&mut mock_filter), - envoy_dynamic_module_type_on_listener_filter_status::Continue - ); - assert_eq!( - filter.on_data(&mut mock_filter, 0), - envoy_dynamic_module_type_on_listener_filter_status::Continue - ); - } #[test] fn test_init() { assert!(init()); } - - #[test] - fn test_on_accept_no_transport_is_cleartext() { - assert_transport_verdict(None, ATE_EGRESS_FILTER_CHAIN_CLEARTEXT); - } - - #[test] - fn test_on_accept_raw_buffer_is_cleartext() { - assert_transport_verdict(Some(b"raw_buffer"), ATE_EGRESS_FILTER_CHAIN_CLEARTEXT); - } - - #[test] - fn test_on_accept_unknown_transport_is_denied() { - assert_transport_verdict(Some(b"quic"), ATE_EGRESS_FILTER_CHAIN_DENIED); - } - - const RULES: &[u8] = - br#"{"rules":[{"pattern":"api.google.com","mode":"mitm"},{"pattern":"*.google.com","mode":"mitm"}]}"#; - - #[test] - fn test_on_accept_matching_sni() { - assert_tls_verdict(Some(b"api.google.com"), Some(RULES), ATE_EGRESS_FILTER_CHAIN_MITM); - } - - #[test] - fn test_on_accept_wildcard_sni() { - assert_tls_verdict(Some(b"mail.google.com"), Some(RULES), ATE_EGRESS_FILTER_CHAIN_MITM); - } - - #[test] - fn test_on_accept_case_insensitive_sni() { - assert_tls_verdict(Some(b"API.Google.COM"), Some(RULES), ATE_EGRESS_FILTER_CHAIN_MITM); - } - - #[test] - fn test_on_accept_mismatched_sni() { - assert_tls_verdict(Some(b"google.com"), Some(RULES), ATE_EGRESS_FILTER_CHAIN_DENIED); - assert_tls_verdict(Some(b"www.example.com"), Some(RULES), ATE_EGRESS_FILTER_CHAIN_DENIED); - } - - #[test] - fn test_on_accept_no_rules() { - assert_tls_verdict(Some(b"api.google.com"), Some(br#"{"rules":[]}"#), ATE_EGRESS_FILTER_CHAIN_DENIED); - } - - #[test] - fn test_on_accept_invalid_policy_json() { - assert_tls_verdict(Some(b"api.google.com"), Some(b"not-json"), ATE_EGRESS_FILTER_CHAIN_DENIED); - } - - #[test] - fn test_on_accept_missing_policy() { - assert_tls_verdict(Some(b"api.google.com"), None, ATE_EGRESS_FILTER_CHAIN_DENIED); - } - - #[test] - fn test_on_accept_missing_sni() { - assert_tls_verdict(None, Some(RULES), ATE_EGRESS_FILTER_CHAIN_DENIED); - } } diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/listener_filter.rs b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/listener_filter.rs new file mode 100644 index 0000000000..3dabdf8279 --- /dev/null +++ b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/listener_filter.rs @@ -0,0 +1,501 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! Listener filter implementation for the egress-policy dynamic module. + +use envoy_proxy_dynamic_modules_rust_sdk::{ + abi::envoy_dynamic_module_type_on_listener_filter_status, envoy_log_trace, EnvoyListenerFilter, + EnvoyListenerFilterConfig, ListenerFilter, ListenerFilterConfig, +}; +pub use crate::egress_policy::{EgressPolicy, SniRule}; +use crate::egress_policy_store::GlobalEgressPolicyStore; + +/// Filter state holding the egress policy hash. +pub const ATE_POLICY_EGRESS_HASH: &[u8] = b"dev.ate.policy.egress.hash"; + +/// Filter state this filter writes the chosen chain name to. +pub const ATE_EGRESS_FILTER_CHAIN: &[u8] = b"dev.ate.egress.filter_chain"; + +/// Verdict for TLS an https rule allows. +pub const ATE_EGRESS_FILTER_CHAIN_MITM: &str = "mitm"; + +/// Verdict for TLS a tls_passthrough rule allows. +pub const ATE_EGRESS_FILTER_CHAIN_PASSTHROUGH: &str = "passthrough"; + +/// Verdict for anything that is not TLS. +pub const ATE_EGRESS_FILTER_CHAIN_CLEARTEXT: &str = "cleartext"; + +/// Verdict for a denied connection. No chain has this name. +pub const ATE_EGRESS_FILTER_CHAIN_DENIED: &str = "denied"; + +/// Mode of a rule whose match terminates the connection. +pub const SNI_MODE_MITM: &str = "mitm"; + +/// Mode of a rule whose match forwards the connection without decryption. +pub const SNI_MODE_PASSTHROUGH: &str = "passthrough"; + +/// Transport protocol tls_inspector sets for TLS. +const TRANSPORT_TLS: &str = "tls"; + +/// Transport protocol Envoy assumes when no inspector detected one. +const TRANSPORT_RAW_BUFFER: &str = "raw_buffer"; + +/// Reports whether a normalized `hostname` matches `pattern`. Must agree with +/// HostnamePattern.Matches in internal/egresspolicy. +pub fn pattern_matches(pattern: &str, hostname: &str) -> bool { + if hostname.is_empty() { + return false; + } + if pattern == "*" { + return true; + } + if let Some(suffix) = pattern.strip_prefix("*.") { + return match hostname + .strip_suffix(suffix) + .and_then(|rest| rest.strip_suffix('.')) + { + Some(label) => !label.is_empty() && !label.contains('.'), + None => false, + }; + } + pattern == hostname +} + +/// ASCII-lowercases an SNI and strips one trailing dot, as the gateway does. +fn normalize_sni(sni: &str) -> String { + let lower = sni.to_ascii_lowercase(); + match lower.strip_suffix('.') { + Some(stripped) => stripped.to_owned(), + None => lower, + } +} + +/// Returns the verdict from the transport alone, or None for TLS, which is +/// decided by SNI. Unknown transports are denied. +pub fn transport_verdict(transport: Option<&str>) -> Option<&'static str> { + match transport { + Some(TRANSPORT_TLS) => None, + None | Some(TRANSPORT_RAW_BUFFER) => Some(ATE_EGRESS_FILTER_CHAIN_CLEARTEXT), + Some(_) => Some(ATE_EGRESS_FILTER_CHAIN_DENIED), + } +} + +/// Returns the mode of the first rule matching the SNI, or denied. +pub fn tls_verdict(policy: Option<&EgressPolicy>, sni: Option<&str>) -> &'static str { + let (Some(policy), Some(sni)) = (policy, sni) else { + return ATE_EGRESS_FILTER_CHAIN_DENIED; + }; + let hostname = normalize_sni(sni); + match policy + .rules + .iter() + .find(|rule| pattern_matches(&rule.pattern, &hostname)) + { + Some(rule) if rule.mode == SNI_MODE_MITM => ATE_EGRESS_FILTER_CHAIN_MITM, + Some(rule) if rule.mode == SNI_MODE_PASSTHROUGH => ATE_EGRESS_FILTER_CHAIN_PASSTHROUGH, + _ => ATE_EGRESS_FILTER_CHAIN_DENIED, + } +} + +/// The filter takes no configuration. +pub struct EgressPolicyFilterConfig; + +impl ListenerFilterConfig for EgressPolicyFilterConfig { + fn new_listener_filter(&self, _envoy: &mut ELF) -> Box> { + Box::new(EgressPolicyFilter) + } +} + +/// Runs after tls_inspector and http_inspector, once per connection. +pub struct EgressPolicyFilter; + +impl ListenerFilter for EgressPolicyFilter { + fn on_accept( + &mut self, + envoy_filter: &mut ELF, + ) -> envoy_dynamic_module_type_on_listener_filter_status { + let transport = envoy_filter + .get_detected_transport_protocol() + .map(|value| String::from_utf8_lossy(value.as_slice()).into_owned()); + + let verdict = match transport_verdict(transport.as_deref()) { + Some(verdict) => verdict, + None => { + let sni = envoy_filter + .get_requested_server_name() + .map(|value| String::from_utf8_lossy(value.as_slice()).into_owned()); + // Read policy hash from the filter state and lookup the policy for this hash + // in the worker thread local store. + // Since the HTTP filter in the CONNECt filter chain holds a reference to the + // policy object in store, it is guaranteed that policy lookup will always succeed. + let policy = envoy_filter + .get_filter_state_bytes(ATE_POLICY_EGRESS_HASH) + .and_then(|value| std::str::from_utf8(value.as_slice()).ok()?.parse::().ok()) + .and_then(|hash| GlobalEgressPolicyStore::get_local_store().lookup(hash)); + let verdict = match policy { + Some(ref policy) => tls_verdict(Some(policy), sni.as_deref()), + None => ATE_EGRESS_FILTER_CHAIN_DENIED, + }; + envoy_log_trace!("egress policy: sni={:?} chain={}", sni, verdict); + verdict + } + }; + + envoy_filter.set_filter_state_bytes(ATE_EGRESS_FILTER_CHAIN, verdict.as_bytes()); + envoy_log_trace!( + "egress policy: transport={:?} chain={}", + transport, + verdict + ); + envoy_dynamic_module_type_on_listener_filter_status::Continue + } +} + +/// Called when a new listener filter configuration is created. +pub fn new_listener_filter_config_fn< + EC: EnvoyListenerFilterConfig, + ELF: EnvoyListenerFilter, +>( + _envoy_filter_config: &mut EC, + _name: &str, + _config: &[u8], +) -> Option>> { + Some(Box::new(EgressPolicyFilterConfig)) +} + +// ========================================== +// Tests +// ========================================== + +#[cfg(test)] +mod tests { + use super::*; + use envoy_proxy_dynamic_modules_rust_sdk::{ + EnvoyBuffer, MockEnvoyListenerFilter, MockEnvoyListenerFilterConfig, + }; + + fn policy(rules: &[(&str, &str)]) -> EgressPolicy { + EgressPolicy { + rules: rules + .iter() + .map(|(pattern, mode)| SniRule { + pattern: pattern.to_string(), + mode: mode.to_string(), + }) + .collect(), + hash: 0, + } + } + + #[test] + fn test_pattern_matches() { + let cases: &[(&str, &str, bool)] = &[ + // "*" matches every name and only names. + ("*", "anything.example.com", true), + ("*", "localhost", true), + ("*", "", false), + // "*.suffix" matches exactly one non-empty leftmost label. + ("*.example.com", "api.example.com", true), + ("*.example.com", "example.com", false), + ("*.example.com", "a.b.example.com", false), + ("*.example.com", ".example.com", false), + ("*.example.com", "xexample.com", false), + ("*.example.com", "api.example.co", false), + ("*.example.com", "api.example.com.evil", false), + // Anything else matches the whole name. + ("api.example.com", "api.example.com", true), + ("api.example.com", "www.api.example.com", false), + ("api.example.com", "api.example.co", false), + ("example.com", "api.example.com", false), + // Normalization is the caller's job. + ("api.example.com", "API.example.com", false), + ("api.example.com", "api.example.com.", false), + ]; + for (pattern, hostname, want) in cases { + assert_eq!( + pattern_matches(pattern, hostname), + *want, + "pattern_matches({pattern:?}, {hostname:?})" + ); + } + } + + #[test] + fn test_normalize_sni() { + assert_eq!(normalize_sni("API.Example.COM."), "api.example.com"); + assert_eq!(normalize_sni("api.example.com"), "api.example.com"); + // One trailing dot, not every one. + assert_eq!(normalize_sni("api.example.com.."), "api.example.com."); + // Only ASCII folds: U+212A KELVIN SIGN stays itself and cannot become "k". + assert_eq!(normalize_sni("\u{212A}.example.com"), "\u{212A}.example.com"); + } + + #[test] + fn test_transport_verdict() { + assert_eq!(transport_verdict(Some("tls")), None); + assert_eq!(transport_verdict(None), Some(ATE_EGRESS_FILTER_CHAIN_CLEARTEXT)); + assert_eq!( + transport_verdict(Some("raw_buffer")), + Some(ATE_EGRESS_FILTER_CHAIN_CLEARTEXT) + ); + assert_eq!(transport_verdict(Some("quic")), Some(ATE_EGRESS_FILTER_CHAIN_DENIED)); + assert_eq!(transport_verdict(Some("")), Some(ATE_EGRESS_FILTER_CHAIN_DENIED)); + } + + #[test] + fn test_tls_verdict_denies_without_inputs() { + let p = policy(&[("*", SNI_MODE_MITM)]); + assert_eq!(tls_verdict(None, Some("api.example.com")), ATE_EGRESS_FILTER_CHAIN_DENIED); + assert_eq!(tls_verdict(Some(&p), None), ATE_EGRESS_FILTER_CHAIN_DENIED); + assert_eq!(tls_verdict(Some(&p), Some("")), ATE_EGRESS_FILTER_CHAIN_DENIED); + assert_eq!( + tls_verdict(Some(&policy(&[])), Some("api.example.com")), + ATE_EGRESS_FILTER_CHAIN_DENIED + ); + } + + #[test] + fn test_tls_verdict_matches_wildcards() { + let p = policy(&[("api.example.com", SNI_MODE_MITM), ("*.example.org", SNI_MODE_MITM)]); + assert_eq!(tls_verdict(Some(&p), Some("api.example.com")), ATE_EGRESS_FILTER_CHAIN_MITM); + assert_eq!(tls_verdict(Some(&p), Some("API.EXAMPLE.COM.")), ATE_EGRESS_FILTER_CHAIN_MITM); + assert_eq!(tls_verdict(Some(&p), Some("www.example.org")), ATE_EGRESS_FILTER_CHAIN_MITM); + assert_eq!(tls_verdict(Some(&p), Some("example.org")), ATE_EGRESS_FILTER_CHAIN_DENIED); + assert_eq!(tls_verdict(Some(&p), Some("a.b.example.org")), ATE_EGRESS_FILTER_CHAIN_DENIED); + assert_eq!(tls_verdict(Some(&p), Some("www.example.com")), ATE_EGRESS_FILTER_CHAIN_DENIED); + + let any = policy(&[("*", SNI_MODE_MITM)]); + assert_eq!(tls_verdict(Some(&any), Some("whatever.test")), ATE_EGRESS_FILTER_CHAIN_MITM); + } + + #[test] + fn test_tls_verdict_first_match_decides() { + // The first match is final, even with an unknown mode. + let unknown_first = policy(&[("*.example.com", "not-a-mode"), ("api.example.com", SNI_MODE_MITM)]); + assert_eq!( + tls_verdict(Some(&unknown_first), Some("api.example.com")), + ATE_EGRESS_FILTER_CHAIN_DENIED + ); + let known_first = policy(&[ + ("api.example.com", SNI_MODE_MITM), + ("pinned.example.com", SNI_MODE_PASSTHROUGH), + ("*.example.com", "not-a-mode"), + ]); + assert_eq!( + tls_verdict(Some(&known_first), Some("api.example.com")), + ATE_EGRESS_FILTER_CHAIN_MITM + ); + assert_eq!( + tls_verdict(Some(&known_first), Some("pinned.example.com")), + ATE_EGRESS_FILTER_CHAIN_PASSTHROUGH + ); + assert_eq!( + tls_verdict(Some(&known_first), Some("www.example.com")), + ATE_EGRESS_FILTER_CHAIN_DENIED + ); + } + + #[test] + fn test_policy_json_shape() { + let parsed: EgressPolicy = serde_json::from_str( + r#"{"rules":[{"pattern":"api.example.com","mode":"mitm"},{"pattern":"*","mode":"mitm"}]}"#, + ) + .unwrap(); + assert_eq!(parsed, policy(&[("api.example.com", "mitm"), ("*", "mitm")])); + assert!(serde_json::from_str::(r#"{"allowed_snis":["api.example.com"]}"#).is_err()); + } + + fn new_filter_config() -> Box> { + let mut mock_config = MockEnvoyListenerFilterConfig::new(); + new_listener_filter_config_fn::( + &mut mock_config, + "envoy_substrate_egress_policy", + b"", + ) + .expect("a filter config") + } + + /// Runs on_accept for a TLS connection with the given SNI and policy hash + /// filter state, asserting the verdict written. + fn assert_tls_verdict_with_hash_state( + sni: Option<&'static [u8]>, + hash_state: Option>, + want: &'static str, + ) { + let config = new_filter_config(); + let mut mock_filter = MockEnvoyListenerFilter::new(); + mock_filter + .expect_get_detected_transport_protocol() + .returning(|| Some(EnvoyBuffer::new(b"tls"))); + mock_filter + .expect_get_requested_server_name() + .returning(move || sni.map(EnvoyBuffer::new)); + mock_filter + .expect_get_filter_state_bytes() + .withf(|key| key == ATE_POLICY_EGRESS_HASH) + .returning(move |_| hash_state.as_deref().map(EnvoyBuffer::new)); + mock_filter + .expect_set_filter_state_bytes() + .withf(move |key, value| key == ATE_EGRESS_FILTER_CHAIN && value == want.as_bytes()) + .times(1) + .returning(|_, _| true); + + let mut filter = config.new_listener_filter(&mut mock_filter); + assert_eq!( + filter.on_accept(&mut mock_filter), + envoy_dynamic_module_type_on_listener_filter_status::Continue + ); + } + + /// Stores the given policy (if any) in the thread-local store and runs + /// on_accept with its hash in filter state. + fn assert_tls_verdict( + sni: Option<&'static [u8]>, + policy: Option, + want: &'static str, + ) { + let hash_state = policy.map(|p| { + let rc = std::rc::Rc::new(p); + let hash = rc.hash; + GlobalEgressPolicyStore::get_local_store().store(hash, &rc); + hash.to_string().into_bytes() + }); + assert_tls_verdict_with_hash_state(sni, hash_state, want); + } + + /// Runs on_accept for a connection with the given detected transport, + /// asserting the verdict written. + fn assert_transport_verdict(transport: Option<&'static [u8]>, want: &'static str) { + let config = new_filter_config(); + let mut mock_filter = MockEnvoyListenerFilter::new(); + mock_filter + .expect_get_detected_transport_protocol() + .returning(move || transport.map(EnvoyBuffer::new)); + mock_filter + .expect_set_filter_state_bytes() + .withf(move |key, value| key == ATE_EGRESS_FILTER_CHAIN && value == want.as_bytes()) + .times(1) + .returning(|_, _| true); + + let mut filter = config.new_listener_filter(&mut mock_filter); + assert_eq!( + filter.on_accept(&mut mock_filter), + envoy_dynamic_module_type_on_listener_filter_status::Continue + ); + assert_eq!( + filter.on_data(&mut mock_filter, 0), + envoy_dynamic_module_type_on_listener_filter_status::Continue + ); + } + + #[test] + fn test_on_accept_no_transport_is_cleartext() { + assert_transport_verdict(None, ATE_EGRESS_FILTER_CHAIN_CLEARTEXT); + } + + #[test] + fn test_on_accept_raw_buffer_is_cleartext() { + assert_transport_verdict(Some(b"raw_buffer"), ATE_EGRESS_FILTER_CHAIN_CLEARTEXT); + } + + #[test] + fn test_on_accept_unknown_transport_is_denied() { + assert_transport_verdict(Some(b"quic"), ATE_EGRESS_FILTER_CHAIN_DENIED); + } + + fn google_rules() -> EgressPolicy { + policy(&[ + ("api.google.com", SNI_MODE_MITM), + ("*.google.com", SNI_MODE_MITM), + ]) + } + + #[test] + fn test_on_accept_matching_sni() { + assert_tls_verdict( + Some(b"api.google.com"), + Some(google_rules()), + ATE_EGRESS_FILTER_CHAIN_MITM, + ); + } + + #[test] + fn test_on_accept_wildcard_sni() { + assert_tls_verdict( + Some(b"mail.google.com"), + Some(google_rules()), + ATE_EGRESS_FILTER_CHAIN_MITM, + ); + } + + #[test] + fn test_on_accept_case_insensitive_sni() { + assert_tls_verdict( + Some(b"API.Google.COM"), + Some(google_rules()), + ATE_EGRESS_FILTER_CHAIN_MITM, + ); + } + + #[test] + fn test_on_accept_mismatched_sni() { + assert_tls_verdict( + Some(b"google.com"), + Some(google_rules()), + ATE_EGRESS_FILTER_CHAIN_DENIED, + ); + assert_tls_verdict( + Some(b"www.example.com"), + Some(google_rules()), + ATE_EGRESS_FILTER_CHAIN_DENIED, + ); + } + + #[test] + fn test_on_accept_no_rules() { + assert_tls_verdict( + Some(b"api.google.com"), + Some(policy(&[])), + ATE_EGRESS_FILTER_CHAIN_DENIED, + ); + } + + #[test] + fn test_on_accept_invalid_policy_hash() { + assert_tls_verdict_with_hash_state( + Some(b"api.google.com"), + Some(b"not-a-hash".to_vec()), + ATE_EGRESS_FILTER_CHAIN_DENIED, + ); + } + + #[test] + fn test_on_accept_unknown_policy_hash() { + assert_tls_verdict_with_hash_state( + Some(b"api.google.com"), + Some(b"9999999999999999999".to_vec()), + ATE_EGRESS_FILTER_CHAIN_DENIED, + ); + } + + #[test] + fn test_on_accept_missing_policy() { + assert_tls_verdict(Some(b"api.google.com"), None, ATE_EGRESS_FILTER_CHAIN_DENIED); + } + + #[test] + fn test_on_accept_missing_sni() { + assert_tls_verdict(None, Some(google_rules()), ATE_EGRESS_FILTER_CHAIN_DENIED); + } +} diff --git a/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/policy_store_filter.rs b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/policy_store_filter.rs new file mode 100644 index 0000000000..d53c2c3963 --- /dev/null +++ b/cmd/dataplane/envoy/dynamic-modules/egress-policy/src/policy_store_filter.rs @@ -0,0 +1,430 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//! HTTP filter implementation for storing the egress policy. + +use std::rc::Rc; + +use envoy_proxy_dynamic_modules_rust_sdk::{ + abi::{ + envoy_dynamic_module_type_on_http_filter_request_headers_status, + }, + envoy_log_trace, EnvoyCounterId, EnvoyGaugeId, EnvoyHttpFilter, EnvoyHttpFilterConfig, + HttpFilter, HttpFilterConfig, +}; +use crate::egress_policy::EgressPolicy; +use crate::egress_policy_store::GlobalEgressPolicyStore; + +/// Filter state key where the unshared egress policy hash is written. +pub const ATE_POLICY_EGRESS_HASH_NOT_SHARED: &[u8] = b"dev.ate.policy.egress.hash.not_shared"; + +/// Gauge for the size of egress policy store. +pub const EGRESS_POLICY_STORE_SIZE: &str = "ate_egress.policy_store_size"; + +/// Counter for egress policy parse errors. +pub const EGRESS_POLICY_PARSE_ERRORS: &str = "ate_egress.parse_errors"; + +/// Counter for shared egress policies already present in the local store. +pub const EGRESS_POLICY_SHARED: &str = "ate_egress.policy_shared"; + +/// Configuration for the egress-policy-store HTTP filter. +pub struct EgressPolicyStoreFilterConfig { + pub store_size: EnvoyGaugeId, + pub parse_errors: EnvoyCounterId, + pub policy_shared: EnvoyCounterId, +} + +impl HttpFilterConfig for EgressPolicyStoreFilterConfig { + fn new_http_filter(&self, _envoy: &mut EHF) -> Box> { + Box::new(EgressPolicyStoreFilter { + policy: Rc::default(), + store_size: self.store_size, + parse_errors: self.parse_errors, + policy_shared: self.policy_shared, + }) + } +} + +/// Per-stream HTTP filter instance for egress-policy-store. +pub struct EgressPolicyStoreFilter { + pub policy: Rc, + pub store_size: EnvoyGaugeId, + pub parse_errors: EnvoyCounterId, + pub policy_shared: EnvoyCounterId, +} + +impl Default for EgressPolicyStoreFilter { + fn default() -> Self { + Self { + policy: Rc::default(), + store_size: EnvoyGaugeId(0), + parse_errors: EnvoyCounterId(0), + policy_shared: EnvoyCounterId(0), + } + } +} + +impl HttpFilter for EgressPolicyStoreFilter { + fn on_request_headers( + &mut self, + envoy_filter: &mut EHF, + _end_of_stream: bool, + ) -> envoy_dynamic_module_type_on_http_filter_request_headers_status { + let Some(policy) = EgressPolicy::new_from_metadata(envoy_filter) else { + let _ = envoy_filter.increment_counter(self.parse_errors, 1); + // TODO: return an error response + return envoy_dynamic_module_type_on_http_filter_request_headers_status::Continue; + }; + envoy_log_trace!("egress policy store: policy={:?}", policy); + let mut local_store = GlobalEgressPolicyStore::get_local_store(); + if let Some(existing_policy) = local_store.lookup(policy.hash) { + // Sanity check that the policy is the same and fail closed on hash collisions. + if existing_policy != policy { + envoy_filter.send_response(400, &[], Some(b"hash collision"), Some("hash_collision")); + return envoy_dynamic_module_type_on_http_filter_request_headers_status::StopIteration; + } + let _ = envoy_filter.increment_counter(self.policy_shared, 1); + // Use the policy that was alredy in the store. + self.policy = existing_policy; + } else { + local_store.store(policy.hash, &policy); + let _ = envoy_filter.increase_gauge(self.store_size, 1); + self.policy = policy; + } + // Dynamic module API does not yet support writing filter state values + // that are shared with the upstream (will be available in Envoy v1.40). + // Write non shared value, and then write shared value in the set_filter_state + // filter which is next in the filter chain. + // TODO(yanavlasov): remove this workaround after Envoy v1.40 is released. + envoy_filter.set_filter_state_bytes( + ATE_POLICY_EGRESS_HASH_NOT_SHARED, + self.policy.hash.to_string().as_bytes(), + ); + envoy_dynamic_module_type_on_http_filter_request_headers_status::Continue + } + + // on_stream_complete is called right before the filter object is dropped + // This is the last callback that still has valid filter ref. + fn on_stream_complete(&mut self, envoy_filter: &mut EHF) { + // Reference count 2 for the policy object means that it is held by the policy + // store table and this filter that is about to be dropped. + // To avoid leaking the policy in the store, remove it from the store. When on_stream_complete + // callback finishes and and filter is dropped, the only remaining reference to the policy is + // decremented and the policy object will be dropped as well. + if Rc::strong_count(&self.policy) == 2 { + GlobalEgressPolicyStore::get_local_store().remove(self.policy.hash); + let _ = envoy_filter.decrease_gauge(self.store_size, 1); + } + } +} + +/// Called when a new HTTP filter configuration is created. +pub fn new_http_filter_config_fn( + envoy_filter_config: &mut EC, + _name: &str, + _config: &[u8], +) -> Option>> { + let store_size = envoy_filter_config.define_gauge(EGRESS_POLICY_STORE_SIZE).ok()?; + let parse_errors = envoy_filter_config + .define_counter(EGRESS_POLICY_PARSE_ERRORS) + .ok()?; + let policy_shared = envoy_filter_config + .define_counter(EGRESS_POLICY_SHARED) + .ok()?; + Some(Box::new(EgressPolicyStoreFilterConfig { + store_size, + parse_errors, + policy_shared, + })) +} + +// ========================================== +// Tests +// ========================================== + +#[cfg(test)] +mod tests { + use super::*; + use envoy_proxy_dynamic_modules_rust_sdk::{ + abi::envoy_dynamic_module_type_metadata_source, EnvoyBuffer, MockEnvoyHttpFilter, + }; + use crate::egress_policy::{ + SniRule, ATE_POLICY_EGRESS_NAMESPACE, ATE_POLICY_JSON_KEY, + }; + + #[test] + fn test_on_request_headers_parses_policy() { + let config = EgressPolicyStoreFilterConfig { + store_size: EnvoyGaugeId(1), + parse_errors: EnvoyCounterId(2), + policy_shared: EnvoyCounterId(3), + }; + let mut filter = EgressPolicyStoreFilter { + policy: Rc::default(), + store_size: config.store_size, + parse_errors: config.parse_errors, + policy_shared: config.policy_shared, + }; + let mut mock_filter = MockEnvoyHttpFilter::default(); + mock_filter + .expect_get_metadata_string() + .withf(|source, namespace, key| { + *source == envoy_dynamic_module_type_metadata_source::Dynamic + && namespace == ATE_POLICY_EGRESS_NAMESPACE + && key == ATE_POLICY_JSON_KEY + }) + .returning(|_, _, _| { + Some(EnvoyBuffer::new( + br#"{"rules":[{"pattern":"api.example.com","mode":"mitm"},{"pattern":"*.example.com","mode":"passthrough"}]}"#, + )) + }) + .once(); + mock_filter + .expect_set_dynamic_metadata_string() + .withf(|namespace, key, value| { + namespace == ATE_POLICY_EGRESS_NAMESPACE && key == ATE_POLICY_JSON_KEY && value.is_empty() + }) + .return_const(()) + .once(); + mock_filter + .expect_increase_gauge() + .withf(|id, value| *id == EnvoyGaugeId(1) && *value == 1) + .returning(|_, _| Ok(())) + .once(); + mock_filter + .expect_set_filter_state_bytes() + .withf(|key, value| { + key == ATE_POLICY_EGRESS_HASH_NOT_SHARED + && std::str::from_utf8(value) + .ok() + .and_then(|s| s.parse::().ok()) + .is_some() + }) + .returning(|_, _| true) + .once(); + + assert_eq!( + filter.on_request_headers(&mut mock_filter, false), + envoy_dynamic_module_type_on_http_filter_request_headers_status::Continue + ); + assert_eq!( + filter.policy.rules, + vec![ + SniRule { + pattern: "api.example.com".to_string(), + mode: "mitm".to_string(), + }, + SniRule { + pattern: "*.example.com".to_string(), + mode: "passthrough".to_string(), + }, + ] + ); + assert_ne!(filter.policy.hash, 0); + + let hash = filter.policy.hash; + assert!(Rc::ptr_eq( + &GlobalEgressPolicyStore::get_local_store() + .lookup(hash) + .unwrap(), + &filter.policy + )); + + // A second filter receiving the same policy reuses the stored Rc without incrementing store_size. + let mut second_filter = EgressPolicyStoreFilter { + policy: Rc::default(), + store_size: config.store_size, + parse_errors: config.parse_errors, + policy_shared: config.policy_shared, + }; + mock_filter + .expect_get_metadata_string() + .withf(|source, namespace, key| { + *source == envoy_dynamic_module_type_metadata_source::Dynamic + && namespace == ATE_POLICY_EGRESS_NAMESPACE + && key == ATE_POLICY_JSON_KEY + }) + .returning(|_, _, _| { + Some(EnvoyBuffer::new( + br#"{"rules":[{"pattern":"api.example.com","mode":"mitm"},{"pattern":"*.example.com","mode":"passthrough"}]}"#, + )) + }) + .once(); + mock_filter + .expect_set_dynamic_metadata_string() + .withf(|namespace, key, value| { + namespace == ATE_POLICY_EGRESS_NAMESPACE && key == ATE_POLICY_JSON_KEY && value.is_empty() + }) + .return_const(()) + .once(); + mock_filter + .expect_increment_counter() + .withf(|id, value| *id == EnvoyCounterId(3) && *value == 1) + .returning(|_, _| Ok(())) + .once(); + let hash_bytes_second = hash.to_string().into_bytes(); + mock_filter + .expect_set_filter_state_bytes() + .withf(move |key, value| { + key == ATE_POLICY_EGRESS_HASH_NOT_SHARED && value == hash_bytes_second + }) + .returning(|_, _| true) + .once(); + + assert_eq!( + second_filter.on_request_headers(&mut mock_filter, false), + envoy_dynamic_module_type_on_http_filter_request_headers_status::Continue + ); + assert!(Rc::ptr_eq(&second_filter.policy, &filter.policy)); + + // Completing and dropping the first filter keeps the policy in the local store while the second filter holds it. + filter.on_stream_complete(&mut mock_filter); + drop(filter); + assert!(GlobalEgressPolicyStore::get_local_store() + .lookup(hash) + .is_some()); + + // Completing and dropping the last filter holding the policy decrements store_size and removes it from the local store. + mock_filter + .expect_decrease_gauge() + .withf(|id, value| *id == EnvoyGaugeId(1) && *value == 1) + .returning(|_, _| Ok(())) + .once(); + second_filter.on_stream_complete(&mut mock_filter); + drop(second_filter); + assert!(GlobalEgressPolicyStore::get_local_store() + .lookup(hash) + .is_none()); + } + + #[test] + fn test_on_request_headers_missing_or_invalid_policy() { + let mut filter = EgressPolicyStoreFilter { + parse_errors: EnvoyCounterId(2), + ..Default::default() + }; + let mut mock_filter = MockEnvoyHttpFilter::default(); + mock_filter + .expect_get_metadata_string() + .withf(|source, namespace, key| { + *source == envoy_dynamic_module_type_metadata_source::Dynamic + && namespace == ATE_POLICY_EGRESS_NAMESPACE + && key == ATE_POLICY_JSON_KEY + }) + .returning(|_, _, _| None) + .once(); + mock_filter + .expect_increment_counter() + .withf(|id, value| *id == EnvoyCounterId(2) && *value == 1) + .returning(|_, _| Ok(())) + .once(); + + assert_eq!( + filter.on_request_headers(&mut mock_filter, false), + envoy_dynamic_module_type_on_http_filter_request_headers_status::Continue + ); + assert_eq!(filter.policy, Rc::default()); + + mock_filter + .expect_get_metadata_string() + .withf(|source, namespace, key| { + *source == envoy_dynamic_module_type_metadata_source::Dynamic + && namespace == ATE_POLICY_EGRESS_NAMESPACE + && key == ATE_POLICY_JSON_KEY + }) + .returning(|_, _, _| Some(EnvoyBuffer::new(b"not-json"))) + .once(); + mock_filter + .expect_increment_counter() + .withf(|id, value| *id == EnvoyCounterId(2) && *value == 1) + .returning(|_, _| Ok(())) + .once(); + + assert_eq!( + filter.on_request_headers(&mut mock_filter, false), + envoy_dynamic_module_type_on_http_filter_request_headers_status::Continue + ); + assert_eq!(filter.policy, Rc::default()); + } + + #[test] + fn test_on_request_headers_hash_collision() { + let mut setup_mock = MockEnvoyHttpFilter::default(); + setup_mock + .expect_get_metadata_string() + .returning(|_, _, _| { + Some(EnvoyBuffer::new( + br#"{"rules":[{"pattern":"api.example.com","mode":"mitm"}]}"#, + )) + }) + .once(); + setup_mock + .expect_set_dynamic_metadata_string() + .return_const(()) + .once(); + let incoming_policy = EgressPolicy::new_from_metadata(&mut setup_mock).unwrap(); + let colliding_hash = incoming_policy.hash; + + // Create a different policy but with the same hash to test filter behavior. + let existing_policy = Rc::new(EgressPolicy { + rules: vec![SniRule { + pattern: "other.example.com".to_string(), + mode: "passthrough".to_string(), + }], + hash: colliding_hash, + }); + GlobalEgressPolicyStore::get_local_store().store(colliding_hash, &existing_policy); + + let mut filter = EgressPolicyStoreFilter::default(); + let mut mock_filter = MockEnvoyHttpFilter::default(); + mock_filter + .expect_get_metadata_string() + .withf(|source, namespace, key| { + *source == envoy_dynamic_module_type_metadata_source::Dynamic + && namespace == ATE_POLICY_EGRESS_NAMESPACE + && key == ATE_POLICY_JSON_KEY + }) + .returning(|_, _, _| { + Some(EnvoyBuffer::new( + br#"{"rules":[{"pattern":"api.example.com","mode":"mitm"}]}"#, + )) + }) + .once(); + mock_filter + .expect_set_dynamic_metadata_string() + .withf(|namespace, key, value| { + namespace == ATE_POLICY_EGRESS_NAMESPACE && key == ATE_POLICY_JSON_KEY && value.is_empty() + }) + .return_const(()) + .once(); + mock_filter + .expect_send_response() + .withf(|status_code, headers, body, details| { + *status_code == 400 + && headers.is_empty() + && body == &Some(&b"hash collision"[..]) + && details == &Some("hash_collision") + }) + .return_const(()) + .once(); + + assert_eq!( + filter.on_request_headers(&mut mock_filter, false), + envoy_dynamic_module_type_on_http_filter_request_headers_status::StopIteration + ); + assert_eq!(filter.policy, Rc::default()); + + GlobalEgressPolicyStore::get_local_store().remove(colliding_hash); + } +} diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md index 81a9528150..5b058683a5 100644 --- a/cmd/kubectl-ate/README.md +++ b/cmd/kubectl-ate/README.md @@ -288,7 +288,7 @@ kubectl ate delete egress-policy -a --uid --versio ### Logs -`kubectl ate logs` requires a resource-type subcommand; running `kubectl ate logs ` on its own prints help. The only supported resource type is `actors`: +`kubectl ate logs` requires a resource-type subcommand. The only supported resource type is `actor`: ```bash # Print the logs an actor has produced on its current worker. diff --git a/cmd/kubectl-ate/internal/cmd/actor.go b/cmd/kubectl-ate/internal/cmd/actor.go index d45db0bdad..20b59a3ad1 100644 --- a/cmd/kubectl-ate/internal/cmd/actor.go +++ b/cmd/kubectl-ate/internal/cmd/actor.go @@ -59,28 +59,13 @@ var ( ) var getActorsCmd = &cobra.Command{ - Use: "actor ", + Use: "actor [actor-name ...]", Aliases: []string{"actors"}, Short: "List all actors or get one or more actors", + Args: cobra.ArbitraryArgs, RunE: func(cmd *cobra.Command, args []string) error { - if len(args) > 0 { - // An actor is addressed by (atespace, name), so the atespace is - // mandatory and "all atespaces" is meaningless here. - if getActorAllAtespacesFlag { - return fmt.Errorf("-A/--all-atespaces cannot be used when getting actors; pass --atespace") - } - if getActorAtespaceFlag == "" { - return fmt.Errorf("--atespace is required when getting actors") - } - } else { - // Listing requires exactly one of --atespace (one atespace) or -A (all - // atespaces). There is no default atespace to fall back on. - if getActorAllAtespacesFlag && getActorAtespaceFlag != "" { - return fmt.Errorf("--atespace and -A/--all-atespaces are mutually exclusive") - } - if !getActorAllAtespacesFlag && getActorAtespaceFlag == "" { - return fmt.Errorf("specify --atespace to list one atespace, or -A/--all-atespaces for all") - } + if err := validateAtespaceFlags("actors", args, getActorAtespaceFlag, getActorAllAtespacesFlag); err != nil { + return err } ctx := cmd.Context() @@ -131,11 +116,11 @@ var getActorsCmd = &cobra.Command{ var createActorCmd = &cobra.Command{ Use: "actor ", Short: "Create an actor", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace", "template"), RunE: func(cmd *cobra.Command, args []string) error { request, err := buildCreateActorRequest(args[0], createActorAtespaceFlag, createActorTemplateFlag, createActorTagFlag) if err != nil { - return err + return wrapUsageError(err) } ctx := cmd.Context() @@ -157,7 +142,7 @@ var createActorCmd = &cobra.Command{ var deleteActorCmd = &cobra.Command{ Use: "actor ", Short: "Delete an actor", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() c, err := newAPIClient(cmd) @@ -183,7 +168,7 @@ var deleteActorCmd = &cobra.Command{ var pauseActorCmd = &cobra.Command{ Use: "actor ", Short: "Pause an actor", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() apiClient, err := newAPIClient(cmd) @@ -207,7 +192,7 @@ var pauseActorCmd = &cobra.Command{ var resumeActorCmd = &cobra.Command{ Use: "actor ", Short: "Resume an actor", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() apiClient, err := newAPIClient(cmd) @@ -231,7 +216,7 @@ var resumeActorCmd = &cobra.Command{ var suspendActorCmd = &cobra.Command{ Use: "actor ", Short: "Suspend an actor", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() apiClient, err := newAPIClient(cmd) @@ -255,7 +240,7 @@ var suspendActorCmd = &cobra.Command{ var logsActorsCmd = &cobra.Command{ Use: "actor ", Short: "Stream logs for a specific actor", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: runLogsActor, } @@ -563,7 +548,7 @@ func runLogsActor(cmd *cobra.Command, args []string) error { actorRef := resources.ActorRef{Atespace: logsActorAtespaceFlag, Name: args[0]} filter, err := newLogLineFilter(actorRef, logsActorSourceFlag, logsActorContainerFlag) if err != nil { - return err + return wrapUsageError(err) } apiClient, err := newAPIClient(cmd) @@ -600,7 +585,7 @@ func runLogsActor(cmd *cobra.Command, args []string) error { // logSource names a class of lines in an actor's log stream, selected with // --source. Container output carries the container's name; the lifecycle -// events ateom emits around a start, checkpoint, restore, or termination +// events worker emits around a start, checkpoint, restore, or termination // carry none, which is what tells the two apart. type logSource string @@ -748,38 +733,31 @@ func filterAndDisplayLogLine(line string, filter logLineFilter, w, stderr io.Wri } func init() { - getActorsCmd.Flags().StringVarP(&getActorAtespaceFlag, "atespace", "a", "", "Atespace to list/get actors in. Required when getting actors; for listing, use this or -A.") - getActorsCmd.Flags().BoolVarP(&getActorAllAtespacesFlag, "all-atespaces", "A", false, "List actors across all atespaces (listing only; mutually exclusive with --atespace)") + getActorsCmd.Flags().StringVarP(&getActorAtespaceFlag, "atespace", "a", "", "get actors in a specific atespace (mutually exclusive with --all-atespaces)") + getActorsCmd.Flags().BoolVarP(&getActorAllAtespacesFlag, "all-atespaces", "A", false, "get actors across all atespaces (mutually exclusive with --atespace)") getCmd.AddCommand(getActorsCmd) - createActorCmd.Flags().StringVar(&createActorTemplateFlag, "template", "", "The name of the ActorTemplate to derive the actor from, as /, or just to use the actor's own atespace (--atespace)") - _ = createActorCmd.MarkFlagRequired("template") - createActorCmd.Flags().StringVarP(&createActorAtespaceFlag, "atespace", "a", "", "Atespace to create the actor in") - _ = createActorCmd.MarkFlagRequired("atespace") - createActorCmd.Flags().StringVar(&createActorTagFlag, "tag", "", "The name of a Tag to initialize the actor from, as /, or just to use the actor's own atespace (--atespace)") + createActorCmd.Flags().StringVar(&createActorTemplateFlag, "template", "", "actor-template to derive the actor from, as /, or just to use the actor's own atespace (see --atespace)") + createActorCmd.Flags().StringVarP(&createActorAtespaceFlag, "atespace", "a", "", "atespace to create the actor in") + createActorCmd.Flags().StringVar(&createActorTagFlag, "tag", "", "tag to initialize the actor from, as /, or just to use the actor's own atespace (see --atespace)") createCmd.AddCommand(createActorCmd) - deleteActorCmd.Flags().StringVarP(&deleteActorAtespaceFlag, "atespace", "a", "", "Atespace the actor lives in") - _ = deleteActorCmd.MarkFlagRequired("atespace") - deleteActorCmd.Flags().BoolVar(&deleteActorAnyStateFlag, "any-state", false, "Delete the actor from any state") + deleteActorCmd.Flags().StringVarP(&deleteActorAtespaceFlag, "atespace", "a", "", "the actor's atespace") + deleteActorCmd.Flags().BoolVar(&deleteActorAnyStateFlag, "any-state", false, "delete the actor regardless of current state") deleteCmd.AddCommand(deleteActorCmd) - pauseActorCmd.Flags().StringVarP(&pauseActorAtespaceFlag, "atespace", "a", "", "Atespace the actor lives in") - _ = pauseActorCmd.MarkFlagRequired("atespace") + pauseActorCmd.Flags().StringVarP(&pauseActorAtespaceFlag, "atespace", "a", "", "the actor's atespace") pauseCmd.AddCommand(pauseActorCmd) - resumeActorCmd.Flags().StringVarP(&resumeActorAtespaceFlag, "atespace", "a", "", "Atespace the actor lives in") - _ = resumeActorCmd.MarkFlagRequired("atespace") + resumeActorCmd.Flags().StringVarP(&resumeActorAtespaceFlag, "atespace", "a", "", "the actor's atespace") resumeCmd.AddCommand(resumeActorCmd) - suspendActorCmd.Flags().StringVarP(&suspendActorAtespaceFlag, "atespace", "a", "", "Atespace the actor lives in") - _ = suspendActorCmd.MarkFlagRequired("atespace") + suspendActorCmd.Flags().StringVarP(&suspendActorAtespaceFlag, "atespace", "a", "", "the actor's atespace") suspendCmd.AddCommand(suspendActorCmd) - logsActorsCmd.Flags().BoolVarP(&logsActorFollowFlag, "follow", "f", false, "Specify if the logs should be streamed.") - logsActorsCmd.Flags().StringVarP(&logsActorAtespaceFlag, "atespace", "a", "", "Atespace the actor lives in") - _ = logsActorsCmd.MarkFlagRequired("atespace") - logsActorsCmd.Flags().StringVarP(&logsActorContainerFlag, "container", "c", "", "Show only logs from this container; use _pause for runsc diagnostics.") - logsActorsCmd.Flags().StringVar(&logsActorSourceFlag, "source", string(logSourceAll), "Which lines to show: all, containers (container output, no lifecycle events), or lifecycle (only the actor's lifecycle events).") + logsActorsCmd.Flags().BoolVarP(&logsActorFollowFlag, "follow", "f", false, "stream logs continuously") + logsActorsCmd.Flags().StringVarP(&logsActorAtespaceFlag, "atespace", "a", "", "the actor's atespace") + logsActorsCmd.Flags().StringVarP(&logsActorContainerFlag, "container", "c", "", "show only logs from a named container; use _pause for runsc diagnostics") + logsActorsCmd.Flags().StringVar(&logsActorSourceFlag, "source", string(logSourceAll), "which lines to show: all, containers (container output, no lifecycle events), or lifecycle (only the actor's lifecycle events)") logsCmd.AddCommand(logsActorsCmd) } diff --git a/cmd/kubectl-ate/internal/cmd/actor_test.go b/cmd/kubectl-ate/internal/cmd/actor_test.go index a9dfa04523..ce4c31bc2b 100644 --- a/cmd/kubectl-ate/internal/cmd/actor_test.go +++ b/cmd/kubectl-ate/internal/cmd/actor_test.go @@ -217,7 +217,7 @@ func TestFilterAndDisplayLogLine(t *testing.T) { wantOutput: `{"time":"2026-05-16T01:03:38Z","level":"info","logging.googleapis.com/labels":{"app":"my-app"},"msg":"Hello"}`, }, { - // ateom drops these at the producer; the CLI strips the whole reserved + // worker drops these at the producer; the CLI strips the whole reserved // namespace too, so a label that reached the stream some other way is // never printed as platform attribution. name: "matching actor, label in substrate's reserved namespace is stripped", diff --git a/cmd/kubectl-ate/internal/cmd/actortemplate.go b/cmd/kubectl-ate/internal/cmd/actortemplate.go index 2c4205ea0f..a27d48fa47 100644 --- a/cmd/kubectl-ate/internal/cmd/actortemplate.go +++ b/cmd/kubectl-ate/internal/cmd/actortemplate.go @@ -34,29 +34,14 @@ var ( ) var getActorTemplatesCmd = &cobra.Command{ - Use: "actor-template ", + Use: "actor-template [template-name ...]", Aliases: []string{"actor-templates"}, Short: "List all actor templates or get one or more actor templates", Long: "List or get actor templates.", + Args: cobra.ArbitraryArgs, RunE: func(cmd *cobra.Command, args []string) error { - if len(args) > 0 { - // A template is addressed by (atespace, name), so the atespace is - // mandatory and "all atespaces" is meaningless here. - if getActorTemplateAllAtespacesFlag { - return fmt.Errorf("-A/--all-atespaces cannot be used when getting actor templates; pass --atespace") - } - if getActorTemplateAtespaceFlag == "" { - return fmt.Errorf("--atespace is required when getting actor templates") - } - } else { - // Listing requires exactly one of --atespace (one atespace) or -A (all - // atespaces). There is no default atespace to fall back on. - if getActorTemplateAllAtespacesFlag && getActorTemplateAtespaceFlag != "" { - return fmt.Errorf("--atespace and -A/--all-atespaces are mutually exclusive") - } - if !getActorTemplateAllAtespacesFlag && getActorTemplateAtespaceFlag == "" { - return fmt.Errorf("specify --atespace to list one atespace, or -A/--all-atespaces for all") - } + if err := validateAtespaceFlags("actor templates", args, getActorTemplateAtespaceFlag, getActorTemplateAllAtespacesFlag); err != nil { + return err } ctx := cmd.Context() @@ -117,7 +102,7 @@ The template's atespace and name come from the manifest's metadata. The atespace must already exist. Actor templates are immutable: there is no update; delete and recreate to change one.`, - Args: cobra.NoArgs, + Args: noArgs("the template name comes from the manifest passed to -f/--filename", "filename"), RunE: func(cmd *cobra.Command, args []string) error { data, err := readFileOrStdin(cmd.InOrStdin(), createActorTemplateFilenameFlag) if err != nil { @@ -149,7 +134,7 @@ var deleteActorTemplateCmd = &cobra.Command{ Long: `Delete an actor template. The server also deletes the template's golden actor and golden snapshot.`, - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() c, err := newAPIClient(cmd) @@ -197,15 +182,13 @@ func readFileOrStdin(in io.Reader, path string) ([]byte, error) { } func init() { - getActorTemplatesCmd.Flags().StringVarP(&getActorTemplateAtespaceFlag, "atespace", "a", "", "Atespace to list/get actor templates in. Required when getting templates; for listing, use this or -A.") - getActorTemplatesCmd.Flags().BoolVarP(&getActorTemplateAllAtespacesFlag, "all-atespaces", "A", false, "List actor templates across all atespaces (listing only; mutually exclusive with --atespace)") + getActorTemplatesCmd.Flags().StringVarP(&getActorTemplateAtespaceFlag, "atespace", "a", "", "get actor-templates in a specific atespace (mutually exclusive with --all-atespaces)") + getActorTemplatesCmd.Flags().BoolVarP(&getActorTemplateAllAtespacesFlag, "all-atespaces", "A", false, "get actor-templates across all atespaces (mutually exclusive with --atespace)") getCmd.AddCommand(getActorTemplatesCmd) - createActorTemplateCmd.Flags().StringVarP(&createActorTemplateFilenameFlag, "filename", "f", "", "Manifest file holding a single protojson-shaped ActorTemplate document; use - for stdin (required)") - _ = createActorTemplateCmd.MarkFlagRequired("filename") + createActorTemplateCmd.Flags().StringVarP(&createActorTemplateFilenameFlag, "filename", "f", "", "manifest file holding a single JSON or YAML actor-template document, or - for stdin") createCmd.AddCommand(createActorTemplateCmd) - deleteActorTemplateCmd.Flags().StringVarP(&deleteActorTemplateAtespaceFlag, "atespace", "a", "", "Atespace the actor template lives in") - _ = deleteActorTemplateCmd.MarkFlagRequired("atespace") + deleteActorTemplateCmd.Flags().StringVarP(&deleteActorTemplateAtespaceFlag, "atespace", "a", "", "the actor-template's atespace") deleteCmd.AddCommand(deleteActorTemplateCmd) } diff --git a/cmd/kubectl-ate/internal/cmd/admin.go b/cmd/kubectl-ate/internal/cmd/admin.go index af30acbdc6..3c9c0c1faf 100644 --- a/cmd/kubectl-ate/internal/cmd/admin.go +++ b/cmd/kubectl-ate/internal/cmd/admin.go @@ -50,14 +50,10 @@ var adminCmd = &cobra.Command{ var makeCaPoolCmd = &cobra.Command{ Use: "make-ca-pool", Short: "Make a new secret that contains a CA pool to be used by a signing controller", + Args: noArgs("use --name to specify the secret name", "name"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - kc, err := newKubeClient(cmd.ErrOrStderr()) - if err != nil { - return err - } - var keyType localca.KeyType switch makeCaPoolKeyTypeFlag { case "ED25519": @@ -65,7 +61,7 @@ var makeCaPoolCmd = &cobra.Command{ case "ECDSAP256": keyType = localca.KeyTypeECDSAP256 default: - return fmt.Errorf("unknown key type %q", makeCaPoolKeyTypeFlag) + return usageErrorf("unknown key type %q", makeCaPoolKeyTypeFlag) } ca, err := localca.GenerateCA( @@ -77,6 +73,11 @@ var makeCaPoolCmd = &cobra.Command{ return fmt.Errorf("while generating CA: %w", err) } + kc, err := newKubeClient(cmd.ErrOrStderr()) + if err != nil { + return err + } + pool := &localca.ConcretePool{ CAs: []*localca.CA{ca}, ActiveForSigning: makeCaPoolIDFlag, @@ -121,15 +122,16 @@ var makeCaPoolCmd = &cobra.Command{ var makeJwtPoolCmd = &cobra.Command{ Use: "make-jwt-pool", Short: "Make a new secret that contains a JWT authority pool to be used by the actor ID broker", + Args: noArgs("use --name to specify the secret name", "name"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() - kc, err := newKubeClient(cmd.ErrOrStderr()) + secret, keyID, err := newJWTPoolSecret(poolSecretNamespaceFlag, poolSecretNameFlag, jwtAlgFlag, jwtKeyIDFlag) if err != nil { - return err + return wrapUsageError(err) } - secret, keyID, err := newJWTPoolSecret(poolSecretNamespaceFlag, poolSecretNameFlag, jwtAlgFlag, jwtKeyIDFlag) + kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { return err } @@ -147,6 +149,7 @@ var makeJwtPoolCmd = &cobra.Command{ var listJwtKeysCmd = &cobra.Command{ Use: "list-jwt-keys", Short: "List the keys in a JWT authority pool secret", + Args: noArgs("use --name to specify the secret name", "name"), RunE: func(cmd *cobra.Command, args []string) error { kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { @@ -169,16 +172,18 @@ var addJwtKeyCmd = &cobra.Command{ The key is published to relying parties but signs nothing until activate-jwt-key makes it the signing key. Activate it only once relying parties have refetched the key set.`, + Args: noArgs("use --name to specify the secret name", "name"), RunE: func(cmd *cobra.Command, args []string) error { - kc, err := newKubeClient(cmd.ErrOrStderr()) + authority, err := localjwtauthority.GenerateAuthority(jwtAlgFlag, jwtKeyIDFlag) if err != nil { - return err + return wrapUsageError(fmt.Errorf("while generating JWT authority: %w", err)) } - authority, err := localjwtauthority.GenerateAuthority(jwtAlgFlag, jwtKeyIDFlag) + kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { - return fmt.Errorf("while generating JWT authority: %w", err) + return err } + if err := updateJWTPool(cmd.Context(), cmd.ErrOrStderr(), kc.CoreV1().Secrets(poolSecretNamespaceFlag), poolSecretNameFlag, func(pool *localjwtauthority.ConcretePool) error { return pool.AddAuthority(authority) }); err != nil { @@ -193,6 +198,7 @@ the key set.`, var activateJwtKeyCmd = &cobra.Command{ Use: "activate-jwt-key", Short: "Make a key in a JWT authority pool secret the one that signs", + Args: noArgs("use --name and --key-id to specify the secret and key", "name", "key-id"), RunE: func(cmd *cobra.Command, args []string) error { kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { @@ -217,6 +223,7 @@ var removeJwtKeyCmd = &cobra.Command{ Tokens the key signed stop verifying once relying parties refetch the key set, so remove it only after the last of them has expired.`, + Args: noArgs("use --name and --key-id to specify the secret and key", "name", "key-id"), RunE: func(cmd *cobra.Command, args []string) error { kc, err := newKubeClient(cmd.ErrOrStderr()) if err != nil { @@ -330,41 +337,36 @@ func newJWTPoolSecret(namespace, name, algorithm, keyID string) (*corev1.Secret, } func init() { + requireSubcommand(adminCmd) rootCmd.AddCommand(adminCmd) - makeCaPoolCmd.Flags().StringVar(&makeCaPoolIDFlag, "ca-id", "", "The ID of the initial CA in the Pool") - makeCaPoolCmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "Create the secret in this namespace") - makeCaPoolCmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "Create the secret with this name") - makeCaPoolCmd.Flags().StringVar(&makeCaPoolKeyTypeFlag, "key-type", "ED25519", "CA key type. One of [ED25519, ECDSAP256]") - makeCaPoolCmd.Flags().DurationVar(&makeCaPoolValidityFlag, "validity", 365*24*time.Hour, "How long the CA certificate is valid for") - _ = makeCaPoolCmd.MarkFlagRequired("name") + makeCaPoolCmd.Flags().StringVar(&makeCaPoolIDFlag, "ca-id", "", "ID of the initial CA in the pool") + makeCaPoolCmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "namespace of the pool secret") + makeCaPoolCmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "name of the pool secret") + makeCaPoolCmd.Flags().StringVar(&makeCaPoolKeyTypeFlag, "key-type", "ED25519", "CA key type: ED25519 or ECDSAP256") + makeCaPoolCmd.Flags().DurationVar(&makeCaPoolValidityFlag, "validity", 365*24*time.Hour, "how long the CA certificate is valid for") adminCmd.AddCommand(makeCaPoolCmd) - makeJwtPoolCmd.Flags().StringVar(&jwtAlgFlag, "alg", "ES256", "Signing algorithm of the initial key. One of [ES256, RS256]; RS256 keys are 4096-bit RSA") - makeJwtPoolCmd.Flags().StringVar(&jwtKeyIDFlag, "key-id", "", "The ID of the initial JWT signing key in the pool. Defaults to the base64url SHA-256 of the key's PKIX encoding") - makeJwtPoolCmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "Create the secret in this namespace") - makeJwtPoolCmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "Create the secret with this name") - _ = makeJwtPoolCmd.MarkFlagRequired("name") + makeJwtPoolCmd.Flags().StringVar(&jwtAlgFlag, "alg", "ES256", "signing algorithm of the initial key: ES256 or RS256 (4096-bit RSA)") + makeJwtPoolCmd.Flags().StringVar(&jwtKeyIDFlag, "key-id", "", "ID of the initial JWT signing key in the pool (defaults to the base64url SHA-256 of the key's PKIX encoding)") + makeJwtPoolCmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "namespace of the pool secret") + makeJwtPoolCmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "name of the pool secret") adminCmd.AddCommand(makeJwtPoolCmd) - listJwtKeysCmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "The namespace of the pool secret") - listJwtKeysCmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "The name of the pool secret") - _ = listJwtKeysCmd.MarkFlagRequired("name") + listJwtKeysCmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "namespace of the pool secret") + listJwtKeysCmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "name of the pool secret") adminCmd.AddCommand(listJwtKeysCmd) - addJwtKeyCmd.Flags().StringVar(&jwtAlgFlag, "alg", "ES256", "Signing algorithm of the new key. One of [ES256, RS256]; RS256 keys are 4096-bit RSA") - addJwtKeyCmd.Flags().StringVar(&jwtKeyIDFlag, "key-id", "", "The ID of the new key. Defaults to the base64url SHA-256 of the key's PKIX encoding") - addJwtKeyCmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "The namespace of the pool secret") - addJwtKeyCmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "The name of the pool secret") - _ = addJwtKeyCmd.MarkFlagRequired("name") + addJwtKeyCmd.Flags().StringVar(&jwtAlgFlag, "alg", "ES256", "signing algorithm of the new key: ES256 or RS256 (4096-bit RSA)") + addJwtKeyCmd.Flags().StringVar(&jwtKeyIDFlag, "key-id", "", "ID of the new key (defaults to the base64url SHA-256 of the key's PKIX encoding)") + addJwtKeyCmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "namespace of the pool secret") + addJwtKeyCmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "name of the pool secret") adminCmd.AddCommand(addJwtKeyCmd) for _, cmd := range []*cobra.Command{activateJwtKeyCmd, removeJwtKeyCmd} { - cmd.Flags().StringVar(&jwtKeyIDFlag, "key-id", "", "The ID of the key") - cmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "The namespace of the pool secret") - cmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "The name of the pool secret") - _ = cmd.MarkFlagRequired("key-id") - _ = cmd.MarkFlagRequired("name") + cmd.Flags().StringVar(&jwtKeyIDFlag, "key-id", "", "ID of the key") + cmd.Flags().StringVar(&poolSecretNamespaceFlag, "secret-namespace", "default", "namespace of the pool secret") + cmd.Flags().StringVar(&poolSecretNameFlag, "name", "", "name of the pool secret") adminCmd.AddCommand(cmd) } } diff --git a/cmd/kubectl-ate/internal/cmd/atespace.go b/cmd/kubectl-ate/internal/cmd/atespace.go index 0427c02dbf..8d208bdc69 100644 --- a/cmd/kubectl-ate/internal/cmd/atespace.go +++ b/cmd/kubectl-ate/internal/cmd/atespace.go @@ -23,9 +23,10 @@ import ( ) var getAtespacesCmd = &cobra.Command{ - Use: "atespace [name ...]", + Use: "atespace [atespace-name ...]", Aliases: []string{"atespaces"}, Short: "List all atespaces or get one or more atespaces", + Args: cobra.ArbitraryArgs, RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() apiClient, err := newAPIClient(cmd) @@ -70,9 +71,9 @@ var getAtespacesCmd = &cobra.Command{ } var createAtespaceCmd = &cobra.Command{ - Use: "atespace [name]", + Use: "atespace ", Short: "Create an atespace", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg(""), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() apiClient, err := newAPIClient(cmd) @@ -97,9 +98,9 @@ var createAtespaceCmd = &cobra.Command{ } var deleteAtespaceCmd = &cobra.Command{ - Use: "atespace [name]", + Use: "atespace ", Short: "Delete an empty atespace (fails if any actors remain)", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg(""), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() apiClient, err := newAPIClient(cmd) diff --git a/cmd/kubectl-ate/internal/cmd/egress_policy.go b/cmd/kubectl-ate/internal/cmd/egress_policy.go index b80171dcc1..fb06708e8e 100644 --- a/cmd/kubectl-ate/internal/cmd/egress_policy.go +++ b/cmd/kubectl-ate/internal/cmd/egress_policy.go @@ -45,7 +45,7 @@ var getEgressPolicyCmd = &cobra.Command{ Aliases: []string{"egress-policies"}, Short: "Get the egress policy of an actor", // TODO(#1550): accept several actors and print a list document. - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: runGetEgressPolicy, } @@ -57,7 +57,7 @@ var createEgressPolicyCmd = &cobra.Command{ The manifest is a YAML or JSON EgressPolicy, as printed by "kubectl ate get egress-policy -a -o yaml". Its metadata may be omitted.`, - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace", "filename"), RunE: runCreateEgressPolicy, } @@ -70,14 +70,14 @@ The manifest is a YAML or JSON EgressPolicy, as printed by "kubectl ate get egress-policy -a -o yaml". Its metadata.uid and metadata.version are required preconditions; the server rejects the update when they no longer match those of the stored egress policy.`, - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace", "filename"), RunE: runUpdateEgressPolicy, } var deleteEgressPolicyCmd = &cobra.Command{ Use: "egress-policy ", Short: "Delete the egress policy of an actor", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: runDeleteEgressPolicy, } @@ -416,7 +416,7 @@ func runDeleteEgressPolicy(cmd *cobra.Command, args []string) error { version: egressPolicyFlags.version, }) if err != nil { - return err + return wrapUsageError(err) } apiClient, err := newAPIClient(cmd) @@ -435,25 +435,19 @@ func runDeleteEgressPolicy(cmd *cobra.Command, args []string) error { } func init() { - getEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.atespace, "atespace", "a", "", "Atespace the actor lives in (required)") - _ = getEgressPolicyCmd.MarkFlagRequired("atespace") + getEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.atespace, "atespace", "a", "", "the actor's atespace") getCmd.AddCommand(getEgressPolicyCmd) - createEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.atespace, "atespace", "a", "", "Atespace the actor lives in (required)") - createEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.filename, "filename", "f", "", "Manifest file holding one EgressPolicy; use - for stdin (required)") - _ = createEgressPolicyCmd.MarkFlagRequired("atespace") - _ = createEgressPolicyCmd.MarkFlagRequired("filename") + createEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.atespace, "atespace", "a", "", "the actor's atespace") + createEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.filename, "filename", "f", "", "a manifest file holding a single JSON or YAML egress-policy document, or - for stdin") createCmd.AddCommand(createEgressPolicyCmd) - updateEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.atespace, "atespace", "a", "", "Atespace the actor lives in (required)") - updateEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.filename, "filename", "f", "", "Manifest file holding one EgressPolicy; use - for stdin (required)") - _ = updateEgressPolicyCmd.MarkFlagRequired("atespace") - _ = updateEgressPolicyCmd.MarkFlagRequired("filename") + updateEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.atespace, "atespace", "a", "", "the actor's atespace") + updateEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.filename, "filename", "f", "", "a manifest file holding a single JSON or YAML egress-policy document, or - for stdin") updateCmd.AddCommand(updateEgressPolicyCmd) - deleteEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.atespace, "atespace", "a", "", "Atespace the actor lives in (required)") - _ = deleteEgressPolicyCmd.MarkFlagRequired("atespace") - deleteEgressPolicyCmd.Flags().StringVar(&egressPolicyFlags.uid, "uid", "", "Delete only if the policy's metadata.uid matches; from get -o yaml") - deleteEgressPolicyCmd.Flags().Int64Var(&egressPolicyFlags.version, "version", 0, "Delete only if the policy's metadata.version matches; from get -o yaml") + deleteEgressPolicyCmd.Flags().StringVarP(&egressPolicyFlags.atespace, "atespace", "a", "", "the actor's atespace") + deleteEgressPolicyCmd.Flags().StringVar(&egressPolicyFlags.uid, "uid", "", "delete only if the policy's metadata.uid matches; from get -o yaml") + deleteEgressPolicyCmd.Flags().Int64Var(&egressPolicyFlags.version, "version", 0, "delete only if the policy's metadata.version matches; from get -o yaml") deleteCmd.AddCommand(deleteEgressPolicyCmd) } diff --git a/cmd/kubectl-ate/internal/cmd/egress_policy_test.go b/cmd/kubectl-ate/internal/cmd/egress_policy_test.go index 6871aa919a..b740d5724a 100644 --- a/cmd/kubectl-ate/internal/cmd/egress_policy_test.go +++ b/cmd/kubectl-ate/internal/cmd/egress_policy_test.go @@ -27,6 +27,7 @@ import ( "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/google/go-cmp/cmp" + "github.com/spf13/cobra" "google.golang.org/grpc" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" @@ -392,6 +393,15 @@ func TestEgressPolicyManifest_RoundTrip(t *testing.T) { } func TestEgressPolicyCommandArgs(t *testing.T) { + resetFlagsRecursive(rootCmd) + t.Cleanup(func() { resetFlagsRecursive(rootCmd) }) + for _, c := range []*cobra.Command{getEgressPolicyCmd, createEgressPolicyCmd, updateEgressPolicyCmd, deleteEgressPolicyCmd} { + _ = c.Flags().Set("atespace", "team-a") + if c.Flags().Lookup("filename") != nil { + _ = c.Flags().Set("filename", "policy.yaml") + } + } + runCommandArgsTests(t, []commandArgsTest{ {name: "get", command: getEgressPolicyCmd, args: []string{"c1"}}, {name: "get requires actor", command: getEgressPolicyCmd, wantErr: true}, diff --git a/cmd/kubectl-ate/internal/cmd/get_test.go b/cmd/kubectl-ate/internal/cmd/get_test.go index 1b5aac6655..9a9410acda 100644 --- a/cmd/kubectl-ate/internal/cmd/get_test.go +++ b/cmd/kubectl-ate/internal/cmd/get_test.go @@ -35,6 +35,7 @@ func TestGetScopeBeforeConnecting(t *testing.T) { }{ {"actors", getActorsCmd, &getActorAtespaceFlag, &getActorAllAtespacesFlag}, {"actor templates", getActorTemplatesCmd, &getActorTemplateAtespaceFlag, &getActorTemplateAllAtespacesFlag}, + {"tags", getTagsCmd, &getTagAtespaceFlag, &getTagAllAtespacesFlag}, } { t.Run(resource.name, func(t *testing.T) { previousAtespace, previousAll := *resource.atespace, *resource.all @@ -54,7 +55,7 @@ func TestGetScopeBeforeConnecting(t *testing.T) { }{ { name: "get without atespace", args: []string{"counter"}, - wantErr: "--atespace is required when getting", + wantErr: "-a/--atespace is required when getting", }, { name: "get with all atespaces", args: []string{"counter"}, all: true, @@ -66,11 +67,11 @@ func TestGetScopeBeforeConnecting(t *testing.T) { }, { name: "list without scope", - wantErr: "specify --atespace to list one atespace, or -A/--all-atespaces for all", + wantErr: "-a/--atespace or -A/--all-atespaces is required", }, { name: "list with both scopes", atespace: "demo", all: true, - wantErr: "--atespace and -A/--all-atespaces are mutually exclusive", + wantErr: "-a/--atespace and -A/--all-atespaces are mutually exclusive", }, // Valid scopes reach client initialization, where the missing kubeconfig fails. { diff --git a/cmd/kubectl-ate/internal/cmd/revert.go b/cmd/kubectl-ate/internal/cmd/revert.go index f27df15fbe..9375384318 100644 --- a/cmd/kubectl-ate/internal/cmd/revert.go +++ b/cmd/kubectl-ate/internal/cmd/revert.go @@ -24,5 +24,6 @@ var revertCmd = &cobra.Command{ } func init() { + requireSubcommand(revertCmd) rootCmd.AddCommand(revertCmd) } diff --git a/cmd/kubectl-ate/internal/cmd/revert_actor.go b/cmd/kubectl-ate/internal/cmd/revert_actor.go index c419330bf4..d4890c9539 100644 --- a/cmd/kubectl-ate/internal/cmd/revert_actor.go +++ b/cmd/kubectl-ate/internal/cmd/revert_actor.go @@ -32,7 +32,7 @@ var revertActorCmd = &cobra.Command{ The actor's external snapshot is left untouched, so a later resume restores from it. Only crashed, running, or paused actors can be reverted.`, - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() apiClient, err := newAPIClient(cmd) @@ -54,7 +54,6 @@ from it. Only crashed, running, or paused actors can be reverted.`, } func init() { - revertActorCmd.Flags().StringVarP(&revertAtespaceFlag, "atespace", "a", "", "Atespace the actor lives in") - _ = revertActorCmd.MarkFlagRequired("atespace") + revertActorCmd.Flags().StringVarP(&revertAtespaceFlag, "atespace", "a", "", "the actor's atespace") revertCmd.AddCommand(revertActorCmd) } diff --git a/cmd/kubectl-ate/internal/cmd/root.go b/cmd/kubectl-ate/internal/cmd/root.go index 592e466a84..e4d1ea1a07 100644 --- a/cmd/kubectl-ate/internal/cmd/root.go +++ b/cmd/kubectl-ate/internal/cmd/root.go @@ -15,11 +15,15 @@ package cmd import ( + "errors" "fmt" "os" + "strings" "github.com/spf13/cobra" + "google.golang.org/grpc/metadata" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/version" ) @@ -31,37 +35,82 @@ var ( outputFmt string traceEnabled bool verbosity int + usePreview bool ) var rootCmd = &cobra.Command{ - Use: "kubectl-ate", - Short: "A kubectl plugin for managing Agent Substrate environments", - Long: `kubectl ate is a CLI tool to manage Actor and Worker lifecycles in an Agent Substrate.`, - Version: version.String(), - SilenceUsage: true, + Use: "kubectl-ate", + Short: "A kubectl plugin for managing Agent Substrate environments", + Long: `kubectl ate is a CLI tool to manage Actor and Worker lifecycles in an Agent Substrate.`, + Version: version.String(), + SilenceUsage: true, // we will handle it ourselves + SilenceErrors: true, // we will handle it ourselves PersistentPreRunE: func(cmd *cobra.Command, args []string) error { + if err := checkSwallowedFlags(cmd); err != nil { + return err + } if outputFmt != "table" && outputFmt != "json" && outputFmt != "yaml" { - return fmt.Errorf("invalid output format %q. Must be one of: table, json, yaml", outputFmt) + return usageErrorf("invalid output format %q. Must be one of: table, json, yaml", outputFmt) + } + if usePreview { + cmd.SetContext(metadata.AppendToOutgoingContext(cmd.Context(), preview.Header, "true")) } return nil }, } +func executeRoot(root *cobra.Command) error { + executedCmd, err := root.ExecuteC() + if err == nil && executedCmd != nil && isGroupCommand(executedCmd) && len(executedCmd.Flags().Args()) > 0 { + err = unknownSubcommandError(executedCmd, executedCmd.Flags().Args()[0]) + } + if err != nil { + if executedCmd == nil { + executedCmd = root + } + errOut := executedCmd.ErrOrStderr() + fmt.Fprintf(errOut, "Error: %v\n", err) + var uerr *usageError + if errors.As(err, &uerr) { + fmt.Fprintf(errOut, "\n%s", executedCmd.UsageString()) + } + return err + } + return nil +} + func Execute() { - if err := rootCmd.Execute(); err != nil { + if err := executeRoot(rootCmd); err != nil { os.Exit(1) } } func init() { + requireSubcommand(rootCmd) + rootCmd.SetFlagErrorFunc(handleFlagError) + defaultHelpFunc := rootCmd.HelpFunc() + rootCmd.SetHelpFunc(func(c *cobra.Command, a []string) { + if isGroupCommand(c) && len(c.Flags().Args()) > 0 { + return + } + defaultHelpFunc(c, a) + }) + rootCmd.SetUsageTemplate(strings.Replace( + rootCmd.UsageTemplate(), + "{{if .Runnable}}", + "{{if and .Runnable (not .HasAvailableSubCommands)}}", + 1, + )) + // Persistent flags are available to this command and all subcommands - rootCmd.PersistentFlags().StringVar(&kubeconfig, "kubeconfig", "", "Path to the kubeconfig file") - rootCmd.PersistentFlags().StringVar(&k8sContext, "context", "", "The name of the kubeconfig context to use") - rootCmd.PersistentFlags().StringVar(&endpoint, "endpoint", "", "Manual override for the gRPC target (e.g., localhost:8080). If omitted, automatically port-forwards.") - rootCmd.PersistentFlags().StringVar(&tokenFile, "token-file", "", "Path to a bearer token for ate-api authentication, or - to read it from stdin. Defaults to a Kubernetes ServiceAccount token.") - rootCmd.PersistentFlags().StringVarP(&outputFmt, "output", "o", "table", "Output format. One of: table|json|yaml") - rootCmd.PersistentFlags().BoolVar(&traceEnabled, "trace", false, "Enable tracing for the request") - rootCmd.PersistentFlags().CountVarP(&verbosity, "verbose", "v", "Increase output verbosity; may be repeated (e.g. -v, -vv, -vvvvv)") + rootCmd.PersistentFlags().StringVar(&kubeconfig, "kubeconfig", "", "path to the kubeconfig file") + rootCmd.PersistentFlags().StringVar(&k8sContext, "context", "", "kubeconfig context to use") + rootCmd.PersistentFlags().StringVar(&endpoint, "endpoint", "", "manual override for the gRPC target (e.g., localhost:8080); if omitted, automatically port-forwards") + rootCmd.PersistentFlags().StringVar(&tokenFile, "token-file", "", "path to a bearer token for ate-api authentication, or - for stdin (defaults to a Kubernetes ServiceAccount token)") + rootCmd.PersistentFlags().StringVarP(&outputFmt, "output", "o", "table", "output format: table, json, or yaml") + rootCmd.PersistentFlags().BoolVar(&traceEnabled, "trace", false, "enable tracing for the request") + rootCmd.PersistentFlags().CountVarP(&verbosity, "verbose", "v", "output verbosity; may be repeated (e.g. -v, -vv, -vvvvv)") + rootCmd.PersistentFlags().BoolVar(&usePreview, "preview", false, "allow the use of preview features enabled on the server") // Register --version without a -v shorthand so -v stays free for --verbose. rootCmd.Flags().Bool("version", false, "version for "+rootCmd.DisplayName()) diff --git a/cmd/kubectl-ate/internal/cmd/root_test.go b/cmd/kubectl-ate/internal/cmd/root_test.go index 336d1962c6..61f5b86fd3 100644 --- a/cmd/kubectl-ate/internal/cmd/root_test.go +++ b/cmd/kubectl-ate/internal/cmd/root_test.go @@ -16,8 +16,14 @@ package cmd import ( "bytes" + "context" "strings" "testing" + + "github.com/agent-substrate/substrate/internal/preview" + "github.com/google/go-cmp/cmp" + "github.com/spf13/cobra" + "google.golang.org/grpc/metadata" ) func resetRootFlags(t *testing.T) { @@ -54,19 +60,6 @@ func TestVersionFlag(t *testing.T) { if f := rootCmd.Flags().Lookup("version"); f != nil && f.Shorthand != "" { t.Errorf("--version shorthand = %q, want empty", f.Shorthand) } - - resetRootFlags(t) - out.Reset() - errOut.Reset() - rootCmd.SetOut(&out) - rootCmd.SetErr(&errOut) - rootCmd.SetArgs([]string{"-v"}) - if err := rootCmd.Execute(); err != nil { - t.Fatalf("Execute(-v) unexpected error: %v", err) - } - if strings.HasPrefix(out.String(), "kubectl-ate version ") { - t.Errorf("Execute(-v) printed version %q, want help output", out.String()) - } } func TestVerboseFlagCounting(t *testing.T) { @@ -101,3 +94,39 @@ func TestVerboseFlagCounting(t *testing.T) { }) } } + +func TestPreviewFlag(t *testing.T) { + for _, tc := range []struct { + name string + args []string + want []string + }{ + {name: "unset", args: nil, want: nil}, + {name: "set", args: []string{"--preview"}, want: []string{"true"}}, + } { + t.Run(tc.name, func(t *testing.T) { + var got []string + probe := &cobra.Command{ + Use: "probe", + RunE: func(cmd *cobra.Command, _ []string) error { + md, _ := metadata.FromOutgoingContext(cmd.Context()) + got = md.Get(preview.Header) + return nil + }, + } + rootCmd.AddCommand(probe) + t.Cleanup(func() { + rootCmd.RemoveCommand(probe) + usePreview = false + }) + + rootCmd.SetArgs(append([]string{"probe"}, tc.args...)) + if err := rootCmd.ExecuteContext(context.Background()); err != nil { + t.Fatal(err) + } + if !cmp.Equal(got, tc.want) { + t.Errorf("outgoing %s = %q, want %q", preview.Header, got, tc.want) + } + }) + } +} diff --git a/cmd/kubectl-ate/internal/cmd/tag.go b/cmd/kubectl-ate/internal/cmd/tag.go index cd3337320b..9f34843457 100644 --- a/cmd/kubectl-ate/internal/cmd/tag.go +++ b/cmd/kubectl-ate/internal/cmd/tag.go @@ -40,15 +40,10 @@ var getTagsCmd = &cobra.Command{ Use: "tag [tag-name ...]", Aliases: []string{"tags"}, Short: "List or get tags", + Args: cobra.ArbitraryArgs, RunE: func(cmd *cobra.Command, args []string) error { - if getTagAllAtespacesFlag && getTagAtespaceFlag != "" { - return fmt.Errorf("--atespace and -A/--all-atespaces are mutually exclusive") - } - if len(args) > 0 && getTagAtespaceFlag == "" { - return fmt.Errorf("--atespace is required when getting tags") - } - if len(args) == 0 && !getTagAllAtespacesFlag && getTagAtespaceFlag == "" { - return fmt.Errorf("specify --atespace , or -A/--all-atespaces") + if err := validateAtespaceFlags("tags", args, getTagAtespaceFlag, getTagAllAtespacesFlag); err != nil { + return err } ctx := cmd.Context() @@ -98,11 +93,11 @@ var createTagCmd = &cobra.Command{ Long: "Tag the external snapshot a suspended actor holds.\n\n" + "The tag gets its own copy of that snapshot, so suspending the actor\n" + "again or deleting it cannot collect what the tag names.", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace", "actor"), RunE: func(cmd *cobra.Command, args []string) error { scope, err := parseTagScope(createTagScopeFlag) if err != nil { - return err + return wrapUsageError(err) } ctx := cmd.Context() client, err := newAPIClient(cmd) @@ -128,11 +123,11 @@ var createTagCmd = &cobra.Command{ var updateTagCmd = &cobra.Command{ Use: "tag ", Short: "Publish or unpublish a tag", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace", "scope"), RunE: func(cmd *cobra.Command, args []string) error { scope, err := parseTagScope(updateTagScopeFlag) if err != nil { - return err + return wrapUsageError(err) } ctx := cmd.Context() client, err := newAPIClient(cmd) @@ -153,7 +148,7 @@ var updateTagCmd = &cobra.Command{ var deleteTagCmd = &cobra.Command{ Use: "tag ", Short: "Delete a tag", - Args: cobra.ExactArgs(1), + Args: exactlyOneArg("", "atespace"), RunE: func(cmd *cobra.Command, args []string) error { ctx := cmd.Context() client, err := newAPIClient(cmd) @@ -202,24 +197,19 @@ func parseTagScope(value string) (ateapipb.TagScope, error) { } func init() { - getTagsCmd.Flags().StringVarP(&getTagAtespaceFlag, "atespace", "a", "", "Atespace to list/get tags in") - getTagsCmd.Flags().BoolVarP(&getTagAllAtespacesFlag, "all-atespaces", "A", false, "List tags across all atespaces") + getTagsCmd.Flags().StringVarP(&getTagAtespaceFlag, "atespace", "a", "", "get tags in a specific atespace (mutually exclusive with --all-atespaces)") + getTagsCmd.Flags().BoolVarP(&getTagAllAtespacesFlag, "all-atespaces", "A", false, "get tags across all atespaces (mutually exclusive with --atespace)") getCmd.AddCommand(getTagsCmd) - createTagCmd.Flags().StringVarP(&createTagAtespaceFlag, "atespace", "a", "", "Atespace the actor lives in; the tag is created here (required)") - _ = createTagCmd.MarkFlagRequired("atespace") - createTagCmd.Flags().StringVar(&createTagActorFlag, "actor", "", "Name of the suspended actor whose external snapshot to tag (required)") - _ = createTagCmd.MarkFlagRequired("actor") - createTagCmd.Flags().StringVar(&createTagScopeFlag, "scope", "atespace", "Tag scope: atespace or published") + createTagCmd.Flags().StringVarP(&createTagAtespaceFlag, "atespace", "a", "", "the actor's atespace; the tag is created here") + createTagCmd.Flags().StringVar(&createTagActorFlag, "actor", "", "suspended actor whose external snapshot to tag") + createTagCmd.Flags().StringVar(&createTagScopeFlag, "scope", "atespace", "tag scope: atespace or published") createCmd.AddCommand(createTagCmd) - updateTagCmd.Flags().StringVarP(&updateTagAtespaceFlag, "atespace", "a", "", "Atespace owning the tag (required)") - updateTagCmd.Flags().StringVar(&updateTagScopeFlag, "scope", "", "Tag scope: atespace or published (required)") - _ = updateTagCmd.MarkFlagRequired("atespace") - _ = updateTagCmd.MarkFlagRequired("scope") + updateTagCmd.Flags().StringVarP(&updateTagAtespaceFlag, "atespace", "a", "", "the tag's atespace") + updateTagCmd.Flags().StringVar(&updateTagScopeFlag, "scope", "", "tag scope: atespace or published") updateCmd.AddCommand(updateTagCmd) - deleteTagCmd.Flags().StringVarP(&deleteTagAtespaceFlag, "atespace", "a", "", "Atespace owning the tag (required)") - _ = deleteTagCmd.MarkFlagRequired("atespace") + deleteTagCmd.Flags().StringVarP(&deleteTagAtespaceFlag, "atespace", "a", "", "the tag's atespace") deleteCmd.AddCommand(deleteTagCmd) } diff --git a/cmd/kubectl-ate/internal/cmd/validation.go b/cmd/kubectl-ate/internal/cmd/validation.go new file mode 100644 index 0000000000..bc3e0de639 --- /dev/null +++ b/cmd/kubectl-ate/internal/cmd/validation.go @@ -0,0 +1,252 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package cmd + +import ( + "errors" + "fmt" + "strings" + + "github.com/spf13/cobra" + "github.com/spf13/pflag" +) + +// usageError marks an error caused by invalid CLI invocation (bad or missing +// flags, arguments, or subcommands), as opposed to a runtime failure. +// executeRoot prints the error and the offending command's usage to stderr. +type usageError struct { + err error +} + +func (e *usageError) Error() string { return e.err.Error() } +func (e *usageError) Unwrap() error { return e.err } + +func usageErrorf(format string, args ...any) error { + return &usageError{err: fmt.Errorf(format, args...)} +} + +func wrapUsageError(err error) error { + if err == nil { + return nil + } + var u *usageError + if errors.As(err, &u) { + return err + } + return &usageError{err: err} +} + +func formatFlagName(f *pflag.Flag) string { + if f.Shorthand != "" { + return fmt.Sprintf("-%s/--%s", f.Shorthand, f.Name) + } + return fmt.Sprintf("--%s", f.Name) +} + +// handleFlagError translates pflag errors into user-friendly usage errors. +func handleFlagError(cmd *cobra.Command, err error) error { + // When a user invokes a group command with an unknown subcommand and + // additional flags (e.g. "kubectl ate logs my-actor -a space"), Find stops + // at the group command and ParseFlags trips on the flag. Report the + // unknown subcommand rather than the flag. + if isGroupCommand(cmd) { + if args := cmd.Flags().Args(); len(args) > 0 { + return unknownSubcommandError(cmd, args[0]) + } + } + + var valReq *pflag.ValueRequiredError + if errors.As(err, &valReq) { + if f := valReq.GetFlag(); f != nil { + return usageErrorf("flag %q requires a value", formatFlagName(f)) + } + } + + var notExist *pflag.NotExistError + if errors.As(err, ¬Exist) { + return usageErrorf("unknown flag %q", specifiedNotExistFlag(notExist)) + } + + return wrapUsageError(err) +} + +func specifiedNotExistFlag(e *pflag.NotExistError) string { + if e.GetSpecifiedShortnames() != "" { + return "-" + e.GetSpecifiedName() + } + return "--" + e.GetSpecifiedName() +} + +// checkSwallowedFlags detects when a string flag consumed the next flag token +// as its argument (for example, "kubectl ate create actor my-actor -a --template foo"). +func checkSwallowedFlags(cmd *cobra.Command) error { + var swallowedErr error + cmd.Flags().Visit(func(f *pflag.Flag) { + if swallowedErr != nil || f.Value.Type() != "string" { + return + } + val := f.Value.String() + if val == "-" && (f.Name == "filename" || f.Name == "token-file") { + return + } + if strings.HasPrefix(val, "-") { + swallowedErr = usageErrorf("flag %q requires a value, got flag %q", formatFlagName(f), val) + } + }) + return swallowedErr +} + +func missingRequiredFlags(cmd *cobra.Command, requiredFlags []string) []string { + var missing []string + for _, name := range requiredFlags { + f := cmd.Flags().Lookup(name) + if f == nil { + continue + } + if !f.Changed || strings.TrimSpace(f.Value.String()) == "" { + missing = append(missing, formatFlagName(f)) + } + } + return missing +} + +func formatMissingFlagsClause(missing []string) string { + if len(missing) == 1 { + return "required flag " + missing[0] + } + return "required flags " + strings.Join(missing, ", ") +} + +func joinQuoted(items []string) string { + quoted := make([]string, len(items)) + for i, s := range items { + quoted[i] = fmt.Sprintf("%q", s) + } + return strings.Join(quoted, ", ") +} + +// exactlyOneArg validates that the command received exactly one positional +// argument (named argPlaceholder, e.g. "") and that all +// requiredFlags were set, reporting everything missing in a single error. +func exactlyOneArg(argPlaceholder string, requiredFlags ...string) cobra.PositionalArgs { + return func(cmd *cobra.Command, args []string) error { + if err := checkSwallowedFlags(cmd); err != nil { + return err + } + missingFlags := missingRequiredFlags(cmd, requiredFlags) + if len(args) == 0 { + if len(missingFlags) > 0 { + return usageErrorf("missing required argument %s and %s", argPlaceholder, formatMissingFlagsClause(missingFlags)) + } + return usageErrorf("missing required argument %s", argPlaceholder) + } + if len(args) > 1 { + return usageErrorf("expected 1 %s argument, received %d (%s)", argPlaceholder, len(args), joinQuoted(args)) + } + if len(missingFlags) > 0 { + return usageErrorf("missing %s", formatMissingFlagsClause(missingFlags)) + } + return nil + } +} + +// noArgs validates that the command received no positional arguments and that +// all requiredFlags were set. If extraArgHint is non-empty, it is included in +// the error when an unexpected positional argument is passed. +func noArgs(extraArgHint string, requiredFlags ...string) cobra.PositionalArgs { + return func(cmd *cobra.Command, args []string) error { + if err := checkSwallowedFlags(cmd); err != nil { + return err + } + if len(args) > 0 { + if extraArgHint != "" { + return usageErrorf("unexpected argument %q (%s)", args[0], extraArgHint) + } + return usageErrorf("%q takes no positional arguments (received %s)", cmd.CommandPath(), joinQuoted(args)) + } + if missingFlags := missingRequiredFlags(cmd, requiredFlags); len(missingFlags) > 0 { + return usageErrorf("missing %s", formatMissingFlagsClause(missingFlags)) + } + return nil + } +} + +const isSubcommandAnnotation = "kubectl-ate/is-a-subcommand" + +func isGroupCommand(cmd *cobra.Command) bool { + return cmd.Annotations[isSubcommandAnnotation] != "" +} + +// requireSubcommand configures a group command (such as root, a verb, or admin) +// so that invoking it with no subcommand or an unknown subcommand returns a +// descriptive usageError instead of exiting 0. +func requireSubcommand(cmd *cobra.Command) { + if cmd.Annotations == nil { + cmd.Annotations = map[string]string{} + } + cmd.Annotations[isSubcommandAnnotation] = "true" + cmd.SuggestionsMinimumDistance = 1 + cmd.Args = func(c *cobra.Command, args []string) error { + if len(args) == 0 { + return usageErrorf("missing required subcommand") + } + return unknownSubcommandError(c, args[0]) + } + cmd.RunE = func(c *cobra.Command, args []string) error { + return usageErrorf("missing required subcommand") + } +} + +func unknownSubcommandError(cmd *cobra.Command, arg string) error { + if cmd.SuggestionsMinimumDistance <= 0 { + cmd.SuggestionsMinimumDistance = 1 + } + if suggestions := cmd.SuggestionsFor(arg); len(suggestions) > 0 { + return usageErrorf("unknown subcommand %q (did you mean %q?)", arg, suggestions[0]) + } + var available []*cobra.Command + for _, sub := range cmd.Commands() { + if sub.IsAvailableCommand() && sub.Name() != "help" && sub.Name() != "completion" { + available = append(available, sub) + } + } + if len(available) == 1 { + return usageErrorf("unknown subcommand %q (did you mean \"%s %s\"?)", + arg, available[0].Name(), arg) + } + return usageErrorf("unknown subcommand %q", arg) +} + +// validateAtespaceFlags checks the --atespace and -A/--all-atespaces flags +// for atespace-scoped "get" commands (actors, actor-templates, tags) before +// connecting to the control plane. +func validateAtespaceFlags(resourcePlural string, args []string, atespace string, allAtespaces bool) error { + if len(args) > 0 { + if allAtespaces { + return usageErrorf("-A/--all-atespaces cannot be used when getting %s by name", resourcePlural) + } + if atespace == "" { + return usageErrorf("-a/--atespace is required when getting %s by name", resourcePlural) + } + return nil + } + if allAtespaces && atespace != "" { + return usageErrorf("-a/--atespace and -A/--all-atespaces are mutually exclusive") + } + if !allAtespaces && atespace == "" { + return usageErrorf("-a/--atespace or -A/--all-atespaces is required") + } + return nil +} diff --git a/cmd/kubectl-ate/internal/cmd/validation_test.go b/cmd/kubectl-ate/internal/cmd/validation_test.go new file mode 100644 index 0000000000..ec8dcf66f9 --- /dev/null +++ b/cmd/kubectl-ate/internal/cmd/validation_test.go @@ -0,0 +1,508 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package cmd + +import ( + "bytes" + "strings" + "testing" + + "github.com/spf13/cobra" + "github.com/spf13/pflag" +) + +func resetFlagsRecursive(c *cobra.Command) { + resetFlagSet := func(fs *pflag.FlagSet) { + fs.VisitAll(func(f *pflag.Flag) { + f.Changed = false + _ = f.Value.Set(f.DefValue) + }) + } + resetFlagSet(c.LocalFlags()) + resetFlagSet(c.PersistentFlags()) + for _, sub := range c.Commands() { + resetFlagsRecursive(sub) + } +} + +func runRootCLI(t *testing.T, args ...string) (stdout, stderr string, err error) { + t.Helper() + resetFlagsRecursive(rootCmd) + var outBuf, errBuf bytes.Buffer + rootCmd.SetOut(&outBuf) + rootCmd.SetErr(&errBuf) + rootCmd.SetArgs(args) + t.Cleanup(func() { + rootCmd.SetOut(nil) + rootCmd.SetErr(nil) + rootCmd.SetArgs(nil) + resetFlagsRecursive(rootCmd) + }) + err = executeRoot(rootCmd) + return outBuf.String(), errBuf.String(), err +} + +func TestHelpOutputGoesToStdout(t *testing.T) { + tests := []struct { + name string + args []string + wantStdoutSub string + }{ + { + name: "root --help", + args: []string{"--help"}, + wantStdoutSub: "Usage:\n kubectl-ate [command]", + }, + { + name: "verb --help", + args: []string{"get", "--help"}, + wantStdoutSub: "Usage:\n kubectl-ate get [command]", + }, + { + name: "leaf -h", + args: []string{"create", "actor", "-h"}, + wantStdoutSub: "Usage:\n kubectl-ate create actor [flags]", + }, + { + name: "help subcommand", + args: []string{"help", "create", "actor"}, + wantStdoutSub: "Usage:\n kubectl-ate create actor [flags]", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + stdout, stderr, err := runRootCLI(t, tc.args...) + if err != nil { + t.Fatalf("unexpected error: %v (stderr: %s)", err, stderr) + } + if stderr != "" { + t.Errorf("expected empty stderr on --help, got:\n%s", stderr) + } + if !strings.Contains(stdout, tc.wantStdoutSub) { + t.Errorf("stdout = %q, want it to contain %q", stdout, tc.wantStdoutSub) + } + }) + } +} + +func TestCLIUsageErrorsGoToStderrWithUsage(t *testing.T) { + tests := []struct { + name string + args []string + wantErrorLine string + wantUsageLine string + }{ + // Flag value / parsing errors + { + name: "get actor -a missing argument", + args: []string{"get", "actor", "-a"}, + wantErrorLine: `Error: flag "-a/--atespace" requires a value`, + wantUsageLine: "kubectl-ate get actor [actor-name ...] [flags]", + }, + { + name: "get actor --atespace missing argument", + args: []string{"get", "actor", "--atespace"}, + wantErrorLine: `Error: flag "-a/--atespace" requires a value`, + wantUsageLine: "kubectl-ate get actor [actor-name ...] [flags]", + }, + { + name: "create actor -a missing argument", + args: []string{"create", "actor", "-a"}, + wantErrorLine: `Error: flag "-a/--atespace" requires a value`, + wantUsageLine: "kubectl-ate create actor [flags]", + }, + { + name: "create actor-template -f missing argument", + args: []string{"create", "actor-template", "-f"}, + wantErrorLine: `Error: flag "-f/--filename" requires a value`, + wantUsageLine: "kubectl-ate create actor-template -f [flags]", + }, + { + name: "global -o missing argument", + args: []string{"get", "atespaces", "-o"}, + wantErrorLine: `Error: flag "-o/--output" requires a value`, + wantUsageLine: "kubectl-ate get atespace [atespace-name ...] [flags]", + }, + { + name: "swallowed flag after -a", + args: []string{"create", "actor", "my-actor", "-a", "--template", "counter"}, + wantErrorLine: `Error: flag "-a/--atespace" requires a value, got flag "--template"`, + wantUsageLine: "kubectl-ate create actor [flags]", + }, + { + name: "invalid output format", + args: []string{"get", "atespaces", "-o", "xml"}, + wantErrorLine: `Error: invalid output format "xml". Must be one of: table, json, yaml`, + wantUsageLine: "kubectl-ate get atespace [atespace-name ...] [flags]", + }, + { + name: "unknown short flag on leaf", + args: []string{"get", "atespaces", "-x"}, + wantErrorLine: `Error: unknown flag "-x"`, + wantUsageLine: "kubectl-ate get atespace [atespace-name ...] [flags]", + }, + { + name: "unknown long flag on leaf", + args: []string{"get", "atespaces", "--bogus"}, + wantErrorLine: `Error: unknown flag "--bogus"`, + wantUsageLine: "kubectl-ate get atespace [atespace-name ...] [flags]", + }, + + // Combined positional-arg and required-flag errors + { + name: "create actor with only -a reports missing arg and --template", + args: []string{"create", "actor", "-a", "my-atespace"}, + wantErrorLine: "Error: missing required argument and required flag --template", + wantUsageLine: "kubectl-ate create actor [flags]", + }, + { + name: "create actor with nothing reports missing arg and both required flags", + args: []string{"create", "actor"}, + wantErrorLine: "Error: missing required argument and required flags -a/--atespace, --template", + wantUsageLine: "kubectl-ate create actor [flags]", + }, + { + name: "create actor with name only reports both missing flags", + args: []string{"create", "actor", "my-actor"}, + wantErrorLine: "Error: missing required flags -a/--atespace, --template", + wantUsageLine: "kubectl-ate create actor [flags]", + }, + { + name: "create actor with name and -a reports missing --template", + args: []string{"create", "actor", "my-actor", "-a", "my-atespace"}, + wantErrorLine: "Error: missing required flag --template", + wantUsageLine: "kubectl-ate create actor [flags]", + }, + { + name: "create actor with both flags but no name reports missing ", + args: []string{"create", "actor", "-a", "my-atespace", "--template", "counter"}, + wantErrorLine: "Error: missing required argument ", + wantUsageLine: "kubectl-ate create actor [flags]", + }, + { + name: "create actor with too many args", + args: []string{"create", "actor", "a1", "a2", "-a", "my-atespace", "--template", "counter"}, + wantErrorLine: `Error: expected 1 argument, received 2 ("a1", "a2")`, + wantUsageLine: "kubectl-ate create actor [flags]", + }, + { + name: "delete actor with nothing", + args: []string{"delete", "actor"}, + wantErrorLine: "Error: missing required argument and required flag -a/--atespace", + wantUsageLine: "kubectl-ate delete actor [flags]", + }, + { + name: "delete actor missing -a", + args: []string{"delete", "actor", "my-actor"}, + wantErrorLine: "Error: missing required flag -a/--atespace", + wantUsageLine: "kubectl-ate delete actor [flags]", + }, + { + name: "revert actor with nothing", + args: []string{"revert", "actor"}, + wantErrorLine: "Error: missing required argument and required flag -a/--atespace", + wantUsageLine: "kubectl-ate revert actor [flags]", + }, + { + name: "logs actor with nothing", + args: []string{"logs", "actor"}, + wantErrorLine: "Error: missing required argument and required flag -a/--atespace", + wantUsageLine: "kubectl-ate logs actor [flags]", + }, + { + name: "create actor-template missing -f", + args: []string{"create", "actor-template"}, + wantErrorLine: "Error: missing required flag -f/--filename", + wantUsageLine: "kubectl-ate create actor-template -f [flags]", + }, + { + name: "create actor-template unexpected positional arg", + args: []string{"create", "actor-template", "my-tpl", "-f", "tpl.yaml"}, + wantErrorLine: `Error: unexpected argument "my-tpl" (the template name comes from the manifest passed to -f/--filename)`, + wantUsageLine: "kubectl-ate create actor-template -f [flags]", + }, + { + name: "delete actor-template with nothing", + args: []string{"delete", "actor-template"}, + wantErrorLine: "Error: missing required argument and required flag -a/--atespace", + wantUsageLine: "kubectl-ate delete actor-template [flags]", + }, + { + name: "create atespace missing name", + args: []string{"create", "atespace"}, + wantErrorLine: "Error: missing required argument ", + wantUsageLine: "kubectl-ate create atespace [flags]", + }, + { + name: "get egress-policy missing all", + args: []string{"get", "egress-policy"}, + wantErrorLine: "Error: missing required argument and required flag -a/--atespace", + wantUsageLine: "kubectl-ate get egress-policy [flags]", + }, + { + name: "get egress-policy rejects multiple args", + args: []string{"get", "egress-policy", "c1", "c2", "-a", "team-a"}, + wantErrorLine: `Error: expected 1 argument, received 2 ("c1", "c2")`, + wantUsageLine: "kubectl-ate get egress-policy [flags]", + }, + { + name: "create egress-policy missing all", + args: []string{"create", "egress-policy"}, + wantErrorLine: "Error: missing required argument and required flags -a/--atespace, -f/--filename", + wantUsageLine: "kubectl-ate create egress-policy -f [flags]", + }, + { + name: "create egress-policy rejects multiple args", + args: []string{"create", "egress-policy", "c1", "c2", "-a", "team-a", "-f", "policy.yaml"}, + wantErrorLine: `Error: expected 1 argument, received 2 ("c1", "c2")`, + wantUsageLine: "kubectl-ate create egress-policy -f [flags]", + }, + { + name: "update egress-policy missing all", + args: []string{"update", "egress-policy"}, + wantErrorLine: "Error: missing required argument and required flags -a/--atespace, -f/--filename", + wantUsageLine: "kubectl-ate update egress-policy -f [flags]", + }, + { + name: "update egress-policy rejects multiple args", + args: []string{"update", "egress-policy", "c1", "c2", "-a", "team-a", "-f", "policy.yaml"}, + wantErrorLine: `Error: expected 1 argument, received 2 ("c1", "c2")`, + wantUsageLine: "kubectl-ate update egress-policy -f [flags]", + }, + { + name: "delete egress-policy missing all", + args: []string{"delete", "egress-policy"}, + wantErrorLine: "Error: missing required argument and required flag -a/--atespace", + wantUsageLine: "kubectl-ate delete egress-policy [flags]", + }, + { + name: "delete egress-policy rejects multiple args", + args: []string{"delete", "egress-policy", "c1", "c2", "-a", "team-a"}, + wantErrorLine: `Error: expected 1 argument, received 2 ("c1", "c2")`, + wantUsageLine: "kubectl-ate delete egress-policy [flags]", + }, + { + name: "create tag missing all", + args: []string{"create", "tag"}, + wantErrorLine: "Error: missing required argument and required flags -a/--atespace, --actor", + wantUsageLine: "kubectl-ate create tag [flags]", + }, + { + name: "update tag missing scope", + args: []string{"update", "tag", "t1", "-a", "space-1"}, + wantErrorLine: "Error: missing required flag --scope", + wantUsageLine: "kubectl-ate update tag [flags]", + }, + { + name: "top workers unexpected positional arg", + args: []string{"top", "workers", "w1"}, + wantErrorLine: `Error: "kubectl-ate top worker" takes no positional arguments (received "w1")`, + wantUsageLine: "kubectl-ate top worker [flags]", + }, + { + name: "admin make-ca-pool missing --name", + args: []string{"admin", "make-ca-pool"}, + wantErrorLine: "Error: missing required flag --name", + wantUsageLine: "kubectl-ate admin make-ca-pool [flags]", + }, + { + name: "admin make-ca-pool unexpected positional arg", + args: []string{"admin", "make-ca-pool", "my-secret", "--name", "my-secret"}, + wantErrorLine: `Error: unexpected argument "my-secret" (use --name to specify the secret name)`, + wantUsageLine: "kubectl-ate admin make-ca-pool [flags]", + }, + { + name: "admin activate-jwt-key missing both required flags", + args: []string{"admin", "activate-jwt-key"}, + wantErrorLine: "Error: missing required flags --name, --key-id", + wantUsageLine: "kubectl-ate admin activate-jwt-key [flags]", + }, + + // Pre-connection flag validation (runs without dialing any cluster) + { + name: "get actor without -a or -A", + args: []string{"get", "actor"}, + wantErrorLine: "Error: -a/--atespace or -A/--all-atespaces is required", + wantUsageLine: "kubectl-ate get actor [actor-name ...] [flags]", + }, + { + name: "get actor with both -a and -A", + args: []string{"get", "actor", "-a", "space-1", "-A"}, + wantErrorLine: "Error: -a/--atespace and -A/--all-atespaces are mutually exclusive", + wantUsageLine: "kubectl-ate get actor [actor-name ...] [flags]", + }, + { + name: "get actor by name without -a", + args: []string{"get", "actor", "my-actor"}, + wantErrorLine: "Error: -a/--atespace is required when getting actors by name", + wantUsageLine: "kubectl-ate get actor [actor-name ...] [flags]", + }, + { + name: "get actor by name with -A", + args: []string{"get", "actor", "my-actor", "-A"}, + wantErrorLine: "Error: -A/--all-atespaces cannot be used when getting actors by name", + wantUsageLine: "kubectl-ate get actor [actor-name ...] [flags]", + }, + { + name: "get actor-template without -a or -A", + args: []string{"get", "actor-template"}, + wantErrorLine: "Error: -a/--atespace or -A/--all-atespaces is required", + wantUsageLine: "kubectl-ate get actor-template [template-name ...] [flags]", + }, + { + name: "get tag without -a or -A", + args: []string{"get", "tag"}, + wantErrorLine: "Error: -a/--atespace or -A/--all-atespaces is required", + wantUsageLine: "kubectl-ate get tag [tag-name ...] [flags]", + }, + { + name: "get tag by name with -A", + args: []string{"get", "tag", "t1", "-A"}, + wantErrorLine: "Error: -A/--all-atespaces cannot be used when getting tags by name", + wantUsageLine: "kubectl-ate get tag [tag-name ...] [flags]", + }, + { + name: "get workers by name with filter flag", + args: []string{"get", "workers", "w1", "-n", "default"}, + wantErrorLine: "Error: -n/--namespace cannot be used when getting workers by name", + wantUsageLine: "kubectl-ate get worker [worker-name ...] [flags]", + }, + { + name: "get workers with invalid label selector", + args: []string{"get", "workers", "-l", "===bad"}, + wantErrorLine: `Error: invalid label selector "===bad"`, + wantUsageLine: "kubectl-ate get worker [worker-name ...] [flags]", + }, + { + name: "top workers with invalid label selector", + args: []string{"top", "workers", "-l", "===bad"}, + wantErrorLine: `Error: invalid label selector "===bad"`, + wantUsageLine: "kubectl-ate top worker [flags]", + }, + { + name: "logs actor with invalid --source", + args: []string{"logs", "actor", "my-actor", "-a", "space-1", "--source=bogus"}, + wantErrorLine: `Error: invalid --source "bogus"`, + wantUsageLine: "kubectl-ate logs actor [flags]", + }, + { + name: "create tag with invalid --scope", + args: []string{"create", "tag", "t1", "-a", "space-1", "--actor", "a1", "--scope=bogus"}, + wantErrorLine: `Error: invalid scope "bogus"; must be atespace or published`, + wantUsageLine: "kubectl-ate create tag [flags]", + }, + { + name: "delete egress-policy with invalid --version", + args: []string{"delete", "egress-policy", "a1", "-a", "space-1", "--version=0"}, + wantErrorLine: "Error: --version must be at least 1, got 0", + wantUsageLine: "kubectl-ate delete egress-policy [flags]", + }, + { + name: "admin make-ca-pool with invalid --key-type", + args: []string{"admin", "make-ca-pool", "--name", "s", "--key-type", "RSA"}, + wantErrorLine: `Error: unknown key type "RSA"`, + wantUsageLine: "kubectl-ate admin make-ca-pool [flags]", + }, + + // Bare verb and unknown subcommand errors + { + name: "bare root command", + args: []string{}, + wantErrorLine: "Error: missing required subcommand", + wantUsageLine: "kubectl-ate [command]", + }, + { + name: "unknown root subcommand", + args: []string{"bogus"}, + wantErrorLine: `Error: unknown subcommand "bogus"`, + wantUsageLine: "kubectl-ate [command]", + }, + { + name: "bare get command", + args: []string{"get"}, + wantErrorLine: "Error: missing required subcommand", + wantUsageLine: "kubectl-ate get [command]", + }, + { + name: "bare create command", + args: []string{"create"}, + wantErrorLine: "Error: missing required subcommand", + wantUsageLine: "kubectl-ate create [command]", + }, + { + name: "create plural actors suggests singular actor", + args: []string{"create", "actors", "my-actor", "-a", "space-1", "--template", "counter"}, + wantErrorLine: `Error: unknown subcommand "actors" (did you mean "actor"?)`, + wantUsageLine: "kubectl-ate create [command]", + }, + { + name: "logs without actor subcommand suggests logs actor ", + args: []string{"logs", "my-actor", "-a", "space-1"}, + wantErrorLine: `Error: unknown subcommand "my-actor" (did you mean "actor my-actor"?)`, + wantUsageLine: "kubectl-ate logs [command]", + }, + { + name: "logs with unknown subcommand and -h still reports unknown subcommand", + args: []string{"logs", "asdasd", "-h"}, + wantErrorLine: `Error: unknown subcommand "asdasd" (did you mean "actor asdasd"?)`, + wantUsageLine: "kubectl-ate logs [command]", + }, + { + name: "flag passed to verb without resource type", + args: []string{"get", "-a", "space-1"}, + wantErrorLine: `Error: unknown flag "-a"`, + wantUsageLine: "kubectl-ate get [command]", + }, + } + + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + stdout, stderr, err := runRootCLI(t, tc.args...) + if err == nil { + t.Fatalf("expected error for args %v, got nil", tc.args) + } + if stdout != "" { + t.Errorf("expected empty stdout on error, got:\n%s", stdout) + } + firstLine, _, _ := strings.Cut(stderr, "\n") + if !strings.Contains(firstLine, tc.wantErrorLine) { + t.Errorf("first stderr line = %q, want it to contain %q\nfull stderr:\n%s", firstLine, tc.wantErrorLine, stderr) + } + if !strings.Contains(stderr, "\n\nUsage:\n") { + t.Errorf("stderr missing Usage section:\n%s", stderr) + } + if !strings.Contains(stderr, tc.wantUsageLine) { + t.Errorf("stderr = %q, want usage line %q", stderr, tc.wantUsageLine) + } + }) + } +} + +func TestRuntimeErrorsDoNotPrintUsage(t *testing.T) { + stdout, stderr, err := runRootCLI(t, "create", "actor-template", "-f", "/nonexistent/path/to/manifest.yaml") + if err == nil { + t.Fatal("expected error for nonexistent manifest file, got nil") + } + if stdout != "" { + t.Errorf("expected empty stdout, got:\n%s", stdout) + } + if !strings.HasPrefix(stderr, "Error: ") { + t.Errorf("expected stderr to start with 'Error: ', got:\n%s", stderr) + } + if strings.Contains(stderr, "Usage:") { + t.Errorf("expected runtime error not to print Usage, got:\n%s", stderr) + } +} diff --git a/cmd/kubectl-ate/internal/cmd/verbose_test.go b/cmd/kubectl-ate/internal/cmd/verbose_test.go index a39d87f38b..c036ae93fd 100644 --- a/cmd/kubectl-ate/internal/cmd/verbose_test.go +++ b/cmd/kubectl-ate/internal/cmd/verbose_test.go @@ -278,7 +278,7 @@ func TestTopWorkersRunner_VerboseSurfacesMetricsIssues(t *testing.T) { } got := errOut.String() for _, want := range []string{ - `Container "ateom" not found in pod metrics for ns-1/pod-1; summing all 1 container(s)`, + `Container "worker" not found in pod metrics for ns-1/pod-1; summing all 1 container(s)`, "No pod metrics returned for worker pod ns-1/pod-2", } { if !strings.Contains(got, want) { diff --git a/cmd/kubectl-ate/internal/cmd/verbs.go b/cmd/kubectl-ate/internal/cmd/verbs.go index f98f86fbc6..87d332d442 100644 --- a/cmd/kubectl-ate/internal/cmd/verbs.go +++ b/cmd/kubectl-ate/internal/cmd/verbs.go @@ -33,5 +33,8 @@ var ( ) func init() { - rootCmd.AddCommand(getCmd, createCmd, updateCmd, deleteCmd, pauseCmd, resumeCmd, suspendCmd, logsCmd, topCmd) + for _, c := range []*cobra.Command{getCmd, createCmd, updateCmd, deleteCmd, pauseCmd, resumeCmd, suspendCmd, logsCmd, topCmd} { + requireSubcommand(c) + rootCmd.AddCommand(c) + } } diff --git a/cmd/kubectl-ate/internal/cmd/worker.go b/cmd/kubectl-ate/internal/cmd/worker.go index e06d82d5b4..cd499985dc 100644 --- a/cmd/kubectl-ate/internal/cmd/worker.go +++ b/cmd/kubectl-ate/internal/cmd/worker.go @@ -44,9 +44,10 @@ var ( ) var getWorkersCmd = &cobra.Command{ - Use: "worker ", + Use: "worker [worker-name ...]", Aliases: []string{"workers"}, Short: "List all workers or get one or more workers", + Args: cobra.ArbitraryArgs, RunE: runGetWorkers, } @@ -54,7 +55,7 @@ var topWorkersCmd = &cobra.Command{ Use: "worker", Aliases: []string{"workers"}, Short: "Display resource (CPU/Memory) usage of workers", - Args: cobra.NoArgs, + Args: noArgs(""), RunE: runTopWorkers, } @@ -177,8 +178,8 @@ type GetWorkersRunner struct { func (r *GetWorkersRunner) Run(ctx context.Context) error { if len(r.names) > 0 { - if r.namespace != "" || r.atespace != "" || r.selector != "" || r.sandboxClass != "" { - return fmt.Errorf("filter flags (--namespace, --atespace, --selector, --sandbox-class) cannot be used when getting workers by name") + if err := validateWorkerFlags(r.names, r.namespace, r.atespace, r.selector, r.sandboxClass); err != nil { + return err } workers := make([]*ateapipb.Worker, 0, len(r.names)) for _, name := range r.names { @@ -205,7 +206,32 @@ func (r *GetWorkersRunner) Run(ctx context.Context) error { return printer.PrintWorkersTo(r.out, filtered, r.outputFmt) } +func validateWorkerFlags(names []string, namespace, atespace, selector, sandboxClass string) error { + if len(names) > 0 { + switch { + case namespace != "": + return usageErrorf("-n/--namespace cannot be used when getting workers by name") + case atespace != "": + return usageErrorf("-a/--atespace cannot be used when getting workers by name") + case selector != "": + return usageErrorf("-l/--selector cannot be used when getting workers by name") + case sandboxClass != "": + return usageErrorf("--sandbox-class cannot be used when getting workers by name") + } + } + if selector != "" { + if _, err := labels.Parse(selector); err != nil { + return usageErrorf("invalid label selector %q: %w", selector, err) + } + } + return nil +} + func runGetWorkers(cmd *cobra.Command, args []string) error { + if err := validateWorkerFlags(args, getWorkerNamespaceFlag, getWorkerAtespaceFlag, getWorkerSelectorFlag, getWorkerClassFlag); err != nil { + return err + } + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) @@ -320,7 +346,7 @@ func (r *TopWorkersRunner) Run(ctx context.Context) error { func extractContainerUsage(pm metricsv1beta1.PodMetrics, stderr io.Writer) (string, string) { var cpuQuant, memQuant *resource.Quantity for _, c := range pm.Containers { - if c.Name == "ateom" { + if c.Name == "worker" { cpu := c.Usage[corev1.ResourceCPU] mem := c.Usage[corev1.ResourceMemory] cpuQuant = &cpu @@ -329,7 +355,7 @@ func extractContainerUsage(pm metricsv1beta1.PodMetrics, stderr io.Writer) (stri } } if cpuQuant == nil && len(pm.Containers) > 0 { - vlogf(stderr, vFlow, "Container \"ateom\" not found in pod metrics for %s/%s; summing all %d container(s)", + vlogf(stderr, vFlow, "Container \"worker\" not found in pod metrics for %s/%s; summing all %d container(s)", pm.Namespace, pm.Name, len(pm.Containers)) var totalCPU, totalMem resource.Quantity for _, c := range pm.Containers { @@ -350,6 +376,10 @@ func extractContainerUsage(pm metricsv1beta1.PodMetrics, stderr io.Writer) (stri } func runTopWorkers(cmd *cobra.Command, args []string) error { + if err := validateWorkerFlags(nil, topWorkerNamespaceFlag, topWorkerAtespaceFlag, topWorkerSelectorFlag, topWorkerClassFlag); err != nil { + return err + } + apiClient, err := newAPIClient(cmd) if err != nil { return fmt.Errorf("failed to connect to ate-api-server: %w", err) @@ -381,15 +411,15 @@ func runTopWorkers(cmd *cobra.Command, args []string) error { } func init() { - getWorkersCmd.Flags().StringVarP(&getWorkerNamespaceFlag, "namespace", "n", "", "Scope output to a specific Kubernetes namespace") - getWorkersCmd.Flags().StringVarP(&getWorkerAtespaceFlag, "atespace", "a", "", "Filter worker pods hosting actors in a specific atespace") - getWorkersCmd.Flags().StringVarP(&getWorkerSelectorFlag, "selector", "l", "", "Filter by worker pool labels") - getWorkersCmd.Flags().StringVar(&getWorkerClassFlag, "sandbox-class", "", "Filter by sandbox class (e.g. gvisor, microvm)") + getWorkersCmd.Flags().StringVarP(&getWorkerNamespaceFlag, "namespace", "n", "", "scope output to a specific Kubernetes namespace") + getWorkersCmd.Flags().StringVarP(&getWorkerAtespaceFlag, "atespace", "a", "", "filter worker pods hosting actors in a specific atespace") + getWorkersCmd.Flags().StringVarP(&getWorkerSelectorFlag, "selector", "l", "", "filter by worker pool labels") + getWorkersCmd.Flags().StringVar(&getWorkerClassFlag, "sandbox-class", "", "filter by sandbox class (e.g. gvisor, microvm)") getCmd.AddCommand(getWorkersCmd) - topWorkersCmd.Flags().StringVarP(&topWorkerNamespaceFlag, "namespace", "n", "", "Scope output to a specific Kubernetes namespace") - topWorkersCmd.Flags().StringVarP(&topWorkerAtespaceFlag, "atespace", "a", "", "Filter worker pods hosting actors in a specific atespace") - topWorkersCmd.Flags().StringVarP(&topWorkerSelectorFlag, "selector", "l", "", "Filter by worker pool labels") - topWorkersCmd.Flags().StringVar(&topWorkerClassFlag, "sandbox-class", "", "Filter by sandbox class (e.g. gvisor, microvm)") + topWorkersCmd.Flags().StringVarP(&topWorkerNamespaceFlag, "namespace", "n", "", "scope output to a specific Kubernetes namespace") + topWorkersCmd.Flags().StringVarP(&topWorkerAtespaceFlag, "atespace", "a", "", "filter worker pods hosting actors in a specific atespace") + topWorkersCmd.Flags().StringVarP(&topWorkerSelectorFlag, "selector", "l", "", "filter by worker pool labels") + topWorkersCmd.Flags().StringVar(&topWorkerClassFlag, "sandbox-class", "", "filter by sandbox class (e.g. gvisor, microvm)") topCmd.AddCommand(topWorkersCmd) } diff --git a/cmd/kubectl-ate/internal/cmd/worker_test.go b/cmd/kubectl-ate/internal/cmd/worker_test.go index 8ee2db0f9b..04a33d8bc3 100644 --- a/cmd/kubectl-ate/internal/cmd/worker_test.go +++ b/cmd/kubectl-ate/internal/cmd/worker_test.go @@ -352,7 +352,7 @@ func TestTopWorkersRunner_Success(t *testing.T) { }, Containers: []metricsv1beta1.ContainerMetrics{ { - Name: "ateom", + Name: "worker", Usage: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("342m"), corev1.ResourceMemory: resource.MustParse("412Mi"), @@ -367,7 +367,7 @@ func TestTopWorkersRunner_Success(t *testing.T) { }, Containers: []metricsv1beta1.ContainerMetrics{ { - Name: "ateom", + Name: "worker", Usage: corev1.ResourceList{ corev1.ResourceCPU: resource.MustParse("2m"), corev1.ResourceMemory: resource.MustParse("64Mi"), diff --git a/cmd/objectstore-plugin/main.go b/cmd/objectstore-plugin/main.go index c311666bee..fba93439c2 100644 --- a/cmd/objectstore-plugin/main.go +++ b/cmd/objectstore-plugin/main.go @@ -35,6 +35,7 @@ import ( "github.com/agent-substrate/substrate/internal/objectstore" "github.com/agent-substrate/substrate/internal/objectstoreplugin" "github.com/agent-substrate/substrate/internal/pluginsocket" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/serverboot" "github.com/agent-substrate/substrate/pkg/objectstorage" objectstorev1 "github.com/agent-substrate/substrate/pkg/proto/objectstorepb/v1" @@ -60,10 +61,14 @@ func main() { assetStagingDir := flags.String("asset-staging-dir", "/var/lib/objectstore-plugin/asset-staging", "node mode: directory sandbox assets are downloaded into and verified before they are copied below --root; must not be below --root or readable by the caller") assetStagingCapacity := flags.String("asset-staging-capacity", "8Gi", "node mode: the most sandbox asset downloads in --asset-staging-dir may hold at once, as a Kubernetes quantity; downloads wait for space, and an asset larger than this is refused") timeout := flags.Duration("timeout", 5*time.Second, "healthcheck mode: how long to wait for the plugin to report that it is serving") + previewFlags := flags.StringSlice("preview", nil, "Preview gates to enable.") _ = flags.Parse(os.Args[2:]) ctx := context.Background() serverboot.InitLogger() + if err := preview.Init(*previewFlags...); err != nil { + serverboot.Fatal(ctx, "Invalid --preview", err) + } if *socket == "" { serverboot.Fatal(ctx, "Missing --socket", fmt.Errorf("--socket is required")) } diff --git a/cmd/podcertcontroller/main.go b/cmd/podcertcontroller/main.go index e8c8774583..16922b292b 100644 --- a/cmd/podcertcontroller/main.go +++ b/cmd/podcertcontroller/main.go @@ -40,6 +40,7 @@ import ( "github.com/agent-substrate/substrate/internal/clustertrustbundle" "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/localca" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/version" "github.com/spf13/pflag" "k8s.io/apimachinery/pkg/types" @@ -120,7 +121,8 @@ var ( "Burst queries allowed against the Kubernetes API. 0 keeps the client-go default.", ) - showVersion = pflag.Bool("version", false, "Print version and exit.") + showVersion = pflag.Bool("version", false, "Print version and exit.") + previewFlags = pflag.StringSlice("preview", nil, "Preview gates to enable.") ) func main() { @@ -132,6 +134,10 @@ func main() { return } slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, nil))) + if err := preview.Init(*previewFlags...); err != nil { + slog.ErrorContext(ctx, "Invalid --preview", slog.Any("err", err)) + os.Exit(1) + } slog.InfoContext(ctx, "podcertcontroller starting", slog.String("version", version.Version)) additionalPostgresClients := make([]postgressigner.Client, 0, len(*postgresClients)) diff --git a/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl b/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl index 3989a57ae7..d870abf04a 100644 --- a/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl +++ b/demos/autoscaled-workerpool/autoscaled-workerpool-template.yaml.tmpl @@ -35,6 +35,14 @@ containers: volumeMounts: - name: data mountPath: /home/counter +# Sandbox size. The pool sets no worker limits, so the worker offers the +# node's allocatable and these only need to fit the node. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://${BUCKET_NAME}/ate-demo-autoscaled-workerpool/ diff --git a/demos/autoscaled-workerpool/autoscaled-workerpool.yaml.tmpl b/demos/autoscaled-workerpool/autoscaled-workerpool.yaml.tmpl index 000fd881b5..04aa6672ec 100644 --- a/demos/autoscaled-workerpool/autoscaled-workerpool.yaml.tmpl +++ b/demos/autoscaled-workerpool/autoscaled-workerpool.yaml.tmpl @@ -37,6 +37,6 @@ metadata: workload: counter-autoscaled spec: replicas: 5 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor diff --git a/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl b/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl index 8ff1002b39..69eb72c48f 100644 --- a/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-luna-template.yaml.tmpl @@ -40,6 +40,14 @@ containers: value: "45" - name: ANTHROPIC_API_KEY value: "${ANTHROPIC_API_KEY}" +# Sandbox size: Claude Code on Node uses several hundred MB and spikes while +# running tools. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 2Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ diff --git a/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl b/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl index 13fa97d5af..080202ec43 100644 --- a/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-mars-template.yaml.tmpl @@ -33,6 +33,13 @@ containers: value: "45" - name: ANTHROPIC_API_KEY value: "${ANTHROPIC_API_KEY}" +# Sandbox size: see agent-luna-template.yaml.tmpl. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 2Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ diff --git a/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl b/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl index f220f255c9..7a162bde47 100644 --- a/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl +++ b/demos/claude-code-multiplex/agent-orion-template.yaml.tmpl @@ -33,6 +33,13 @@ containers: value: "45" - name: ANTHROPIC_API_KEY value: "${ANTHROPIC_API_KEY}" +# Sandbox size: see agent-luna-template.yaml.tmpl. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 2Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/claude-multiplex-demo/ diff --git a/demos/claude-code-multiplex/claude-code-multiplex.yaml.tmpl b/demos/claude-code-multiplex/claude-code-multiplex.yaml.tmpl index ef1360fdcc..5267e91017 100644 --- a/demos/claude-code-multiplex/claude-code-multiplex.yaml.tmpl +++ b/demos/claude-code-multiplex/claude-code-multiplex.yaml.tmpl @@ -34,6 +34,6 @@ metadata: workload: claude-multiplex spec: replicas: 2 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor diff --git a/demos/counter/README.md b/demos/counter/README.md index ee31c67d90..ea6ececc19 100644 --- a/demos/counter/README.md +++ b/demos/counter/README.md @@ -32,7 +32,7 @@ To enable validation of reading from an external volume (e.g. `/external-data/test.txt`), run: ```bash -./hack/install-ate.sh --deploy-demo-counter-with-external-volume +./hack/install-ate.sh --deploy-demo-counter-with-external-volume --preview=ExternalVolumes ``` This command will: @@ -140,7 +140,7 @@ prioritizing it, not a fundamental limitation of the design. ## Micro-VM variant The same in-RAM-counter suspend/resume-continuity demo also runs on the micro-VM -sandbox class (`ateom-microvm`: a Kata guest on Cloud Hypervisor), proving that +sandbox class (`ateworker-microvm`: a Kata guest on Cloud Hypervisor), proving that the guest-memory snapshot round-trips just as gVisor's process snapshot does. The micro-VM class is also the one that serves `SNAPSHOT_FIDELITY_ROOTFS`, a snapshot of the volumes and each container's rootfs writes without guest diff --git a/demos/counter/counter-microvm-csi-test-template.yaml b/demos/counter/counter-microvm-csi-test-template.yaml index 436aba8e6a..fd20354aae 100644 --- a/demos/counter/counter-microvm-csi-test-template.yaml +++ b/demos/counter/counter-microvm-csi-test-template.yaml @@ -32,6 +32,14 @@ containers: volumeMounts: - name: data mountPath: /home/counter +# Sandbox size: vCPUs and guest RAM (see internal/sizing). The pool sets no +# worker limits, so these only need to fit the node. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES storageLocation: gs://ate-snapshots/ate-demo-counter-microvm-csi/ diff --git a/demos/counter/counter-microvm-csi-test.yaml b/demos/counter/counter-microvm-csi-test.yaml index 3306226a75..55bfbac1b9 100644 --- a/demos/counter/counter-microvm-csi-test.yaml +++ b/demos/counter/counter-microvm-csi-test.yaml @@ -34,4 +34,4 @@ spec: replicas: 1 sandboxClasses: - name: microvm - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-microvm + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-microvm diff --git a/demos/counter/counter-microvm.yaml.tmpl b/demos/counter/counter-microvm.yaml.tmpl index 08e1048abb..ba7b252886 100644 --- a/demos/counter/counter-microvm.yaml.tmpl +++ b/demos/counter/counter-microvm.yaml.tmpl @@ -43,7 +43,7 @@ spec: - name: microvm configRef: name: microvm - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-microvm + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-microvm template: # Per-worker resources: these size the worker POD and advertise its # scheduling capacity (the per-actor ceiling). The micro-VM sandbox itself is diff --git a/demos/counter/counter.go b/demos/counter/counter.go index 5720f3d1c7..6d4ae6783f 100644 --- a/demos/counter/counter.go +++ b/demos/counter/counter.go @@ -120,7 +120,7 @@ func main() { w.WriteHeader(http.StatusOK) w.Write([]byte(response)) }) - // /readyz is the endpoint the ateom-gvisor wakeup probe polls. It returns + // /readyz is the endpoint the ateworker-gvisor wakeup probe polls. It returns // 200 only once initialization (the random-file write) has completed. // After a checkpoint+restore the atomic flag is part of the snapshot, so // the endpoint returns 200 immediately on resume. diff --git a/demos/counter/counter.yaml.tmpl b/demos/counter/counter.yaml.tmpl index 4cc0ea0516..9919666b35 100644 --- a/demos/counter/counter.yaml.tmpl +++ b/demos/counter/counter.yaml.tmpl @@ -37,7 +37,7 @@ spec: # pool that cannot place every replica is worse than a smaller one -- the # deploy blocks on a rollout that never completes. replicas: 3 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor # No configRef: the pool follows the class default, the newest SandboxConfig # of the class annotated sandboxconfig.ate.dev/is-class-default=true # (gvisor-default as installed), and moves to a new default when one appears. diff --git a/demos/egress/README.md b/demos/egress/README.md index 229f54bead..685cefefd5 100644 --- a/demos/egress/README.md +++ b/demos/egress/README.md @@ -11,7 +11,7 @@ intercepted and carried over mTLS to a gateway that verifies who is making the r ## What it demonstrates ```text - ┌──────────────── ateom worker pod ─────────────────┐ + ┌─────────────────── worker pod ────────────────────┐ │ Actor (gVisor) │ │ GET http://:80/ (plain HTTP) │ │ │ │ diff --git a/demos/egress/egress-microvm-template.yaml.tmpl b/demos/egress/egress-microvm-template.yaml.tmpl index dfcabd1957..ad4400f831 100644 --- a/demos/egress/egress-microvm-template.yaml.tmpl +++ b/demos/egress/egress-microvm-template.yaml.tmpl @@ -14,8 +14,8 @@ # Substrate ActorTemplate resource (protojson-shaped ateapipb.ActorTemplate, # NOT the ate.dev/v1alpha1 CRD) for the micro-VM egress demo. The actor's -# outbound TCP is redirected into atunnel by nftables in the ateom -# (prepareActorEgress), which the micro-VM ateom implements the same way the +# outbound TCP is redirected into atunnel by nftables in the ateworker +# (prepareActorEgress), which the micro-VM ateworker implements the same way the # gVisor one does — this fixture is what proves it end to end. Delivery of the # projected trust bundle into the sandbox also differs by class — a read-only # bind for gVisor, the unified virtio-fs share for the guest here — so the @@ -62,9 +62,8 @@ volumes: names: - egress-mitm.ate.dev path: trust-bundle.pem -# Sandbox size: without these the guest boots at ateom's default size -# (2GiB). ateom applies them to the VM (see internal/sizing). Quantities are -# strings in the proto. +# Sandbox size: ateworker applies these to the VM (see internal/sizing). +# Quantities are strings in the proto. resources: limits: - name: cpu diff --git a/demos/egress/egress-microvm.yaml.tmpl b/demos/egress/egress-microvm.yaml.tmpl index 3facf92508..7f515c9b70 100644 --- a/demos/egress/egress-microvm.yaml.tmpl +++ b/demos/egress/egress-microvm.yaml.tmpl @@ -38,9 +38,9 @@ spec: replicas: 2 sandboxClasses: - name: microvm - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-microvm + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-microvm # No template.resources, unlike the counter demo's micro-VM variant: an - # unlimited ateom container reports no capacity, which the scheduler reads as + # unlimited worker container reports no capacity, which the scheduler reads as # unconstrained, and the pod requests nothing so it stays placeable next to # the counter pool's reservation on CI's single kind node. What actually # bounds the memory here is the ActorTemplate's resources, which size the diff --git a/demos/egress/egress-template.yaml.tmpl b/demos/egress/egress-template.yaml.tmpl index 81988bc7b6..81271c18af 100644 --- a/demos/egress/egress-template.yaml.tmpl +++ b/demos/egress/egress-template.yaml.tmpl @@ -53,6 +53,14 @@ volumes: - egress-mitm.ate.dev - system-roots.ate.dev path: egress-trust-bundle/trust-bundle.pem +# Sandbox size. The pool sets no worker limits, so the worker offers the +# node's allocatable and these only need to fit the node. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi sandboxConfig: sandboxClass: gvisor configName: gvisor-default diff --git a/demos/egress/egress.yaml.tmpl b/demos/egress/egress.yaml.tmpl index 28265fda39..09a798a4c9 100644 --- a/demos/egress/egress.yaml.tmpl +++ b/demos/egress/egress.yaml.tmpl @@ -33,6 +33,6 @@ metadata: workload: egress spec: replicas: 2 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor diff --git a/demos/jupyter/jupyter-template.yaml.tmpl b/demos/jupyter/jupyter-template.yaml.tmpl index f793a3a093..f7edabde18 100644 --- a/demos/jupyter/jupyter-template.yaml.tmpl +++ b/demos/jupyter/jupyter-template.yaml.tmpl @@ -39,6 +39,14 @@ containers: httpGet: path: /api port: 80 +# Sandbox size: room for a few kernels and their data next to the Jupyter +# server. +resources: + limits: + - name: cpu + quantity: "2" + - name: memory + quantity: 4Gi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-jupyter/ diff --git a/demos/jupyter/jupyter.yaml.tmpl b/demos/jupyter/jupyter.yaml.tmpl index 9c49c176a6..fb9fcaef2b 100644 --- a/demos/jupyter/jupyter.yaml.tmpl +++ b/demos/jupyter/jupyter.yaml.tmpl @@ -32,7 +32,7 @@ metadata: workload: jupyter spec: replicas: 1 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor diff --git a/demos/multi-template/counter-template.yaml.tmpl b/demos/multi-template/counter-template.yaml.tmpl index ab1414d238..c37a608dc4 100644 --- a/demos/multi-template/counter-template.yaml.tmpl +++ b/demos/multi-template/counter-template.yaml.tmpl @@ -27,6 +27,14 @@ workerSelector: containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter +# Sandbox size. The pool sets no worker limits, so the worker offers the +# node's allocatable and these only need to fit the node. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-multi-template-counter/ diff --git a/demos/multi-template/fspersist-template.yaml.tmpl b/demos/multi-template/fspersist-template.yaml.tmpl index d99c15644f..8701ca002d 100644 --- a/demos/multi-template/fspersist-template.yaml.tmpl +++ b/demos/multi-template/fspersist-template.yaml.tmpl @@ -27,6 +27,14 @@ workerSelector: containers: - name: fspersist image: ko://github.com/agent-substrate/substrate/demos/multi-template/fspersist +# Sandbox size. The pool sets no worker limits, so the worker offers the +# node's allocatable and these only need to fit the node. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-multi-template-fspersist/ diff --git a/demos/multi-template/multi-template.yaml.tmpl b/demos/multi-template/multi-template.yaml.tmpl index 1f8824ad75..d24aa04946 100644 --- a/demos/multi-template/multi-template.yaml.tmpl +++ b/demos/multi-template/multi-template.yaml.tmpl @@ -14,7 +14,7 @@ # This demo shows that two ActorTemplates running two different binaries can # share a single WorkerPool, even from different atespaces. A WorkerPool is -# image-agnostic capacity (its pods only run the ateom supervisor); each +# image-agnostic capacity (its pods only run the ateworker supervisor); each # ActorTemplate carries its own workload image and gates on the pool via # workerSelector, a label selector matched against the pool's labels (no # atespace or namespace scoping involved: pool selection is cluster-wide). @@ -36,6 +36,6 @@ metadata: workload: multi-template-shared spec: replicas: 3 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor diff --git a/demos/parking/parking-template.yaml.tmpl b/demos/parking/parking-template.yaml.tmpl index 2f71d51d80..e926fd9e21 100644 --- a/demos/parking/parking-template.yaml.tmpl +++ b/demos/parking/parking-template.yaml.tmpl @@ -26,6 +26,14 @@ containers: - name: counter image: ko://github.com/agent-substrate/substrate/demos/counter command: ["/ko-app/counter"] +# Sandbox size. The pool sets no worker limits, so the worker offers the +# node's allocatable and these only need to fit the node. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://${BUCKET_NAME}/ate-demo-parking/ diff --git a/demos/parking/parking.yaml.tmpl b/demos/parking/parking.yaml.tmpl index bc09c25271..9c908f093e 100644 --- a/demos/parking/parking.yaml.tmpl +++ b/demos/parking/parking.yaml.tmpl @@ -43,6 +43,6 @@ spec: # Intentionally small: 2 workers, so 3+ concurrently-active actors saturate # the pool and exercise the parking path. replicas: 2 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor diff --git a/demos/sandbox/manual-test-multi-template.yaml b/demos/sandbox/manual-test-multi-template.yaml index 61c844bd57..3adaa45c92 100644 --- a/demos/sandbox/manual-test-multi-template.yaml +++ b/demos/sandbox/manual-test-multi-template.yaml @@ -34,6 +34,14 @@ containers: mountPath: /hostpath-data - name: vol-nfs mountPath: /nfs-data +# Sandbox size. The pool sets no worker limits, so the worker offers the +# node's allocatable and these only need to fit the node. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi snapshotConfig: preferredFidelity: SNAPSHOT_FIDELITY_MEMORY storageLocation: gs://ate-snapshots/ate-manual-test-multi/ diff --git a/demos/sandbox/manual-test-multi.yaml b/demos/sandbox/manual-test-multi.yaml index 74cab46198..3b19ff5213 100644 --- a/demos/sandbox/manual-test-multi.yaml +++ b/demos/sandbox/manual-test-multi.yaml @@ -30,6 +30,6 @@ metadata: workload: sandbox spec: replicas: 1 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor diff --git a/demos/sandbox/sandbox.yaml.tmpl b/demos/sandbox/sandbox.yaml.tmpl index 3f7614b424..6b60b6a152 100644 --- a/demos/sandbox/sandbox.yaml.tmpl +++ b/demos/sandbox/sandbox.yaml.tmpl @@ -30,7 +30,7 @@ metadata: workload: sandbox spec: replicas: 2 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor template: diff --git a/docs/api-guide.md b/docs/api-guide.md index 09c585b2c4..8958b99338 100644 --- a/docs/api-guide.md +++ b/docs/api-guide.md @@ -11,7 +11,7 @@ The `WorkerPool` defines the pool of physical "warm" compute capacity. It manage | Field | Type | Description | | :--- | :--- | :--- | | `replicas` | `int32` | **Required.** Number of physical standby pods to maintain in the cluster. | -| `workerImage` | `string` | **Required.** The container image for the `ateom` herder process (e.g. `ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor`). | +| `workerImage` | `string` | **Required.** The container image for the `ateworker` herder process (e.g. `ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor`). | | `sandboxClasses` | `[]WorkerPoolSandboxClass` | **Required.** The sandbox runtime families the pool runs; exactly one entry today. Each entry has `name` (**required**, `gvisor` or `microvm`, immutable), which drives the worker pod shape (e.g. KVM device mounts, node placement), and `configRef.name` (optional), which names a cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) of the same class. Without `configRef`, the pool uses the class default (see [Class default](#class-default)). The controller records the result in `status.sandboxClasses`. The sandbox binaries an actor boots with still come from the `SandboxConfig` its `ActorTemplate` selects. | | `template` | `WorkerPoolPodTemplate` | **Optional.** Metadata, scheduling, and resource settings for worker workloads. | @@ -39,10 +39,10 @@ metadata; they do not affect actor scheduling. Actor selectors match #### Worker Capacity (`spec.template.resources`) -Setting `resources.limits` (CPU and Memory) on a `WorkerPool` establishes each worker pod's **capacity** — the envelope its actor sandboxes share, taken from the `ateom` container's limits. The scheduler only places an actor on a worker whose remaining capacity is `>=` the actor's declared resource limits (see [Sandbox Right-Sizing](#sandbox-right-sizing-resources) on the `ActorTemplate`). +Setting `resources.limits` (CPU and Memory) on a `WorkerPool` establishes each worker pod's **capacity** — the envelope its actor sandboxes share, taken from the `ateworker` container's limits. The scheduler only places an actor on a worker whose remaining capacity is `>=` the actor's declared resource limits (see [Sandbox Right-Sizing](#sandbox-right-sizing-resources) on the `ActorTemplate`). - Worker capacity is a shared budget: each actor placed on a worker subtracts its declared limits from what is left. Size a pool's `limits` for the actors it should host together. -- A worker also has an actor limit, set by the ateom's `--max-actors` flag (default 1000). Placement stops at whichever runs out first. +- A worker also has an actor limit, set by the ateworker's `--max-actors` flag (default 1000). Placement stops at whichever runs out first. - Capacity is advisory for placement only: a worker that declares no CPU/memory limit reports zero capacity for that dimension, which the scheduler treats as **unconstrained** (placement is never blocked by missing data). The actual sandbox size still comes from the `ActorTemplate`. ### Example @@ -57,7 +57,7 @@ metadata: workload: secret-agent spec: replicas: 10 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor configRef: @@ -103,7 +103,7 @@ The `ActorTemplate` defines the code, environment, and state-management policies | `workerSelector` | `*Selector` | Optional. Gates which `WorkerPool`s actors from this template may use, by matching against each pool's labels (`matchLabels`). If unset, all pools are eligible (subject to the actor's own `worker_selector`). | | `snapshotConfig` | `SnapshotConfig` | **Required.** The base object-storage location snapshots are written under, the snapshot fidelity (`preferredFidelity`) every Pause and Suspend captures (`SNAPSHOT_FIDELITY_ROOTFS` requires `sandboxClass: microvm`), and automated golden snapshot creation (`goldenSnapshotConfig.mode`). See [Snapshot Storage Layout](#snapshot-storage-layout). | | `volumes` | `[]Volume` | Optional. Volumes the containers may mount, each a `durableDir`, an `externalVolumeTemplate` (see [CSI Volumes Guide](csi-volumes.md)), or a `systemInfo` volume (see [SystemInfo Volumes](#systeminfo-volumes)). Every declared volume must be mounted by at least one container. A `microvm` template may declare several `durableDir` volumes; a `gvisor` template is limited to one. | -| `resources` | `*ResourceRequirements` | Optional. Declares each actor's compute size via `limits` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. | +| `resources` | `*ResourceRequirements` | Required. Declares each actor's compute size via `limits`, which must set both `cpu` and `memory` — see [Sandbox Right-Sizing](#sandbox-right-sizing-resources). Immutable, like the rest of the template. | The sandbox itself — the binaries (e.g. the gVisor `runsc` binary) and the `pauseImage` holding the sandbox's namespaces — comes from the cluster-scoped [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) object the template names via `sandboxConfig.configName`. An actor always resolves the config from its current template — repointing the actor at another template requires the same config. @@ -111,13 +111,12 @@ Because a snapshot is not restorable across sandbox runtimes, `sandboxClass` is ### Sandbox Right-Sizing (`resources`) -Unlike a Pod, an actor is sized by its **`limits`** (CPU and Memory): the size is a property of the template, baked into snapshots, so it lives on the immutable `ActorTemplate`. Declared limits do three things: +Unlike a Pod, an actor is sized by its **`limits`** (CPU and Memory): the size is a property of the template, baked into snapshots, so it lives on the immutable `ActorTemplate`. Both are required. Declared limits do two things: -1. **Size the sandbox.** The limits are supplied to the sandbox over the actor RPCs (control plane → atelet → ateom): - - **gVisor (`ateom-gvisor`)** — applied to the container OCI spec: `limits.cpu` sets the cgroup v2 CPU quota (`cpu.max`) and the Sentry vCPU count (`--cpu-num-from-quota`); `limits.memory` sets the cgroup v2 memory limit (`memory.max`) and bounds the virtual total memory the sandbox reports (so JVM/Go do not over-allocate from host RAM). - - **Micro-VM (`ateom-microvm`)** — `limits.cpu` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs); `limits.memory` sets guest RAM, reserving a small configurable margin (default 256 MiB, `--vmm-mem-reserve-mib`) for the VMM and virtiofsd so the pod cgroup does not OOM. +1. **Size the sandbox.** The limits are supplied to the sandbox over the actor RPCs (control plane → atelet → ateworker): + - **gVisor (`ateworker-gvisor`)** — applied to the container OCI spec: `limits.cpu` sets the cgroup v2 CPU quota (`cpu.max`) and the Sentry vCPU count (`--cpu-num-from-quota`); `limits.memory` sets the cgroup v2 memory limit (`memory.max`) and bounds the virtual total memory the sandbox reports (so JVM/Go do not over-allocate from host RAM). + - **Micro-VM (`ateworker-microvm`)** — `limits.cpu` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs); `limits.memory` sets guest RAM, reserving a small configurable margin (default 256 MiB, `--vmm-mem-reserve-mib`) for the VMM and virtiofsd so the pod cgroup does not OOM. 2. **Gate scheduling.** An actor is only placed on a `WorkerPool` whose [worker capacity](#worker-capacity-spectemplateresources) is `>=` these limits. -3. **Fall back to runtime defaults.** A zero or absent limit leaves that dimension at the runtime default: unlimited for gVisor, and 2 GiB / 1 vCPU for the micro-VM. `requests` are not consulted today (an actor occupies its whole worker). Because the size is baked into snapshots, a **micro-VM MEMORY-fidelity restore reuses the size in the snapshot**; changing an actor's limits takes effect on its next cold boot. @@ -281,7 +280,7 @@ A container that exceeds its memory limit is OOM-killed on its own; the actor's Per-container limits are micro-VM only today. gVisor applies cgroup limits at the sandbox level: one sentry backs every container in the actor, so a per-container cgroup is created and then stays empty ([google/gvisor#190](https://github.com/google/gvisor/issues/190)). A template that sets `resources` with `sandboxClass: gvisor` is rejected. -These limits subdivide the sandbox that [`resources`](#sandbox-right-sizing-resources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A micro-VM guest is sized from `resources.limits.memory` minus the VMM reserve, or from the template's [`SandboxConfig`](#3-sandboxconfig-the-sandbox-itself) when the template declares no actor-level limit. The CPU ceiling is the guest's vCPU count, which falls back to the config's `default_vcpus` (1 unless the `SandboxConfig` raises it), so a template that declares no `resources.limits.cpu` caps each container, and their sum, at `1000m`. A limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. +These limits subdivide the sandbox that [`resources`](#sandbox-right-sizing-resources) already sized; a container that declares none is bounded by the guest as a whole, not by a copy of the actor's total. A micro-VM guest's memory is `resources.limits.memory` minus the VMM reserve, and its vCPU count is `resources.limits.cpu` rounded up to whole cores. A container limit above either ceiling can never bind, so the actor fails to start with an error naming both the limit and the ceiling. Each limit is validated on its own at apply, but the sum across the actor's containers is only checked when the actor first runs, against the real guest size. A template whose limits do not fit is accepted by the API server and fails on its first actor. @@ -334,6 +333,13 @@ workerSelector: sandboxConfig: sandboxClass: gvisor configName: gvisor-default +# The actor's size, booked against worker capacity. +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 1Gi snapshotConfig: storageLocation: gs://my-bucket/secret-agent ``` diff --git a/docs/architecture.md b/docs/architecture.md index 39b11d0018..bd1a0b4a87 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -273,7 +273,7 @@ classDiagram } class Deployment class WorkerPod { - ateom + ateworker runsc } } @@ -317,25 +317,25 @@ manage actor lifecycles. * **Workflow Engine**: Orchestrates the multi-step Resume/Suspend sequences (lock acquisition, storage download, sandbox restore). -### Node Supervisor (`atelet` + `ateom`) +### Node Supervisor (`atelet` + `ateworker`) The node-level subsystem manages the physical execution of sandboxes and the movement of snapshots. * **atelet**: A lightweight supervisor running on each node as a DaemonSet. It acts as the "Herder," managing a pool of physical pods and communicating with the Control Plane. - * **ateom**: A specialized sandbox-herder container image — one per sandbox class (`ateom-gvisor`, `ateom-microvm`) — that runs inside the physical worker pods. It provides a gRPC interface for `atelet` to trigger `RunWorkload`, `CheckpointWorkload`, and `RestoreWorkload` operations. This separation ensures that the physical pod lifecycle remains decoupled from the sandboxed agent process. + * **ateworker**: A specialized sandbox-herder container image — one per sandbox class (`ateworker-gvisor`, `ateworker-microvm`) — that runs inside the physical worker pods. It provides a gRPC interface for `atelet` to trigger `RunWorkload`, `CheckpointWorkload`, and `RestoreWorkload` operations. This separation ensures that the physical pod lifecycle remains decoupled from the sandboxed agent process. - * **Lifecycle Management**: The `ateom` process invokes the sandbox runtime to checkpoint or restore processes within the physical pod boundaries — `runsc` for gVisor, or the Kata + Cloud Hypervisor stack for micro-VMs. (Note: the gVisor backend currently requires a `runsc` version with the `--allow-connected-on-save` flag to work around a bug in networking resumption during checkpointing.) + * **Lifecycle Management**: The `ateworker` process invokes the sandbox runtime to checkpoint or restore processes within the physical pod boundaries — `runsc` for gVisor, or the Kata + Cloud Hypervisor stack for micro-VMs. (Note: the gVisor backend currently requires a `runsc` version with the `--allow-connected-on-save` flag to work around a bug in networking resumption during checkpointing.) * **Storage Mover**: External snapshots move through an *object-store plugin*, a sidecar that serves the `objectstore.v1` gRPC API over a Unix socket in its pod. `atelet`'s sidecar (`NodeProvider`) uploads and fetches a snapshot's files; `ate-api-server`'s sidecar (`ControlProvider`) copies and deletes whole snapshots. Neither main process talks to object storage for snapshots. The in-tree plugin, `cmd/objectstore-plugin`, backs the API with GCS or S3; `ATE_STORAGE_BACKEND` on the sidecar (unset or `gcs`, or `s3` with the `AWS_*` variables) selects which, see `manifests/ate-install/atelet.yaml`, `ate-api-server.yaml`, and the kind overlay for an S3 example. The same `atelet` sidecar serves `AssetProvider`, which fetches every sandbox asset; `atelet` reads a `gs://` asset anonymously only when the plugin cannot serve it, which covers public buckets such as `gs://gvisor`. The plugin stages each download where `atelet` cannot read it, and copies it into `atelet`'s cache only if its sha256 matches; it refuses any URL under a snapshot location. `atelet` verifies the sha256 again and has no object storage credentials or settings of its own. ### Sandbox Classes -A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateom` herder image. The sandbox binaries themselves are not baked into the worker image — they, and the pause image holding the sandbox's namespaces, come at runtime from a cluster-scoped [`SandboxConfig`](api-guide.md#3-sandboxconfig-the-sandbox-itself) the `ActorTemplate` names in its sandbox config (naming one is currently required; per-class cluster defaults are planned) and are pinned into each snapshot's manifest so restores stay reproducible across runtime upgrades. +A `WorkerPool` selects a **sandbox class** (`spec.sandboxClasses[].name`), and each class has a matching `ateworker` herder image. The sandbox binaries themselves are not baked into the worker image — they, and the pause image holding the sandbox's namespaces, come at runtime from a cluster-scoped [`SandboxConfig`](api-guide.md#3-sandboxconfig-the-sandbox-itself) the `ActorTemplate` names in its sandbox config (naming one is currently required; per-class cluster defaults are planned) and are pinned into each snapshot's manifest so restores stay reproducible across runtime upgrades. - * **gVisor** (`ateom-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume leverage gVisor's native checkpoint/restore of the sandboxed process tree. + * **gVisor** (`ateworker-gvisor`, the default): Runs the workload under `runsc` for kernel-level sandboxing. Suspend and resume leverage gVisor's native checkpoint/restore of the sandboxed process tree. - * **micro-VM** (`ateom-microvm`): Runs the workload inside a [Kata Containers](https://katacontainers.io/) guest on the [Cloud Hypervisor](https://www.cloudhypervisor.org/) VMM. Suspend and resume capture a memory-only VM snapshot and restore it on-demand using `userfaultfd` memory demand-paging. Container rootfs writes are host-backed: the overlay is assembled on the host (read-only OCI image lower plus a per-actor writable upper) and served to the guest over the single virtio-fs share, so they cost reclaimable host page cache rather than guest RAM, and `ROOTFS` and `MEMORY` snapshots ship the upper as its own tar (a `ROOTFS` restore puts it back and cold-boots on it). `DurableDir` volumes travel over that same share and are likewise shipped as a tar, so a `VOLUMES` snapshot can capture them without any guest memory. Each volume is a subdirectory of the share, so an actor can have several at no extra cost in devices, which is why the micro-VM class lifts the single-`DurableDir` limit that still applies to gVisor. + * **micro-VM** (`ateworker-microvm`): Runs the workload inside a [Kata Containers](https://katacontainers.io/) guest on the [Cloud Hypervisor](https://www.cloudhypervisor.org/) VMM. Suspend and resume capture a memory-only VM snapshot and restore it on-demand using `userfaultfd` memory demand-paging. Container rootfs writes are host-backed: the overlay is assembled on the host (read-only OCI image lower plus a per-actor writable upper) and served to the guest over the single virtio-fs share, so they cost reclaimable host page cache rather than guest RAM, and `ROOTFS` and `MEMORY` snapshots ship the upper as its own tar (a `ROOTFS` restore puts it back and cold-boots on it). `DurableDir` volumes travel over that same share and are likewise shipped as a tar, so a `VOLUMES` snapshot can capture them without any guest memory. Each volume is a subdirectory of the share, so an actor can have several at no extra cost in devices, which is why the micro-VM class lifts the single-`DurableDir` limit that still applies to gVisor. ### Networking Stack (`atenet` + `atunnel`) @@ -350,7 +350,7 @@ Handles actor-aware routing and automatic re-animation. * **Worker Tunnel**: After resolving the assignment, `atenet-router` opens an mTLS `CONNECT` tunnel to the worker's `atunnel` listener on port 8443. - `atunnel`, hosted by `ateom`, forwards the request to the active Actor over + `atunnel`, hosted by `ateworker`, forwards the request to the active Actor over its private veth interface. Worker pod port 80 is not a direct Actor ingress path. With `--upstream-connect=false`, the Envoy router instead reaches `atunnel` over HTTPS on port 443. @@ -391,7 +391,7 @@ sequenceDiagram participant Gateway as atenet-router participant API as ate-api-server participant Atelet as atelet - participant Tunnel as ateom / atunnel + participant Tunnel as ateworker / atunnel participant A as Actor participant Store as snapshot storage @@ -457,7 +457,7 @@ Triggered by an inbound request at the Gateway or an explicit API call. 2. **Assignment**: The Control Plane claims a warm worker from the `WorkerPool`. - 3. **Hydration**: The `atelet` supervisor coordinates with the `ateom` process inside the worker pod to restore the ActorTemplate's golden external snapshot (for first-run) or the Actor's own external snapshot (for recurring runs) into the sandbox. + 3. **Hydration**: The `atelet` supervisor coordinates with the `ateworker` process inside the worker pod to restore the ActorTemplate's golden external snapshot (for first-run) or the Actor's own external snapshot (for recurring runs) into the sandbox. 4. **State**: State transitions to `ACTOR_STATE_RUNNING`. The actor now has an active Worker IP. @@ -470,7 +470,7 @@ Triggered by an inbound request at the Gateway or an explicit API call. Triggered by an explicit `SuspendActor` call. - 1. **Checkpoint**: The `atelet` instructs `ateom` to freeze the process and capture a memory+disk snapshot. + 1. **Checkpoint**: The `atelet` instructs `ateworker` to freeze the process and capture a memory+disk snapshot. 2. **Persistence**: The `atelet` streams the snapshot from the pod to durable storage (e.g., GCS). diff --git a/docs/csi-deployment.md b/docs/csi-deployment.md index ec9a2e4862..11172207cc 100644 --- a/docs/csi-deployment.md +++ b/docs/csi-deployment.md @@ -284,7 +284,7 @@ The CSI Node plugin runs as a DaemonSet on each worker node and handles `NodeSta ### Mount Propagation Requirements -When the CSI Node plugin mounts an external volume (such as an NFS share or a formatted block device), the filesystem mount is created inside the plugin container. For Substrate's node supervisor (`atelet`) and worker sandboxes (`ateom-gvisor`) on the host to see this filesystem mount, the following requirements must be met: +When the CSI Node plugin mounts an external volume (such as an NFS share or a formatted block device), the filesystem mount is created inside the plugin container. For Substrate's node supervisor (`atelet`) and worker sandboxes (`ateworker-gvisor`) on the host to see this filesystem mount, the following requirements must be met: 1. **Bidirectional Mount Propagation (`mountPropagation: Bidirectional`):** The volume mount on the host Substrate target directory (e.g. `/var/lib/ate`) in the CSI Node plugin container must have `mountPropagation: Bidirectional`. This ensures that any mounts made by the CSI plugin inside the container propagate back to the host filesystem. 2. **Unix Domain Socket Accessibility:** The CSI Node plugin must place its Unix domain socket under `/var/lib/kubelet/plugins//` (or the path defined in `CSIDriverConfig.spec.nodeSocketOverride`), which is shared with `atelet`. @@ -334,7 +334,7 @@ spec: - name: socket-dir mountPath: /csi # Target directory with bidirectional mount propagation - - name: ateom-dir + - name: ate-base mountPath: /var/lib/ate mountPropagation: Bidirectional volumes: @@ -342,7 +342,7 @@ spec: hostPath: path: /var/lib/kubelet/plugins/csi-nfsplugin type: DirectoryOrCreate - - name: ateom-dir + - name: ate-base hostPath: path: /var/lib/ate type: DirectoryOrCreate diff --git a/docs/csi-volumes.md b/docs/csi-volumes.md index 7aca525fa2..59b77af85f 100644 --- a/docs/csi-volumes.md +++ b/docs/csi-volumes.md @@ -2,6 +2,14 @@ Substrate integrates with the **Container Storage Interface (CSI)** to provide dynamically provisioned, per-actor external volumes that seamlessly attach and detach as actors transition through their lifecycle. +External volumes are a preview feature. The ate API server accepts them only +when it runs with `--preview=ExternalVolumes` (ate-setup's `preview` setting) +and the client opts in to preview features: `kubectl ate --preview`, or the +`Ate-Preview-Allowed: true` gRPC metadata header. The opt-in is needed to +create an actor template with external volumes, or an actor from such a +template. Later operations on that actor, such as resume, suspend and delete, +do not need it. + --- ## 1. CSI in Substrate vs. Standard Kubernetes @@ -153,13 +161,13 @@ metadata: workload: stateful-agent spec: replicas: 5 - workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor + workerImage: ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor sandboxClasses: - name: gvisor ``` The `ActorTemplate` is a protojson-shaped `ateapipb.ActorTemplate`, created -through the ate API with `kubectl ate create actor-template -f -` (the +through the ate API with `kubectl ate --preview create actor-template -f -` (the `ate-demo` atespace must exist): ```yaml @@ -182,6 +190,12 @@ containers: sandboxConfig: sandboxClass: gvisor configName: gvisor-default +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 1Gi snapshotConfig: storageLocation: gs://my-snapshots-bucket/stateful-agent volumes: diff --git a/docs/dev/best-practices/metrics.md b/docs/dev/best-practices/metrics.md index 9990b2a7e0..83f7e9cbb1 100644 --- a/docs/dev/best-practices/metrics.md +++ b/docs/dev/best-practices/metrics.md @@ -280,7 +280,7 @@ The meter provider itself is set up once per binary by `serverboot.InitMetricsBridged` (OTLP push with a foreign Prometheus registry bridged onto it, or that registry alone under `OTEL_METRICS_EXPORTER=prometheus`, and whether to serve it; atecontroller), or `serverboot.InitMetricsPushOnlyVia` -(OTLP push only, over the atelet relay when given one; ateom). +(OTLP push only, over the atelet relay when given one; ateworker). A new component calls one of these and defers `ShutdownProvider`; a new package inside an existing component adds nothing there. diff --git a/docs/dev/best-practices/otel-collector.md b/docs/dev/best-practices/otel-collector.md index d58505c705..5851a154c3 100644 --- a/docs/dev/best-practices/otel-collector.md +++ b/docs/dev/best-practices/otel-collector.md @@ -298,7 +298,7 @@ spec: # A node with no collector pod silently blackholes telemetry under # internalTrafficPolicy: Local, so tolerate everything. Any tainted pool # that runs Substrate workloads -- a dedicated sandbox pool, for - # instance -- would otherwise lose its actor and ateom spans. + # instance -- would otherwise lose its actor and ateworker spans. tolerations: - operator: Exists containers: @@ -424,17 +424,18 @@ these manifests: Setting the variable is sufficient for every component. Two also expose a flag that defaults to it, so you should not normally need either: `ate-controller`'s `--otel-exporter-otlp-endpoint`, which it also propagates to -the `ateom` worker pods it creates, and `atenet-router`'s +the worker pods it creates, and `atenet-router`'s `--otlp-collector-address`, which is what its Envoy is given over xDS. Rather than editing the base manifests, prefer a kustomize overlay that -patches the variable — see `manifests/ate-install/kind/kustomization.yaml` -for a worked example. +replaces the `ate-otel-config` ConfigMap — see +`manifests/ate-install/kind/ate-otel-config.yaml`, which the kind overlay +lists instead of the base file, for a worked example. ### A Note on Logs **Substrate exports one thing over OTLP: the actor events**, ateapi's -lifecycle records and the ateoms' usage samples, through +lifecycle records and the ateworkers' usage samples, through `serverboot.InitLogging`. `OTEL_LOGS_EXPORTER=otlp` sends them over OTLP instead of stdout, and `otlp,console` to both. The kind overlay selects `otlp,console`; the Helm chart selects `otlp` when `otel.logs.enabled` is true @@ -442,9 +443,9 @@ and a logs endpoint resolves. See [the same records over OTLP](../../observability.md#the-same-records-over-otlp). Everything else is stdout. `serverboot.InitLogger` writes structured JSON there, -and `ateom` wraps actor container output with the `ate.*` metadata labels -described in [Actor Observability](../../observability.md). The ateom relay -carries logs, traces, and metrics, including the ateoms' actor usage events. +and `ateworker` wraps actor container output with the `ate.*` metadata labels +described in [Actor Observability](../../observability.md). The ateworker relay +carries logs, traces, and metrics, including the ateworkers' actor usage events. Those labels sit in a nested group (`labels`, or `logging.googleapis.com/labels` on GKE, where the key promotes the group into `LogEntry.labels`). A filelog @@ -546,7 +547,7 @@ quantity of work that they do: datapoints/min ≈ nodes × atelet_dp + ateapi_replicas × ateapi_dp + router_replicas × router_dp - + worker_pods × ateom_dp + + worker_pods × worker_dp + instrumented_actors × actor_dp ``` @@ -584,7 +585,7 @@ it from the meter for the scenario you run. > [!IMPORTANT] > **With `ParentBased`, the caller makes the root decision.** Since > [#711](https://github.com/agent-substrate/substrate/pull/711) the defaults are -> `ParentBased(TraceIDRatioBased)` at 0.1 for `ateapi`, `atelet`, and `ateom-*`, +> `ParentBased(TraceIDRatioBased)` at 0.1 for `ateapi`, `atelet`, and `ateworker-*`, > and 0.01 for `atenet-router` and Envoy. These rates apply only to an operation > that arrives with no `traceparent`. A client that sends a `traceparent` makes > the decision for the full chain, and the defaults of substrate do not apply. diff --git a/docs/dev/best-practices/tracing.md b/docs/dev/best-practices/tracing.md index 18f0c4d5c9..3f9ffd9eee 100644 --- a/docs/dev/best-practices/tracing.md +++ b/docs/dev/best-practices/tracing.md @@ -57,7 +57,7 @@ Samplers are resolved with `serverboot.ResolveTraceSampling`, which applies the | Component | Default | |---|---| -| ateapi, atelet, ateom-gvisor, ateom-microvm | `parentbased_traceidratio` 0.1 | +| ateapi, atelet, ateworker-gvisor, ateworker-microvm | `parentbased_traceidratio` 0.1 | | atenet router (data plane root) | `parentbased_traceidratio` 0.01, mirrored into Envoy's `RandomSampling` | | glutton (benchmarking) | `parentbased_always_off` | | boomer (benchmarking) | runtime-controlled via dynconfig, ignores the env vars | @@ -72,7 +72,7 @@ These are head sampling ratios that bound what leaves the process. Keep decision ### Disabling tracing (perf/load tests) -Set `OTEL_TRACES_SAMPLER=always_off` on the components under test (for ateom workers, via the controller's `--otel-traces-sampler` flag). `parentbased_always_off` is not enough under a load generator: boomer and locust send ratio-sampled trace context, and parent based samplers honor it. Alternatively set the generator's `trace_probability` to 0 and leave the servers alone. On kind, also override ateapi's `parentbased_always_on` pin. +Set `OTEL_TRACES_SAMPLER=always_off` on the components under test (for workers, via the controller's `--otel-traces-sampler` flag). `parentbased_always_off` is not enough under a load generator: boomer and locust send ratio-sampled trace context, and parent based samplers honor it. Alternatively set the generator's `trace_probability` to 0 and leave the servers alone. On kind, also override ateapi's `parentbased_always_on` pin. The YAML manifest for your server needs `OTEL_EXPORTER_OTLP_ENDPOINT` set so the exporter knows where to push spans. Do not hardcode it — consume the shared diff --git a/docs/dev/code-layout.md b/docs/dev/code-layout.md index fb370af851..a527f0aa7f 100644 --- a/docs/dev/code-layout.md +++ b/docs/dev/code-layout.md @@ -53,8 +53,8 @@ Each subdirectory of `cmd/` corresponds to one compiled binary: | `cmd/atecontroller` | Kubernetes controller for WorkerPools (and network/egress resources) | | `cmd/atelet` | Node supervisor (DaemonSet) | | `cmd/atenet` | Network proxy / Envoy external-processing server | -| `cmd/ateom-gvisor` | In-pod gVisor container image entry point | -| `cmd/ateom-microvm` | In-pod kata + cloud-hypervisor micro-VM container image entry point | +| `cmd/ateworker-gvisor` | In-pod gVisor container image entry point | +| `cmd/ateworker-microvm` | In-pod kata + cloud-hypervisor micro-VM container image entry point | | `cmd/kubectl-ate` | `kubectl` plugin for interacting with Substrate | | `cmd/podcertcontroller` | Controller that issues pod TLS certificates | @@ -101,5 +101,5 @@ When adding a new Go package, ask: 3. **Is it a deliberately public API for external users?** → `pkg/` 4. **Is it a protobuf-generated package?** - Public gRPC API (control plane) → `pkg/proto/` - - Internal gRPC API (atelet, ateom) → `internal/proto/` + - Internal gRPC API (atelet, ateworker) → `internal/proto/` 5. **Is it a script or dev tool?** → `hack/` (shell) or `tools/` (Go) diff --git a/docs/dev/microvm-local.md b/docs/dev/microvm-local.md index 4af3632028..0a1a1650a1 100644 --- a/docs/dev/microvm-local.md +++ b/docs/dev/microvm-local.md @@ -1,6 +1,6 @@ # Running the microVM runtime locally -The microVM sandbox class (`ateom-microvm`: a Kata guest on Cloud Hypervisor) +The microVM sandbox class (`ateworker-microvm`: a Kata guest on Cloud Hypervisor) needs `/dev/kvm`, which takes some extra setup compared to the default gVisor path. This guide covers just that delta: getting a KVM-capable Docker environment — on Linux, or on Apple Silicon macOS via diff --git a/docs/dev/preview-gates.md b/docs/dev/preview-gates.md new file mode 100644 index 0000000000..6a55c0952c --- /dev/null +++ b/docs/dev/preview-gates.md @@ -0,0 +1,149 @@ +# Preview gates + +A preview gate lets a new feature ship in a release while staying off by +default. This gives users an opportunity to try things but protects them from +accidentally using things that we are not confident in. Operators enable gates +explicitly on the server side, and API clients must also say they mean to use +preview features. Gates are coarse switches; they are not access control. + +There are only two states: a feature is either **preview** (gated, off unless +enabled) or **GA** (always on). There is no alpha or beta. + +## How a gate is enabled + +Two independent switches must both be on before a request can use a gated +feature: + +1. **Server side:** every substrate binary takes `--preview=`, a + comma-separated list of gate names, or `ALL` for all of them. Install with + ate-setup's `preview` setting (`ATE_PREVIEW`, `--preview=`), which + passes the same value to every substrate container. atecontroller passes its + own gates on to the ateworker containers of each WorkerPool. +2. **Client side:** a gRPC request to ate-api must send the + `Ate-Preview-Allowed: true` metadata header (`preview.Header`). + `kubectl ate --preview` sends it. A request without it is treated as if + every gate were off. + +Substrate's own components (atelet, atenet, atecontroller) dial ate-api with +`preview.DialOptionsWithOptIn()`, so they are never blocked by the header +check. ate-setup does the same when it is run with preview gates. + +## Adding a gate + +Declare the gate in `internal/preview/preview.go`: + +```go +const ( + GateExternalVolumes Gate = "ExternalVolumes" + GateMyFeature Gate = "MyFeature" +) + +var realKnownGates = []Gate{ + GateExternalVolumes, + GateMyFeature, +} +``` + +Gate should always be named in UpperCamelCase. + +## Gating API fields + +Gate the fields a **user** sets with a declarative validation tag in the +`.proto`, then regenerate (`hack/update/codegen.sh`): + +```proto + // +ate:preview=MyFeature + // +k8s:ifDisabled(MyFeature)=+k8s:forbidden + // +k8s:optional + MyFeatureConfig my_feature = 7; +``` + +- `+ate:preview=` marks the field as part of a preview feature, for + readers and docs. It does not enforce anything. +- `+k8s:ifDisabled()=+k8s:forbidden` rejects the field when the gate is + off for this request. + +The gate state reaches the validators through `apivalidation.MakeCreateOp(ctx)` +and `apivalidation.MakeUpdateOp(ctx)`. Every `Validate*` wrapper must build its +`operation.Operation` with one of these, not with a bare +`operation.Operation{...}` literal; otherwise the field is never gated. + +Unchanged values are not re-validated on update, so an object created while a +gate was on stays usable after the gate is turned off. + +**Do not gate fields the server writes**, such as `status`. The server writes +them while doing system work, such as deleting an actor, and gating them can +leave objects that cannot be cleaned up after a gate is turned off. This is +subtle and will require careful thinking when implementing a gated feature. + +## Gating behavior in code + +For a check made while handling a user request, use the context-aware form: + +```go +if !preview.IsEnabledFor(ctx, preview.GateMyFeature) { + return nil, resources.ToAPIError(field.ErrorList{field.Forbidden( + path, "requires the MyFeature preview gate and a client that opts in to "+ + "preview features (kubectl ate --preview, or the "+preview.Header+" header)")}) +} +``` + +Use `preview.IsEnabled(gate)` only for checks that are about the process, not +the request, e.g. whether a component should start a feature at all. + +Be deliberate about what happens when a gate is turned **off** while the +feature is in use. Prefer to keep cleanup paths (delete, suspend, detach) +working regardless of the gate, so that turning it off cannot strand +resources. + +## Request context and workflows + +ate-api's gRPC interceptor marks a request's context as opted out unless it +sent the header. Work that a request starts but that runs on the system's +behalf must not inherit that restriction. Each exported `ActorWorkflow` and +`WorkerWorkflow` method therefore begins with: + +```go +ctx = preview.ContextWithOptIn(ctx) +``` + +`TestWorkflowsOptInToPreview` enforces this. Contexts that never went +through the interceptor, such as background reconcilers, are not opted out. + +## Testing + +Test gated code both with and without the gate. + +- Unit tests: `preview.InitForTest(t, "MyFeature")`, or with no gate names for + all-off. It restores the previous state when the test ends. Tests that use + it cannot run in parallel. +- Tests of the gate mechanism itself use `preview.InitForTestFake` with fake + gate names, so they don't depend on real gates. +- `apivalidation` tables: `assertValidate` takes `want` (gates off) and + `wantWithPreview`, keyed by a `--preview` value such as `"ALL"`. +- Functional tests (`controlapi/functionaltest`): `setupPreviewTest` gives a + client that sends the header, and `tc.noOptInClient` one that does not. + Neither enables a gate; call `preview.InitForTest` for that. +- E2E tests: start a test of a preview feature with + `e2e.RequirePreview(t, preview.GateMyFeature)`. It skips unless the suite + was run with `--preview` naming that gate, which must match what the + cluster was installed with: + + ```shell + hack/run-e2e-kind.sh -v -args --preview=ALL + ``` + + CI runs every e2e lane with preview gates both off and on, and requires + `--preview` to be passed (even as empty) so a lane cannot silently skip + every preview test. + +## Graduating or removing a gate + +When the feature goes GA, or is removed, keep the gate name in +`realKnownGates` for a while so existing `--preview` values still parse, but +make the feature always on (GA) or always off (removed). Then delete the +`ifDisabled` tags, `IsEnabled`/`IsEnabledFor` checks, and `RequirePreview` +calls. + + + diff --git a/docs/dev/suspend-resume-phase-breakdown.md b/docs/dev/suspend-resume-phase-breakdown.md index c4c3247c4e..d32d10f672 100644 --- a/docs/dev/suspend-resume-phase-breakdown.md +++ b/docs/dev/suspend-resume-phase-breakdown.md @@ -7,8 +7,8 @@ elsewhere; follow the links when a step needs more than the command shown. The breakdown comes from two developer-facing log records, `Checkpoint timing breakdown` and `Restore timing breakdown`, written once per operation by -atelet (the outer phases: `download`, `persist`, `ateom_*`, …) and by -ateom-microvm (the phases inside `ateom_*`: `pause`, `snapshot`, `vm_restore`, +atelet (the outer phases: `download`, `persist`, `worker_*`, …) and by +ateworker-microvm (the phases inside `worker_*`: `pause`, `snapshot`, `vm_restore`, `wakeup_probe`, …). `benchmarking/analysis/` collects the records from the pod logs and aggregates them. See [docs/observability.md](../observability.md#actor-attributed-component-logs) @@ -24,7 +24,7 @@ the report. or the [GKE Quickstart](../../README.md#gke-quickstart-development). For GKE the scripts below read `.ate-dev-env.sh` from the repository root (`collect_logs.sh` sources it itself; set `NO_DEV_ENV` to opt out). -* A **micro-VM** or **gVisor** pool: both ateoms write the inner records, +* A **micro-VM** or **gVisor** pool: both ateworkers write the inner records, so the report shows both layers for either class (a worker image that predates the gVisor records shows the atelet layer only on a gVisor pool). For micro-VM see [microvm-local.md](microvm-local.md). @@ -136,16 +136,16 @@ or the images predate the records. ## 5. Read it * **Phase percentiles** rank the phases. For a suspend, compare atelet's - `persist` against `ateom_checkpoint`, then the ateom rows (`snapshot`, + `persist` against `worker_checkpoint`, then the ateworker rows (`snapshot`, `teardown`, `pause`, …) against each other; for a resume, `download` against - `ateom_restore`, then `vm_restore` and `wakeup_probe`. The `unattributed` + `worker_restore`, then `vm_restore` and `wakeup_probe`. The `unattributed` row is the part of a checkpoint no logged phase names. -* **Waterfalls** show the slowest operations with the ateom record nested +* **Waterfalls** show the slowest operations with the ateworker record nested under the atelet phase it decomposes; a tail that is always the same phase is systematic, one that moves between phases is environmental. -* Two sanity checks on a healthy run: the atelet and ateom rows for one - operation have the same `n`, and atelet's `ateom_checkpoint` p50 is a few - tens of milliseconds above ateom's `total` p50 (the `(gap)` line in a +* Two sanity checks on a healthy run: the atelet and ateworker rows for one + operation have the same `n`, and atelet's `worker_checkpoint` p50 is a few + tens of milliseconds above ateworker's `total` p50 (the `(gap)` line in a waterfall is that RPC overhead). `/tmp/run1/report/phase_percentiles.csv` holds the same table for comparing @@ -160,8 +160,8 @@ phases scale with dirty memory. ## Troubleshooting -* **Only atelet rows, no ateom rows.** The worker image predates the ateom - records (check step 1's grep against a worker pod), or the ateom records +* **Only atelet rows, no ateworker rows.** The worker image predates the ateworker + records (check step 1's grep against a worker pod), or the ateworker records fell outside the collection window. * **Counts differ between layers.** The collection window cut through a cycle, or a worker pod restarted and took its log with it (the script dumps @@ -170,5 +170,5 @@ phases scale with dirty memory. * **A record has `error.type`.** That operation failed; the report leaves it out of the percentiles and the waterfalls. To see where it died, grep the record itself: a `DeadlineExceeded` whose time sits in `download` or - `persist` with a normal-looking `ateom_*` phase points at object storage, + `persist` with a normal-looking `worker_*` phase points at object storage, not the runtime. diff --git a/docs/glossary.md b/docs/glossary.md index 5eab2b4df5..7b99b303b0 100644 --- a/docs/glossary.md +++ b/docs/glossary.md @@ -61,13 +61,15 @@ for etcd. example, it turns a `WorkerPool` into a `Deployment`). - **atelet**: the node-level supervisor, run as a DaemonSet. It pulls images, - assembles OCI bundles, drives the sandbox lifecycle on the node via ateom, + assembles OCI bundles, drives the sandbox lifecycle on the node via ateworker, and streams snapshots to and from snapshot storage. -- **ateom**: the coordinator that runs inside each worker pod and drives the +- **ateworker**: the coordinator that runs inside each worker pod and drives the sandbox runtime on behalf of atelet. This decouples the physical pod lifecycle from the sandboxed agent process. It embeds a networking service called `atunnel` that handles network traffic for the sandboxed Actor. + ateworker is Substrate's worker implementation, one binary per sandbox class + (`ateworker-gvisor`, `ateworker-microvm`). - **atenet**: the networking stack. Its router resumes suspended Actors on demand and routes traffic to the right worker pod. diff --git a/docs/metrics/registry/events.yaml b/docs/metrics/registry/events.yaml index 5b4a1ee694..e119254217 100644 --- a/docs/metrics/registry/events.yaml +++ b/docs/metrics/registry/events.yaml @@ -106,14 +106,14 @@ groups: stability: development brief: A measurement of the CPU and the memory of one actor. note: > - The ateom that runs the actor writes one record per actor on a timer, + The ateworker that runs the actor writes one record per actor on a timer, plus an initial and a final record per activation. ate.stats.kind tells them apart. The records of an activation are written in that order: the initial one, then the periodic ones, then the final one. Each is dated when its sample was read, thus a final record that repeats an older sample can carry an earlier timestamp than the records before it. The initial record is absent when the first reading fails. The final record - is written when the ateom tears the activation down: a checkpoint, a + is written when the ateworker tears the activation down: a checkpoint, a terminate, a graceful shutdown of the worker, or a re-host on a micro-VM. It is read just before the teardown and repeats the newest sample when that read fails or times out, or is pending when none was @@ -136,8 +136,8 @@ groups: consumer can sample or drop it without touching the lifecycle events. annotations: substrate: - emitted_by: [ateom-gvisor, ateom-microvm] - code_anchor: internal/ateomstats/usage.go + emitted_by: [ateworker-gvisor, ateworker-microvm] + code_anchor: internal/workerstats/usage.go cuj: How much CPU and memory does this actor use, and since when? attributes: - ref: ate.atespace diff --git a/docs/metrics/registry/metrics.yaml b/docs/metrics/registry/metrics.yaml index f9382534ba..3d4a2f0f32 100644 --- a/docs/metrics/registry/metrics.yaml +++ b/docs/metrics/registry/metrics.yaml @@ -283,7 +283,7 @@ groups: has killed, over the life of the leaf (memory.events oom_kill). note: > Logs only. It is only ever recorded beside actor identity, which no - metric may carry. ateom-gvisor sets it on the record it writes when a + metric may carry. ateworker-gvisor sets it on the record it writes when a hosted actor's sandbox has no process left. A leaf left behind by an earlier activation of the same actor carries its count forward. examples: [0, 3] @@ -367,14 +367,14 @@ groups: stability: development value: oci_unpack brief: Restore. atelet unpacks the layers of the OCI image. - - id: ateom_restore + - id: worker_restore stability: development - value: ateom_restore - brief: Restore. This is the ateom restore call. - - id: ateom_checkpoint + value: worker_restore + brief: Restore. This is the worker restore call. + - id: worker_checkpoint stability: development - value: ateom_checkpoint - brief: Checkpoint. This is the ateom checkpoint call. + value: worker_checkpoint + brief: Checkpoint. This is the worker checkpoint call. - id: persist stability: development value: persist @@ -584,7 +584,7 @@ groups: - id: periodic stability: development value: periodic - brief: The timer of the ateom took the sample. + brief: The timer of the ateworker took the sample. - id: initial stability: development value: initial @@ -594,8 +594,8 @@ groups: value: final brief: > The last sample of an activation, written when a checkpoint, a - terminate, a graceful shutdown, or a re-host on ateom-microvm - ends it. When the sandbox is still running, ateom reads the + terminate, a graceful shutdown, or a re-host on ateworker-microvm + ends it. When the sandbox is still running, ateworker reads the stats just before the teardown; when no read runs, or the read fails or times out, it repeats the newest measured sample, or writes a pending record when there is none. @@ -909,7 +909,7 @@ groups: golden_signals: [latency] code_anchor: cmd/atelet/metrics.go buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 60] - cuj: Cold starts are slow. Is the cause the download, the unpack, or ateom? + cuj: Cold starts are slow. Is the cause the download, the unpack, or ateworker? attributes: - ref: ate.template.atespace requirement_level: required @@ -935,7 +935,7 @@ groups: brief: The time of one phase of an actor checkpoint on the worker node. note: > The phases are the same as the phases of the restore histogram. Thus you - can find the cause of a slow suspend in ateom or in the upload. The phases + can find the cause of a slow suspend in ateworker or in the upload. The phases occur at the same time. They do not add up to the total. annotations: substrate: @@ -943,7 +943,7 @@ groups: golden_signals: [latency] code_anchor: cmd/atelet/metrics.go buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 20, 30, 60] - cuj: Suspends are slow. Is the cause ateom or the upload? + cuj: Suspends are slow. Is the cause ateworker or the upload? attributes: - ref: ate.template.atespace requirement_level: required @@ -1003,7 +1003,7 @@ groups: It does not count usage twice. It can count less: the end of an activation after the last sample atelet read of it, which is up to one - poll interval plus one ateom sample interval plus the sweep's own time + poll interval plus one ateworker sample interval plus the sweep's own time (up to 45s on a micro-VM), so an activation shorter than that can lose most or all of its CPU here, while its usage records keep it; the time around an atelet restart; and, in a @@ -1279,13 +1279,13 @@ groups: brief: The time, the rate and the errors of each gRPC method on the server. note: > The otelgrpc interceptor sends this instrument. The code of Substrate does - not make it. This instrument is the only metric of ateom. ateom has it on + not make it. This instrument is the only metric of ateworker. ateworker has it on the gRPC interface that atelet calls. credprovider is the kubernetes-secrets credential provider in cmd/credential-provider/kubernetes-secrets. annotations: substrate: - emitted_by: [ateapi, atelet, atenet-router, ateom-gvisor, ateom-microvm, credprovider] + emitted_by: [ateapi, atelet, atenet-router, ateworker-gvisor, ateworker-microvm, credprovider] golden_signals: [latency, traffic, errors] source: external instrumentation: go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc @@ -1312,7 +1312,7 @@ groups: is much larger than the server duration of the same method, the delay is not in the handler. otelgrpc also sets server.address, but serverboot removes it with a View. The key holds the IP address of one server pod, - or the ateom socket path with the pod UID, so it has one value for each + or the ateworker socket path with the pod UID, so it has one value for each pod. annotations: substrate: diff --git a/docs/metrics/substrate.yaml b/docs/metrics/substrate.yaml index 66fb64e709..6eb455f2a9 100644 --- a/docs/metrics/substrate.yaml +++ b/docs/metrics/substrate.yaml @@ -245,7 +245,7 @@ blind_spots: brief: > ate.actor.stats.cpu.time counts only the samples atelet reads while an actor is hosted. It loses the end of each activation, up to one poll - interval plus one ateom sample interval. Thus an activation shorter than + interval plus one ateworker sample interval. Thus an activation shorter than that, as a checkpoint-when-idle agent, can show little or no CPU here. Its ate.actor.usage_sampled records keep it. Issue #2362 moves the - counter to the ateoms. + counter to the ateworkers. diff --git a/docs/network-egress.md b/docs/network-egress.md index 05ec840efc..842e046c4f 100644 --- a/docs/network-egress.md +++ b/docs/network-egress.md @@ -10,7 +10,7 @@ Agent Substrate enforces that all outbound traffic from an actor, except traffic At present, actor TCP traffic exits the sandbox through [`atunnel`](./glossary.md#components) and is forwarded on to the PEP. The exception is DNS: TCP and UDP traffic to port 53 on the sandbox's gateway bypasses the PEP and is answered by the sandbox's DNS relay, which forwards a query to the worker pod's resolvers only when the queried hostname matches a host or host wildcard in the actor's egress policy; queries from an actor without a policy, or for a hostname not matched by any rule, are refused. All other protocols are blocked. `atunnel` is configured with the PEP's network address and port at startup and uses that address to open an mTLS HTTP/1.1 CONNECT tunnel to the PEP. The CONNECT request target and `Host` header are both set to the original destination IP and port that `atunnel` derived from the actor's TCP connection. Actor networking is currently IPv4-only. -Before traffic reaches the egress PEP, ateom rejects packets destined for the IPv4 link-local range `169.254.0.0/16` or the IPv6 link-local range `fe80::/10`. The internal actor network `169.254.17.0/30` is exempt because it carries traffic for the actor gateway, DNS relay, ingress, and wakeup probes. Localhost traffic remains inside the actor sandbox and does not traverse the PEP. Access to non-link-local destinations remains controlled by substrate egress policy and, at the `WorkerPool` boundary, Kubernetes NetworkPolicy. +Before traffic reaches the egress PEP, ateworker rejects packets destined for the IPv4 link-local range `169.254.0.0/16` or the IPv6 link-local range `fe80::/10`. The internal actor network `169.254.17.0/30` is exempt because it carries traffic for the actor gateway, DNS relay, ingress, and wakeup probes. Localhost traffic remains inside the actor sandbox and does not traverse the PEP. Access to non-link-local destinations remains controlled by substrate egress policy and, at the `WorkerPool` boundary, Kubernetes NetworkPolicy. ## Trust Boundary @@ -31,7 +31,7 @@ The PEP MUST require and verify the client certificate before accepting a CONNEC ```text CURRENT CONNECT EGRESS PATH (one tunnel per actor TCP connection except destination port 53) - Actor sandbox Worker pod / ateom Egress PEP Authorized upstream + Actor sandbox Worker pod / ateworker Egress PEP Authorized upstream +------------------+ +--------------------------+ +-------------------------+ +------------------+ | Actor process | | nftables | | mTLS listener | | Target selected | | (untrusted) | | TCP REDIRECT | | + CONNECT terminator | | under policy | diff --git a/docs/observability.md b/docs/observability.md index 8674f7f601..a62fc38be1 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -150,7 +150,7 @@ atelet's `Restore timing breakdown` and `Checkpoint timing breakdown` are the un "ate.snapshot.fidelity":"memory","ate.snapshot.kind":"latest","ate.sandbox.class":"gvisor", "ate.actor.restore.duration.download":0.310, "ate.actor.restore.duration.oci_unpack":0.050, - "ate.actor.restore.duration.ateom_restore":0.060, + "ate.actor.restore.duration.worker_restore":0.060, "ate.actor.restore.duration.total":0.420, "trace_id":"4bf92f…","span_id":"00f067…","trace_flags":"01"} ``` @@ -159,7 +159,7 @@ The duration keys are the [`ate.actor.restore.duration`](#the-metric-registry) i This is the record to use for a per-actor wake-up distribution. The histogram cannot answer that question at all, because actor identity is barred from metric labels; traces can, but the data plane is head-sampled at 1%. -Both ateoms write records with the same two messages from inside the `ateom_restore` and `ateom_checkpoint` phases, under `ateom.actor.restore.duration.` and `ateom.actor.checkpoint.duration.`. The phases are each runtime's own and are listed in `cmd/ateom-gvisor/phaselog.go` and `cmd/ateom-microvm/phaselog.go`; `internal/ateomphaselog` renders the record. They are not instruments. The same identity keys make the two layers' records joinable per actor, and a failed operation's record carries `error.type`. +Both ateworkers write records with the same two messages from inside the `worker_restore` and `worker_checkpoint` phases, under `ateworker.actor.restore.duration.` and `ateworker.actor.checkpoint.duration.`. The phases are each runtime's own and are listed in `cmd/ateworker-gvisor/phaselog.go` and `cmd/ateworker-microvm/phaselog.go`; `internal/ateworker/phaselog` renders the record. They are not instruments. The same identity keys make the two layers' records joinable per actor, and a failed operation's record carries `error.type`. ateapi's `Actor state changed` is written once per committed actor state transition. ateapi owns the state machine, so this is where an actor's state and the time it reached it come from: @@ -183,7 +183,7 @@ Creating an actor counts as a change. A new actor is born suspended, so it gets **What this stream won't tell you.** It only writes when something changes. So an actor that has been sitting in the same state since before your logs roll over has no record, and no state. Ask the control plane what state something is in right now. Use this stream to see how it got there and when. Records can also go missing, like any other log, and a gap looks the same as an actor that just sat still. If you want to count activations, use the router's access log instead. -`Actor crashed` is the exception, and carries the same two keys with `ate.actor.state="crashed"`. It is written once per committed transition into `ACTOR_STATE_CRASHED`, beside the [`ate.actor.crashes`](#the-metric-registry) increment and under the same already-crashed guard, so the two can never disagree about how many crashes happened. A consumer deriving state therefore selects on `ate.actor.state` under ateapi's resource (`service.name=ateapi`), not on the message. The resource is what makes the record authoritative. [The ateom relay](#the-ateom-otlp-relay) admits only ateom resources, so once worker pods have no direct path to the collector, a record carrying these keys under any other resource cannot come from a worker pod. Until then this is a rule the consumer applies, not one the transport enforces. +`Actor crashed` is the exception, and carries the same two keys with `ate.actor.state="crashed"`. It is written once per committed transition into `ACTOR_STATE_CRASHED`, beside the [`ate.actor.crashes`](#the-metric-registry) increment and under the same already-crashed guard, so the two can never disagree about how many crashes happened. A consumer deriving state therefore selects on `ate.actor.state` under ateapi's resource (`service.name=ateapi`), not on the message. The resource is what makes the record authoritative. [The ateworker relay](#the-ateworker-otlp-relay) admits only ateworker resources, so once worker pods have no direct path to the collector, a record carrying these keys under any other resource cannot come from a worker pod. Until then this is a rule the consumer applies, not one the transport enforces. ```json {"time":"…","level":"ERROR","msg":"Actor crashed", @@ -195,7 +195,7 @@ Creating an actor counts as a change. A new actor is born suspended, so it gets The counter carries no actor identity, so this record is the only way to attribute a crash to one agent. The decision-point line that precedes it (`Setting Actor to crashed due to error`) carries only `ate.atespace` and `ate.actor.name`: it is written before the Actor is loaded, so no uid exists yet. -ateom-gvisor's `Sandbox has no processes left while the actor is hosted` is a WARN written when a running actor's gVisor sandbox has died: the sentry crashed, was killed, or was OOM-killed. The ateom's usage sweep finds it, so it shows up within one sweep (`--usage-sample-interval`, a minute by default), once per activation. `ate.sandbox.oom_kills` is the cgroup's `oom_kill` count, so a non-zero value points at the kernel OOM killer: +ateworker-gvisor's `Sandbox has no processes left while the actor is hosted` is a WARN written when a running actor's gVisor sandbox has died: the sentry crashed, was killed, or was OOM-killed. The ateworker's usage sweep finds it, so it shows up within one sweep (`--usage-sample-interval`, a minute by default), once per activation. `ate.sandbox.oom_kills` is the cgroup's `oom_kill` count, so a non-zero value points at the kernel OOM killer: ```json {"time":"…","level":"WARN","msg":"Sandbox has no processes left while the actor is hosted", @@ -208,7 +208,7 @@ The actor still reports `RUNNING` after this record; nothing moves it to `CRASHE #### The same records over OTLP -With `OTEL_LOGS_EXPORTER=otlp` these records go out as OTLP log events instead of stdout, so a collector reads them without knowing substrate's stdout envelope. Unset or `none` keeps them on stdout. The Helm chart selects `otlp` when `otel.logs.enabled` is true and a logs endpoint resolves. `otlp,console` writes both, as kind does for debugging. `console` here is the component's own JSON log line, not the OTel SDK's console exporter format. The value is a comma-separated list: names are trimmed, lowercased, and counted once, and an unknown or empty item logs a warning and is skipped while the rest apply. `none` with an exporter, or a value that names no known exporter, logs a warning and keeps the default, `none`. ateapi emits lifecycle events and the ateoms usage samples, on the same switch. The ateoms export through [the ateom relay](#the-ateom-otlp-relay), which carries logs, traces, and metrics. atecontroller copies `OTEL_LOGS_EXPORTER` into each worker pod, so the same selection applies to ateapi and the ateoms. +With `OTEL_LOGS_EXPORTER=otlp` these records go out as OTLP log events instead of stdout, so a collector reads them without knowing substrate's stdout envelope. Unset or `none` keeps them on stdout. The Helm chart selects `otlp` when `otel.logs.enabled` is true and a logs endpoint resolves. `otlp,console` writes both, as kind does for debugging. `console` here is the component's own JSON log line, not the OTel SDK's console exporter format. The value is a comma-separated list: names are trimmed, lowercased, and counted once, and an unknown or empty item logs a warning and is skipped while the rest apply. `none` with an exporter, or a value that names no known exporter, logs a warning and keeps the default, `none`. ateapi emits lifecycle events and the ateworkers usage samples, on the same switch. The ateworkers export through [the ateworker relay](#the-ateworker-otlp-relay), which carries logs, traces, and metrics. atecontroller copies `OTEL_LOGS_EXPORTER` into each worker pod, so the same selection applies to ateapi and the ateworkers. Three `event.name` values, which is the OTLP LogRecord's own field rather than an attribute: @@ -218,7 +218,7 @@ Three `event.name` values, which is the OTLP LogRecord's own field rather than a | `ate.actor.crashed` | `Actor crashed` | 17 | the same keys | | `ate.actor.usage_sampled` | `Actor usage sampled` | 9 | the five identity keys, `ate.workerpool.*`, `ate.sandbox.class`, `ate.stats.*`, `ate.actor.epoch` | -`ate.actor.usage_sampled` is the ateoms' record: one per actor per sampling interval, plus an `initial` record per activation and a `final` one when the ateom tears it down, told apart by `ate.stats.kind`. Its timestamp is when the measurement was read. Its measurements are named after the `ate.actor.stats.*` instruments and share their units, so `ate.stats.cpu.time` is seconds. They are absent, not zero, while the actor is not measurable, which the record says with `ate.stats.source` unspecified. `ate.stats.cpu.time` restarts at zero with each `ate.actor.epoch`, the unix-nano time the activation began, so a lifetime figure is the sum over epochs of each epoch's highest value; `ate.stats.memory.usage` and `ate.stats.memory.working_set` are absolute, and `ate.stats.memory.peak` is as the source reports it. The same measurements ride `WorkloadStatsSample` on the stats RPCs, with the epoch beside them. +`ate.actor.usage_sampled` is the ateworkers' record: one per actor per sampling interval, plus an `initial` record per activation and a `final` one when the ateworker tears it down, told apart by `ate.stats.kind`. Its timestamp is when the measurement was read. Its measurements are named after the `ate.actor.stats.*` instruments and share their units, so `ate.stats.cpu.time` is seconds. They are absent, not zero, while the actor is not measurable, which the record says with `ate.stats.source` unspecified. `ate.stats.cpu.time` restarts at zero with each `ate.actor.epoch`, the unix-nano time the activation began, so a lifetime figure is the sum over epochs of each epoch's highest value; `ate.stats.memory.usage` and `ate.stats.memory.working_set` are absolute, and `ate.stats.memory.peak` is as the source reports it. The same measurements ride `WorkloadStatsSample` on the stats RPCs, with the epoch beside them. A crash is its own name because an event name promises a set of attributes and a crash has a different severity and shape. There is no name per state: `ate.actor.state` already says which transition happened, so a consumer still selects on that one attribute and needs no map from a name to a state. All three names are in [`docs/metrics/registry/events.yaml`](metrics/registry/events.yaml), which `hack/verify-all.sh` checks. @@ -232,7 +232,7 @@ The attributes are the same flat `ate.*` keys as the stdout record, so they arri ### Per-Actor Usage Events -Each ateom samples its actors on its own timer and writes [`ate.actor.usage_sampled`](#the-same-records-over-otlp) for each one, in this order: an `initial` record when an activation becomes measurable, a `periodic` record per interval, and a `final` record when the ateom tears the activation down: a checkpoint, a terminate, a graceful shutdown, or a re-host on a micro-VM. The sampler reads an actor only after its initial reading and until its final record; until then the discovery read reports it as pending. A failed first reading writes no `initial` record, and periodic records start anyway, so an unreachable guest is still reported. These records and the [`ate.actor.stats.*` metrics](#the-metric-registry) are the halves of one split: the metrics aggregate to the bounded template and pool label set a TSDB can hold, and everything carrying actor identity travels here, on the log stream, where cardinality is free. +Each ateworker samples its actors on its own timer and writes [`ate.actor.usage_sampled`](#the-same-records-over-otlp) for each one, in this order: an `initial` record when an activation becomes measurable, a `periodic` record per interval, and a `final` record when the ateworker tears the activation down: a checkpoint, a terminate, a graceful shutdown, or a re-host on a micro-VM. The sampler reads an actor only after its initial reading and until its final record; until then the discovery read reports it as pending. A failed first reading writes no `initial` record, and periodic records start anyway, so an unreachable guest is still reported. These records and the [`ate.actor.stats.*` metrics](#the-metric-registry) are the halves of one split: the metrics aggregate to the bounded template and pool label set a TSDB can hold, and everything carrying actor identity travels here, on the log stream, where cardinality is free. The stdout record carries the same flat `ate.*` keys as the OTLP event, dated when the measurement was read: @@ -254,7 +254,7 @@ jsonPayload.msg="Actor usage sampled" AND jsonPayload."ate.actor.uid"="8f2a…" ``` * **The measurements** follow [the rules above](#the-same-records-over-otlp): absent while the actor is pending, CPU in seconds since the activation began, memory usage and working set absolute. An activation that ends any other way, as by a crash, has no `final` record, so take the highest CPU per epoch rather than wait for one. The `final` reading is taken just before the teardown, bounded on a micro-VM to 10ms. When it fails or times out, the `final` record repeats the newest measured sample, dated when that sample was read, or is pending when none was measured. -* **Cadence**: `--usage-sample-interval` on the ateom sets the timer (default `1m`, at least `50s`). Each sweep writes one periodic record per actor, and `GetActiveWorkloadStats` serves the latest sample rather than reading again. +* **Cadence**: `--usage-sample-interval` on the ateworker sets the timer (default `1m`, at least `50s`). Each sweep writes one periodic record per actor, and `GetActiveWorkloadStats` serves the latest sample rather than reading again. * **Trace context**: `initial` records come from the Run or Restore call (on a micro-VM, from a goroutine the call starts, so the record can land just before or after the call returns) and `final` records from the call that tears the activation down, a Checkpoint, a Terminate, or, on a micro-VM, the Run or Restore that re-hosts it; both carry that call's span, like any record in [Joining Logs to Traces](#joining-logs-to-traces). `periodic` records come from the sampler, and a shutdown's `final` records from the drain, which serve no request, so they carry none. * **Delivery is best-effort**: the stdout record ignores `--log-level` and sits behind a bounded queue, so a stalled log consumer costs records, with a warning, never the sampler. @@ -270,7 +270,7 @@ Agent Substrate emits foundational OpenTelemetry system and server metrics to mo | Metric | Emitted by | Type | Measures | |--------|------------|------|----------| -| `rpc.server.call.duration` | ateapi, atelet, atenet-router, ateom-gvisor, ateom-microvm & credprovider (gRPC servers, via `otelgrpc`) | histogram | per-method gRPC latency, request rate, and errors (labels `rpc.method`, `rpc.response.status_code`) | +| `rpc.server.call.duration` | ateapi, atelet, atenet-router, ateworker-gvisor, ateworker-microvm & credprovider (gRPC servers, via `otelgrpc`) | histogram | per-method gRPC latency, request rate, and errors (labels `rpc.method`, `rpc.response.status_code`) | | `ate.actor.crashes` | ateapi | counter | Number of times actors transitioned to `ACTOR_STATE_CRASHED` (labels `ate.actor.operation.name`, `ate.template.atespace`, `ate.template.name`, `ate.workerpool.namespace`, `ate.workerpool.name`, `ate.sandbox.class`) | | `atenet.router.route.duration` | atenet-router | histogram | Substrate E2E — Envoy receiving a request to Envoy forwarding it to the resolved worker, excluding actor compute and the response (labels `ate.template.atespace`, `ate.template.name`, `ate.router.outcome`, `ate.router.resume`) | | `atelet.snapshot.size` | atelet | histogram | uncompressed allocated size in bytes of each snapshot image written during checkpoint (`st_blocks * 512`, excluding sparse holes in micro-VM `memory-ranges`) (labels `file.name`, `ate.template.atespace`, `ate.template.name`) | @@ -282,7 +282,7 @@ Agent Substrate emits foundational OpenTelemetry system and server metrics to mo | `ate.actor.lifecycle.operation.duration` | ateapi | histogram | how long each actor operation (create/resume/suspend/pause/delete/revert) takes and whether it failed (`error.type` present = failure, absent = success); labeled by operation, template, pool (`ate.workerpool.namespace` + `ate.workerpool.name`), sandbox class, snapshot kind on resume, and snapshot fidelity on resume, suspend, and pause; already-running resume no-ops are not recorded so the histogram tracks actual activations, not router traffic | | `ate.scheduler.assignment.duration` | ateapi | histogram | time of one attempt to assign a worker to an actor, with one record for each resume (ateapi retries an attempt that loses a race for a worker and records only the last attempt), with the outcome (`assigned` / `no_capacity` / `error`), the assigned pool (`ate.workerpool.namespace` + `ate.workerpool.name`) and sandbox class to catch scheduling latency and capacity starvation problems | | `ate.actor.restore.duration` | atelet | histogram | how long each phase of a restore takes on the worker node, which is where cold-start latency actually goes once ateapi hands off (labels `ate.snapshot.phase`, `ate.snapshot.kind`, `ate.snapshot.fidelity`, `ate.template.atespace`, `ate.template.name`, `ate.sandbox.class`) | -| `ate.actor.checkpoint.duration` | atelet | histogram | the same phase breakdown for writing a snapshot, so a slow suspend can be attributed to ateom or to the upload (same labels as the restore histogram) | +| `ate.actor.checkpoint.duration` | atelet | histogram | the same phase breakdown for writing a snapshot, so a slow suspend can be attributed to ateworker or to the upload (same labels as the restore histogram) | | `ate.imagecache.requests` | atelet | counter | image lookups in the node-local image cache, by outcome (`ate.imagecache.outcome`), with `error.type` on the `error` outcome. A miss pays for the pull and the unpack, so the hit ratio per node is a leading indicator of resume latency | The table lists the OpenTelemetry instrument names. How a name appears in a query depends on the backend (Cloud Monitoring (GMP) / Kind collector). @@ -307,7 +307,7 @@ For `ate.imagecache.requests`: The three snapshot labels are orthogonal and mean the same thing on every histogram that carries them: * `ate.snapshot.kind`: which snapshot the operation reads or writes. `local` (node-local, written by a pause), `latest` (the actor's own durable snapshot), `golden` (the template's image), or `boot` (from scratch, so it never appears on the atelet histograms). * `ate.snapshot.fidelity`: what state it holds. `volumes`, `rootfs`, or `memory`. -* `ate.snapshot.phase`: which step was timed. `volume_mount`, `manifest_fetch`, `sandbox_assets`, `download`, `oci_unpack`, `ateom_restore` on restore; `sandbox_assets`, `ateom_checkpoint`, `persist` on checkpoint; `total` on both. +* `ate.snapshot.phase`: which step was timed. `volume_mount`, `manifest_fetch`, `sandbox_assets`, `download`, `oci_unpack`, `worker_restore` on restore; `sandbox_assets`, `worker_checkpoint`, `persist` on checkpoint; `total` on both. **Phases overlap and do not sum to `total`.** The download runs concurrently with the asset fetch and OCI unpack, so each is an independent observation; use `total` as the denominator. A phase that never started is absent rather than zero. @@ -362,7 +362,7 @@ A queue that has never processed an item bridges as an exponential histogram wit ### Scraping instead of pushing -ateapi, atelet, atenet-router and the credential provider also serve every instrument on their Prometheus `/metrics` endpoint. A cluster that scrapes those endpoints sets `OTEL_METRICS_EXPORTER=prometheus` on the components, so each series reaches the backend once. With `prometheus` the components install no OTLP metric reader and keep the Prometheus one; traces and logs are unaffected. `none` stops the push too. For now `/metrics` stays on for every value, including `otlp` and `none`. This is temporary until the probes move off `:9090`; `otlp` and `none` will then turn it off ([#2067](https://github.com/agent-substrate/substrate/issues/2067)). The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`. Unset is `otlp,prometheus`, or `otlp` alone on ateom. `otlp` keeps the push, and `prometheus` or `none` stops it. ateom serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning, and the push stays on unless the variable is `none`. +ateapi, atelet, atenet-router and the credential provider also serve every instrument on their Prometheus `/metrics` endpoint. A cluster that scrapes those endpoints sets `OTEL_METRICS_EXPORTER=prometheus` on the components, so each series reaches the backend once. With `prometheus` the components install no OTLP metric reader and keep the Prometheus one; traces and logs are unaffected. `none` stops the push too. For now `/metrics` stays on for every value, including `otlp` and `none`. This is temporary until the probes move off `:9090`; `otlp` and `none` will then turn it off ([#2067](https://github.com/agent-substrate/substrate/issues/2067)). The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`. Unset is `otlp,prometheus`, or `otlp` alone on ateworker. `otlp` keeps the push, and `prometheus` or `none` stops it. ateworker serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning, and the push stays on unless the variable is `none`. atecontroller's `:8080` listener follows the variable. When the variable is unset or `otlp,prometheus`, atecontroller pushes every metric over OTLP, and `:8080` serves only the controller-runtime families ([#2402](https://github.com/agent-substrate/substrate/issues/2402)). With `otlp`, atecontroller pushes every metric and closes `:8080`. With `prometheus`, atecontroller does not push, and `:8080` serves the controller-runtime families and the `ate.workerpool.*` instruments. With `none`, atecontroller does not push and closes `:8080`. @@ -418,9 +418,9 @@ To visualize traces locally: ``` The kind overlay pins `ateapi` to `parentbased_always_on`, so API calls show up even without `--trace`; the flag additionally prints the trace ID and forces sampling on every hop. -4. **Search and Inspect**: Copy the printed Trace ID from the CLI output and paste it into the Jaeger search box (top right), or select `ateapi`, `atelet`, or `ateom-gvisor` under the **Service** dropdown and click **Find Traces** to inspect detailed call stacks, DB transactions, state updates, and worker pod handoffs. +4. **Search and Inspect**: Copy the printed Trace ID from the CLI output and paste it into the Jaeger search box (top right), or select `ateapi`, `atelet`, or `ateworker-gvisor` under the **Service** dropdown and click **Find Traces** to inspect detailed call stacks, DB transactions, state updates, and worker pod handoffs. -> ateom carries no manual spans — its only instrumentation is the `otelgrpc` interceptor on the gRPC surface `atelet` calls. So it produces a span for an actor lifecycle operation (`suspend`, `resume`) and nothing at all for a read like `kubectl ate get actor`. Its sampler is parent based, so a lifecycle command is traced end to end into ateom whenever `ateapi` roots a sampled trace, which the kind overlay makes unconditional; the per-component ratio never enters into it. To check whether ateom exported its spans through the [OTLP relay](#the-ateom-otlp-relay) rather than falling back to direct network egress, inspect the span's resource attributes: `ate.otlp.relay` will be set to `"relay"` (instead of `"direct"`). +> ateworker carries no manual spans — its only instrumentation is the `otelgrpc` interceptor on the gRPC surface `atelet` calls. So it produces a span for an actor lifecycle operation (`suspend`, `resume`) and nothing at all for a read like `kubectl ate get actor`. Its sampler is parent based, so a lifecycle command is traced end to end into ateworker whenever `ateapi` roots a sampled trace, which the kind overlay makes unconditional; the per-component ratio never enters into it. To check whether ateworker exported its spans through the [OTLP relay](#the-ateworker-otlp-relay) rather than falling back to direct network egress, inspect the span's resource attributes: `ate.otlp.relay` will be set to `"relay"` (instead of `"direct"`). > **Developer Guide:** For detailed instructions on configuring OpenTelemetry tracer providers, middleware, and exporters in your servers or clients, please refer to the [Tracing Best Practices](dev/best-practices/tracing.md) guide. @@ -435,32 +435,32 @@ Telemetry is emitted the same way everywhere; only the backend differs between a | Path | service → in-cluster `opentelemetry-collector` | service → Google Managed Prometheus (GMP) | | Metrics | collector Prometheus exporter on `:8889` | Google Cloud Monitoring | | Traces | Jaeger UI | Google Cloud Trace | -| Logs | pod stdout; the [actor events](#the-same-records-over-otlp), ateapi's lifecycle records and the ateoms' usage samples, also to the collector's `debug` exporter | pod stdout. No OTLP logs unless `OTEL_LOGS_EXPORTER` is set | +| Logs | pod stdout; the [actor events](#the-same-records-over-otlp), ateapi's lifecycle records and the ateworkers' usage samples, also to the collector's `debug` exporter | pod stdout. No OTLP logs unless `OTEL_LOGS_EXPORTER` is set | | Dashboards | Not supported | Google Cloud Monitoring (see [Dashboards](#5-dashboards)) | -> In Kind, `ateapi`, `atelet`, `ate-controller`, and `atenet-router` are pointed at the in-cluster collector, and the controller propagates the endpoint to the ateom worker pods it creates, so all component telemetry lands locally. +> In Kind, `ateapi`, `atelet`, `ate-controller`, and `atenet-router` are pointed at the in-cluster collector, and the controller propagates the endpoint to the worker pods it creates, so all component telemetry lands locally. > > Every component reads that endpoint from the shared `ate-otel-config` ConfigMap ([`manifests/ate-install/ate-otel-config.yaml`](../manifests/ate-install/ate-otel-config.yaml), with a Kind replacement of the same name under [`manifests/ate-install/kind/`](../manifests/ate-install/kind/ate-otel-config.yaml)). Editing it does not restart the pods that consume it — follow a change with `kubectl rollout restart`. > -> ateom workers don't read the ConfigMap at all — `ate-controller` copies the value into each worker pod at creation. A new endpoint reaches them only once the controller itself restarts, and that restart then rolls every WorkerPool Deployment, replacing the running workers along with the actors on them. +> Workers don't read the ConfigMap at all — `ate-controller` copies the value into each worker pod at creation. A new endpoint reaches them only once the controller itself restarts, and that restart then rolls every WorkerPool Deployment, replacing the running workers along with the actors on them. -### The ateom OTLP relay +### The ateworker OTLP relay -ateom is the one component that does not talk to the collector directly. It exports logs, traces, and metrics over a unix socket at `/var/lib/ate/atelet-otlp.sock`, which `atelet` serves and forwards to the collector on the node's network ([`internal/otlprelay`](../internal/otlprelay)): +ateworker is the one component that does not talk to the collector directly. It exports logs, traces, and metrics over a unix socket at `/var/lib/ate/atelet-otlp.sock`, which `atelet` serves and forwards to the collector on the node's network ([`internal/otlprelay`](../internal/otlprelay)): ``` -ateom ──OTLP/gRPC over unix socket──► atelet relay ──OTLP/gRPC──► collector +ateworker ──OTLP/gRPC over unix socket──► atelet relay ──OTLP/gRPC──► collector ``` -The socket sits in the `BasePath` hostPath already mounted into both, so nothing new is mounted. `atelet` is a DaemonSet, so every ateom on a node shares one relay, and the many per-pod collector connections collapse into one per node. Four things motivate it: the worker pod runs untrusted agent code and will not need egress to the collector once direct fallback is phased out; the connection count drops; ateom's own telemetry stays clear of the transparent egress redirect it installs for the actor; and `atelet` outlives the worker pod, so spans still queued at teardown are not lost with it. +The socket sits in the `BasePath` hostPath already mounted into both, so nothing new is mounted. `atelet` is a DaemonSet, so every ateworker on a node shares one relay, and the many per-pod collector connections collapse into one per node. Four things motivate it: the worker pod runs untrusted agent code and will not need egress to the collector once direct fallback is phased out; the connection count drops; ateworker's own telemetry stays clear of the transparent egress redirect it installs for the actor; and `atelet` outlives the worker pod, so spans still queued at teardown are not lost with it. -The relay is best-effort. If the socket is absent when ateom starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateom logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT` as before. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `ate.otlp.relay` resource attribute (`relay` vs `direct`). +The relay is best-effort. If the socket is absent when ateworker starts — `atelet` not up yet, `--otlp-relay-socket=""`, or no collector configured for the relay to forward to — ateworker logs it and exports directly to `OTEL_EXPORTER_OTLP_ENDPOINT` as before. That fallback is decided once at startup, not per export, and is stamped on telemetry as the `ate.otlp.relay` resource attribute (`relay` vs `direct`). > **Note on Network Egress Lockdown:** Complete network policy lockdown of worker pod egress to the collector is planned as a Phase 2 milestone once the relay path is fully proven and direct fallback is deprecated. While the fallback path remains active, worker pods retain network egress to the collector and `ate-controller` continues to inject `OTEL_EXPORTER_OTLP_ENDPOINT`. -For verified ateom sources, the relay forwards each request verbatim rather than decoding and re-exporting, which is what keeps every ateom its own service in Jaeger/GCP Trace instead of being absorbed into `atelet`'s. `ate-controller` injects `k8s.pod.name`, `k8s.namespace.name`, `k8s.pod.uid`, `k8s.node.name`, and `service.instance.id` directly into `OTEL_RESOURCE_ATTRIBUTES` via the Kubernetes Downward API; because the relay preserves resources verbatim, Kubernetes attributes remain intact even though the TCP connection to the collector originates from `atelet` rather than the worker pod IP (bypassing reliance on collector-side IP-based `k8sattributes` enrichment). +For verified ateworker sources, the relay forwards each request verbatim rather than decoding and re-exporting, which is what keeps every ateworker its own service in Jaeger/GCP Trace instead of being absorbed into `atelet`'s. `ate-controller` injects `k8s.pod.name`, `k8s.namespace.name`, `k8s.pod.uid`, `k8s.node.name`, and `service.instance.id` directly into `OTEL_RESOURCE_ATTRIBUTES` via the Kubernetes Downward API; because the relay preserves resources verbatim, Kubernetes attributes remain intact even though the TCP connection to the collector originates from `atelet` rather than the worker pod IP (bypassing reliance on collector-side IP-based `k8sattributes` enrichment). -Verbatim forwarding is restricted to known ateom sources and refuses anything else with `PermissionDenied`. Actor telemetry is what that excludes: actors share a hostname (`actor`) and an interior IP, so their series merge unless identity is injected from outside the actor ([#761](https://github.com/agent-substrate/substrate/issues/761)) — a rewrite, which will be implemented as an explicit rewriting path alongside this forwarder. +Verbatim forwarding is restricted to known ateworker sources and refuses anything else with `PermissionDenied`. Actor telemetry is what that excludes: actors share a hostname (`actor`) and an interior IP, so their series merge unless identity is injected from outside the actor ([#761](https://github.com/agent-substrate/substrate/issues/761)) — a rewrite, which will be implemented as an explicit rewriting path alongside this forwarder. ### Actor telemetry needs an egress policy @@ -496,7 +496,7 @@ Three things to know when applying this: * **A running actor recovers slowly.** If the policy is created while the actor is already exporting and failing, the SDK's gRPC client is in reconnect backoff, the sandbox's DNS relay takes up to 5s to notice the policy, and the gateway caches the missing policy for `--egress-policy-cache-ttl` (10s by default). Expect the first successful export 45s to two minutes after the policy appears. An actor whose policy exists before it starts has no such delay. * **The collector sees the gateway, not the actor.** The TCP connection to the collector comes from the egress gateway pod, so a collector that enriches by source IP (the `k8sattributes` processor, which the GKE managed collector runs) labels the actor's spans and metrics with the gateway's `k8s.pod.name`, `k8s.deployment.name=atenet-egress`, and `k8s.namespace.name`. The actor's `service.name` and anything it puts in `OTEL_RESOURCE_ATTRIBUTES` survive. Set the actor's identity there yourself if you need it on the backend; the relay-based path that will do this for actors is tracked in [#853](https://github.com/agent-substrate/substrate/issues/853) and [#761](https://github.com/agent-substrate/substrate/issues/761). -**Logs written to stdout and stderr need none of this.** ateom reads a container's output stream inside the worker pod and writes it to the pod's own stdout, which is where [`kubectl ate logs`](#active-actor-inspection-via-cli) and a node log agent pick it up. That path never leaves the sandbox over the network, so it is untouched by the egress lockdown and works for an actor with no policy at all. The policy is only for telemetry an SDK inside the actor pushes itself: traces, metrics, and OTLP logs if `OTEL_LOGS_EXPORTER=otlp` is set in the actor. +**Logs written to stdout and stderr need none of this.** ateworker reads a container's output stream inside the worker pod and writes it to the pod's own stdout, which is where [`kubectl ate logs`](#active-actor-inspection-via-cli) and a node log agent pick it up. That path never leaves the sandbox over the network, so it is untouched by the egress lockdown and works for an actor with no policy at all. The policy is only for telemetry an SDK inside the actor pushes itself: traces, metrics, and OTLP logs if `OTEL_LOGS_EXPORTER=otlp` is set in the actor. Substrate puts no OTLP configuration into an actor container. Set `OTEL_EXPORTER_OTLP_ENDPOINT` in the ActorTemplate's container `env`; without it the SDK defaults to `localhost:4317`, where nothing listens inside the sandbox, and every export is dropped with no error at the collector. diff --git a/docs/operator-install.md b/docs/operator-install.md index 51ef67c42a..ad84e7cb50 100644 --- a/docs/operator-install.md +++ b/docs/operator-install.md @@ -301,6 +301,7 @@ demos, the [benchmarks](../benchmarking/README.md#settings), and the | `gcp.projectID` | `PROJECT_ID` | `--gcp-project-id` | — | (global) | | `images.repo` | `ATE_IMAGE_REPO` | `--image-repo` | — | (global) | | `images.tag` | `ATE_IMAGE_TAG` | `--image-tag` | — | (global) | +| `ingressAuthMode` | `ATE_INGRESS_AUTH_MODE` | `--ingress-auth-mode` | `deprecated-insecure` | (global) | | `kindCluster.enabled` | `ATE_INSTALL_KIND` | `--kind` | `false` | (global) | | `kindCluster.name` | `KIND_CLUSTER_NAME` | `--kind-cluster-name` | `kind` | (global) | | `ko.defaultPlatforms` | `KO_DEFAULTPLATFORMS` | `--ko-default-platforms` | — | (global) | @@ -309,6 +310,7 @@ demos, the [benchmarks](../benchmarking/README.md#settings), and the | `namespace` | `ATE_NAMESPACE` | `--namespace` | `ate-system` | (global) | | `otlpEndpoint` | `ATE_OTLP_ENDPOINT` | `--otlp-endpoint` | — | (global) | | `podcert.workersPerSigner` | `ATE_INSTALL_PODCERT_WORKERS_PER_SIGNER` | `--podcert-workers-per-signer` | `0` | (global) | +| `preview` | `ATE_PREVIEW` | `--preview` | — | (global) | | `record.dir` | `ATE_RECORD_DIR` | `--record-dir` | — | (global) | | `rolloutTimeout` | `ATE_INSTALL_ROLLOUT_TIMEOUT` | `--rollout-timeout` | `1m0s` | (global) | | `storage.bucketName` | `BUCKET_NAME` | `--storage-bucket-name` | — | (global) | diff --git a/docs/security.md b/docs/security.md index 0221bd35d5..dbde5ec01f 100644 --- a/docs/security.md +++ b/docs/security.md @@ -58,7 +58,7 @@ Other forms of tenancy: | :--- | :--- | :--- | | **Zero Ambient Credentials** | Default actor sandboxes execute completely credential-free. Actors may hold credential identifiers, encrypted credentials, or unprivileged tokens that need to be hydrated or decrypted out-of-band by Substrate infrastructure before accessing external services. | Met (no credentials inside sandbox) | | **Distinct Workload Identity** | Every actor receives a short lifetime (1h) SPIFFE ID X.509 certificate. It is possible to distinguish an actor that was deleted and then re-created with the same name using their UID. | Met (X.509)
In-Progress (OIDC discovery) | -| **Universal Component mTLS** | All control plane and data plane components (`ate-api-server`, `atelet`, `ateom`, routers, tunnels) authenticate and communicate exclusively over mutual TLS using node- and service-attested identities. Unauthenticated traffic is rejected, exempting only local unauthenticated health/liveness probes (`/healthz`, `/readyz`). | Met (Mutual TLS)
In-Progress (Authz) | +| **Universal Component mTLS** | All control plane and data plane components (`ate-api-server`, `atelet`, `ateworker`, routers, tunnels) authenticate and communicate exclusively over mutual TLS using node- and service-attested identities. Unauthenticated traffic is rejected, exempting only local unauthenticated health/liveness probes (`/healthz`, `/readyz`). | Met (Mutual TLS)
In-Progress (Authz) | | **Sub-Actor Provenance** | Delegated sub-actors inherit cryptographically signed lineage. Every tool invocation, data access, and egress transaction can be deterministically resolved to the root actor and execution chain. This provenance/hierarchy needs to be recorded in the system that calls substrate. | Out of Scope | ### Storage diff --git a/docs/threat-model.md b/docs/threat-model.md index 92ef6db759..a8da7edf18 100644 --- a/docs/threat-model.md +++ b/docs/threat-model.md @@ -35,10 +35,10 @@ Substrate is an early, fast moving product. It is full of debate and subject to * **ate-api-server:** Actor creation and scheduling and credential issuance. * **atelet:** Per-node daemon, performs snapshot/resume. -* **ateom:** Per-worker Pod sidecar, running inside the worker Pod. Ateom sets up "interior" sandboxes in the worker Pod and manages sandbox lifecycle, including image pulls. It currently uses gvisor but Substrate will support multiple microvm solutions. +* **ateworker:** Per-worker Pod sidecar, running inside the worker Pod. It sets up "interior" sandboxes in the worker Pod and manages sandbox lifecycle, including image pulls. It currently uses gvisor but Substrate will support multiple microvm solutions. * **Worker:** Preprovisioned Pods that actors get scheduled to. * **Actor:** The core compute primitive, gets scheduled to/from worker via Run for cold start and Resume for snapshot resume. -* **Actor Network:** `ateom` creates a private point-to-point veth network for the active Actor inside a worker Pod. The Actor is not served directly from the worker Pod's port 80; ingress enters through `atunnel`'s authenticated listeners: mTLS `CONNECT` on port 8443, which the router uses by default, and HTTPS on port 443. +* **Actor Network:** `ateworker` creates a private point-to-point veth network for the active Actor inside a worker Pod. The Actor is not served directly from the worker Pod's port 80; ingress enters through `atunnel`'s authenticated listeners: mTLS `CONNECT` on port 8443, which the router uses by default, and HTTPS on port 443. * **Actor Routing:** Requests sent to `atenet-router` identify the target Actor with `ate-target-actor: /`. Host and authority values are application metadata and do not select the Actor. * **atenet-router:** Substrate runs Envoy with an `ext_proc` external processor to handle Actor ingress. With `--ingress-auth-mode=static-mtls`, Envoy accepts connections only on its TLS listeners and only from clients presenting a certificate that chains to the configured client CA bundle and carries one of a static list of SPIFFE IDs; any listed client may reach any Actor. The default, `deprecated-insecure`, rejects a TLS client whose certificate fails those checks but accepts one that presents none, and also serves plaintext listeners that accept any client. The ext_proc reads and validates `ate-target-actor`, calls the Substrate API to resume the Actor and obtain its current worker assignment, and selects that worker as a dynamic backend. It overwrites the routing header before forwarding. The router then connects with mTLS to `atunnel`; `atunnel` validates the router identity and authorizes the header against the Actor currently assigned to that worker. * **Object Storage:** Used to store actor snapshots. @@ -70,7 +70,7 @@ While it's probably unlikely that Substrate, which is expected to be an internal | Threat ID | Priority | Threat | Mitigating Invariants | Suggested Concrete Mitigations | Notes | | :---- | :---- | :---- | :---- | :---- | :---- | -| T-04 | Critical | Access to the internal network allows arbitrary actions to be performed on ate-apiserver, atelet, substrate backend database, etc. | All system components must have basic mutual authentication and authorization, and communicate over TLS. All clients (including end users and actors) must be authenticated and authorized. Unauthenticated traffic must be rejected. Where possible, use firewalls to block access from ranges that have no business communicating with Substrate components. | mTLS or other secure channel (e.g. UDS) between networked system components (ate-apiserver, atelet, ateom, etc); each atelet has a unique identity cryptographically tied to the node identity. The ingress gateway should check client permissions before resuming actors or forwarding traffic to actors. The only component authorized to connect directly to the backend database should be ate-apiserver. Use Kubernetes NetworkPolicy to block access to the Substrate API and other core components by default. | | +| T-04 | Critical | Access to the internal network allows arbitrary actions to be performed on ate-apiserver, atelet, substrate backend database, etc. | All system components must have basic mutual authentication and authorization, and communicate over TLS. All clients (including end users and actors) must be authenticated and authorized. Unauthenticated traffic must be rejected. Where possible, use firewalls to block access from ranges that have no business communicating with Substrate components. | mTLS or other secure channel (e.g. UDS) between networked system components (ate-apiserver, atelet, ateworker, etc); each atelet has a unique identity cryptographically tied to the node identity. The ingress gateway should check client permissions before resuming actors or forwarding traffic to actors. The only component authorized to connect directly to the backend database should be ate-apiserver. Use Kubernetes NetworkPolicy to block access to the Substrate API and other core components by default. | | | T-05 | High | Privilege escalation via access to sensitive labels. | If Substrate offers its own resource labeling mechanism, it must also offer a way to authorize label updates on a per-label basis. | Substrate authorization system requires explicit authorization to update metadata, separate from updating resource body. Substrate authorization system supports per-label authorization rules. | Plenty of attacks in K8s were possible because labels had semantic meaning, but the permission model could implicitly granted access to modify labels, even if it was inappropriate. For example, /status subresource allows label updates. Substrate shouldn't repeat this mistake. | | T-06 | High | Attacker gains control of the Substrate API server or an ingress/egress gateway. | Isolate the control plane from the data plane, and isolate data plane ingress from sandboxes. | Don't co-locate ate-apiserver or other control-plane components on the same machines as the untrusted sandboxes. This also supports control-plane / data-plane isolation from a reliability and performance perspective. Consider running any gateway that enables direct interaction with sandboxes on a separate VM from the sandboxes. Consider using a zero-trust architecture where traffic is encrypted and authenticated end-to-end and routed via a mesh. | While the sandbox is trusted as an isolation layer, the residual risk is typically not worth the marginal cost savings of colocating, and isolation is recommend to avoid noisy neighbor problems or other reliability issues that stem from mixing the control-plane and data-plane, especially for large scale deployments of Substrate. | | T-07 | High | Attacker who can create ActorTemplates specifies malicious runtime. | Ensure available runtime can only be configured by administrators. | Consider a mechanism like RuntimeClass to decouple configuration of available runtimes from consumption of available runtimes. | | @@ -93,9 +93,9 @@ While it's probably unlikely that Substrate, which is expected to be an internal | :---- | :---- | :---- | :---- | :---- | :---- | | T-15 | Critical | Malicious actor gains access to the underlying node or other actors via container breakout (Linux local privilege escalation) | Actors are always sandboxed using a hardened sandbox solution like gvisor or microvm. Traditional containers are not a secure sandbox. | Use Gvisor for strong container isolation. Run sentry in a user namespace and pivot\_root to limit broader file system access if the gvisor boundary is broken. Use Seccomp for sentry/gofer as well. Limit capabilities granted to directfs/sentry Any sidecar containers running alongside actors or warmpool pods should be as low privileged as possible. Sandbox lifecycle must be controlled from outside the sandbox | | | T-16 | Critical | Malicious actor gains access to the underlying node via node-local endpoints exposed on the network | Network policies must prevent the actor from accessing system services on the node (e.g. instance metadata, host network namespace, host network interface). | Enable network namespacing, and do not provide host network access to any workload. Any host services should have targeted ingress policies so that only the correct clients even have network access to those services | | -| T-17 | Critical | Malicious actor gains access to other actors via network | Network policies must deny ingress and egress by default, and selectively allow access to specific actors only when necessary. Network policies must be synchronized with actor lifecycle. | Implement default-deny network policy that blocks any cross-communication between actors (this could be implemented by ateom instead of Kubernetes, to speed up policy updates when actor groups do need to communicate). | | +| T-17 | Critical | Malicious actor gains access to other actors via network | Network policies must deny ingress and egress by default, and selectively allow access to specific actors only when necessary. Network policies must be synchronized with actor lifecycle. | Implement default-deny network policy that blocks any cross-communication between actors (this could be implemented by ateworker instead of Kubernetes, to speed up policy updates when actor groups do need to communicate). | | | T-18 | Critical | Malicious actor gains access to the underlying node or other actors via filesystem | Filesystem access is limited to the actor's local fs and remote filesystems the actor is directly, explicitly authorized to access. | Ensure that actor access to the filesystem is protected by, for example, mapping each actor to a unique Linux uid and using filesystem permissions, or by using user namespaces to isolate actor filesystem access on the same host. Ensure that actor management APIs and filesystem setup protects against traversal via symlinks, mount trickery, etc. | Possible vectors: Arbitrary file read vuln in fs implementation. Symlink traversal vulns. Vuln in OCI image unpacking. Path traversal attacks in mount configuration. Shared directories only gated via filesystem permissions \+ workers running as root or privileged. | -| T-19 | Critical | Malicious actor gains access to the underlying Worker Pod or node via local Substrate services (atelet, ateom, etc). | Actor access to system services must be denied or explicitly scoped to the actor. Local services must run with the fewest privileges possible to limit the ability for an actor to escalate privilege. | Limit actor access to system services. Ensure system services are aware of actor identity, and authenticate and authorize actors when access is required. | | +| T-19 | Critical | Malicious actor gains access to the underlying Worker Pod or node via local Substrate services (atelet, ateworker, etc). | Actor access to system services must be denied or explicitly scoped to the actor. Local services must run with the fewest privileges possible to limit the ability for an actor to escalate privilege. | Limit actor access to system services. Ensure system services are aware of actor identity, and authenticate and authorize actors when access is required. | | | T-20 | Critical | Malicious actor gains access to the underlying node or other actors via Substrate APIs (remote ate-apiserver) | Either actors are completely unable to access Substrate APIs, or actor access is authenticated and authorized such that the actor cannot escalate beyond its intended scope. Especially prevent self-modification, which was a common escalation path in K8s. | Do not allow actors or workers to self-modify, e.g. by: reading or writing their own snapshots self-labeling or other self-manipulations of KRM or ATE resources Logic to modify resource definitions including actors, workers, etc lives in the control plane, rather than the data plane, as much as possible. | | | T-21 | Critical | Attacker escalates to host via excessive Worker Pod permissions | Ensure Worker pods use the minimum necessary permissions to set manage sandbox lifecycle | Deprivilege the worker Pods. Run them as non-root or in a user namespace, without privileged, and with only required caps/devices. | Possible example, accidentally putting things in the same cgroup: [https://github.com/agent-substrate/substrate/issues/288](https://github.com/agent-substrate/substrate/issues/288) | | T-22 | Critical | Malicious actor gains access to the underlying node or other actors via Kubernetes APIs | Either actors are completely unable to access the Kubernetes APIs, or actor access is authenticated and authorized such that the actor cannot escalate beyond its intended scope. **Strong preference on blocking actor access to Kubernetes.** | Block network access to the kubernetes API Ensure the default kubernetes service account token in each pod has 0 privileges | | diff --git a/docs/threats.json b/docs/threats.json index 4ee01cc49f..7f6734b237 100644 --- a/docs/threats.json +++ b/docs/threats.json @@ -28,7 +28,7 @@ "priority": "Critical", "threat": "Access to the internal network allows arbitrary actions to be performed on ate-apiserver, atelet, substrate backend database, etc.", "mitigating_invariants": "All system components must have basic mutual authentication and authorization, and communicate over TLS. All clients (including end users and actors) must be authenticated and authorized. Unauthenticated traffic must be rejected. Where possible, use firewalls to block access from ranges that have no business communicating with Substrate components.", - "suggested_concrete_mitigations": "mTLS or other secure channel (e.g. UDS) between networked system components (ate-apiserver, atelet, ateom, etc); each atelet has a unique identity cryptographically tied to the node identity. The ingress gateway should check client permissions before resuming actors or forwarding traffic to actors. The only component authorized to connect directly to the backend database should be ate-apiserver. Use Kubernetes NetworkPolicy to block access to the Substrate API and other core components by default.", + "suggested_concrete_mitigations": "mTLS or other secure channel (e.g. UDS) between networked system components (ate-apiserver, atelet, ateworker, etc); each atelet has a unique identity cryptographically tied to the node identity. The ingress gateway should check client permissions before resuming actors or forwarding traffic to actors. The only component authorized to connect directly to the backend database should be ate-apiserver. Use Kubernetes NetworkPolicy to block access to the Substrate API and other core components by default.", "notes": "" }, { @@ -132,7 +132,7 @@ "priority": "Critical", "threat": "Malicious actor gains access to other actors via network", "mitigating_invariants": "Network policies must deny ingress and egress by default, and selectively allow access to specific actors only when necessary. Network policies must be synchronized with actor lifecycle.", - "suggested_concrete_mitigations": "Implement default-deny network policy that blocks any cross-communication between actors (this could be implemented by ateom instead of Kubernetes, to speed up policy updates when actor groups do need to communicate).", + "suggested_concrete_mitigations": "Implement default-deny network policy that blocks any cross-communication between actors (this could be implemented by ateworker instead of Kubernetes, to speed up policy updates when actor groups do need to communicate).", "notes": "" }, { @@ -146,7 +146,7 @@ { "threat_id": "T-19", "priority": "Critical", - "threat": "Malicious actor gains access to the underlying Worker Pod or node via local Substrate services (atelet, ateom, etc).", + "threat": "Malicious actor gains access to the underlying Worker Pod or node via local Substrate services (atelet, ateworker, etc).", "mitigating_invariants": "Actor access to system services must be denied or explicitly scoped to the actor. Local services must run with the fewest privileges possible to limit the ability for an actor to escalate privilege.", "suggested_concrete_mitigations": "Limit actor access to system services. Ensure system services are aware of actor identity, and authenticate and authorize actors when access is required.", "notes": "" diff --git a/docs/upgrade.md b/docs/upgrade.md index 4f9fdda840..7f5e73d0aa 100644 --- a/docs/upgrade.md +++ b/docs/upgrade.md @@ -119,7 +119,7 @@ Before you patch any pool, list the pools and save the list for rollback. Each p kubectl get workerpools -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.sandboxClasses[0].name,IMAGE:.spec.workerImage' | tee ~/ate-upgrade/before-workerpools.txt ``` -If you build from source, build and push both worker images once. This touches nothing in the cluster and ends by printing `ateom-gvisor: ` and `ateom-microvm: `: +If you build from source, build and push both worker images once. This touches nothing in the cluster and ends by printing `ateworker-gvisor: ` and `ateworker-microvm: `: ```bash go run ./cmd/ate-setup publish worker-images --config ~/ate-upgrade/new.yaml | tee ~/ate-upgrade/worker-images.txt @@ -135,20 +135,20 @@ Then move the pools, starting with a small pool of each CLASS that you can affor CLASS=$(kubectl -n $NS get workerpool $WORKERPOOL -o jsonpath='{.spec.sandboxClasses[0].name}') ``` -2. Set its new image. From source, take the ref after `ateom-$CLASS: ` in `~/ate-upgrade/worker-images.txt`: +2. Set its new image. From source, take the ref after `ateworker-$CLASS: ` in `~/ate-upgrade/worker-images.txt`: ```bash - NEW_IMAGE=$(sed -n "s/^ateom-$CLASS: //p" ~/ate-upgrade/worker-images.txt) + NEW_IMAGE=$(sed -n "s/^ateworker-$CLASS: //p" ~/ate-upgrade/worker-images.txt) ``` - With prebuilt images, take `ateom-$CLASS` from your image repo at the new tag, pinned by digest: + With prebuilt images, take `ateworker-$CLASS` from your image repo at the new tag, pinned by digest: ```bash - NEW_IMAGE=$(sed -n 's/^ repo: //p' ~/ate-upgrade/new.yaml)/ateom-$CLASS:$NEW_RELEASE + NEW_IMAGE=$(sed -n 's/^ repo: //p' ~/ate-upgrade/new.yaml)/ateworker-$CLASS:$NEW_RELEASE NEW_IMAGE=$NEW_IMAGE@$(crane digest $NEW_IMAGE) ``` -3. Check that `echo "$NS/$WORKERPOOL $CLASS $NEW_IMAGE"` shows `ateom-` plus the pool's CLASS and a full `@sha256:` digest. A bad image still removes the pool's first old worker. +3. Check that `echo "$NS/$WORKERPOOL $CLASS $NEW_IMAGE"` shows `ateworker-` plus the pool's CLASS and a full `@sha256:` digest. A bad image still removes the pool's first old worker. 4. Patch the worker pool and wait until ate-controller moves its Deployment to the new image: ```bash diff --git a/hack/cleanup-e2e.sh b/hack/cleanup-e2e.sh index 906225a727..2c96ca35e5 100755 --- a/hack/cleanup-e2e.sh +++ b/hack/cleanup-e2e.sh @@ -18,7 +18,7 @@ set -o errexit -o nounset -o pipefail # Deletes the namespaces a failed e2e run left behind. # -# A failed suite keeps its namespaces so the worker pods — and the ateom logs +# A failed suite keeps its namespaces so the worker pods — and the worker logs # inside them, which is where an actor failure is actually explained — survive # long enough to read (see RetainNamespaces in internal/e2e/namespace.go). # Nothing reclaims them afterwards, and each holds a WorkerPool's worth of diff --git a/hack/install-ate.sh b/hack/install-ate.sh index 4031b8bdaf..1438b1f8ee 100755 --- a/hack/install-ate.sh +++ b/hack/install-ate.sh @@ -90,6 +90,7 @@ usage() { echo "Overall infrastructure (all infrastructure components):" echo "" echo " --deploy-ate-system Deploy core system (CRDs, atelet, apiserver)" + echo " --preview GATES Enable preview gates (comma-separated, ALL for all, '' for none)" echo " --setup-csi[=DRIVER] Setup CSI driver: nfs, hostpath, both, none (default: none;" echo " a bare --setup-csi means nfs; hostpath is Kind only)" echo " --delete-ate-system Delete core system" @@ -103,6 +104,12 @@ usage() { echo " --fake-data-plane Install fake-atelet in place of atelet and fake-workersync in place of" echo " ate-controller, for control-plane benchmarks. Actors do not exist; never" echo " use it on a cluster that serves real actors" + echo " --ingress-auth-mode=deprecated-insecure|static-mtls" + echo " How the ingress router authenticates clients (default: deprecated-insecure)." + echo " Both check a client certificate on the TLS ports against the podidentity CA and" + echo " the SPIFFE IDs listed in manifests/ate-install/atenet-router.yaml." + echo " deprecated-insecure also accepts TLS clients without one and serves plaintext" + echo " ports; static-mtls requires the certificate and --atenet-dataplane=envoy." echo " --rollout-timeout DURATION Per-workload readiness wait timeout, kubectl-style Go duration (default: 60s)" echo " --otlp-endpoint URL Send all control plane telemetry to URL, not to the cluster default (see benchmarking/telemetry/README.md)" echo "" @@ -271,6 +278,14 @@ for ((i = 0; i < ${#prescan_args[@]}; i++)); do fi GLOBAL_FLAGS+=("--atenet-dataplane=${prescan_args[$((i + 1))]}") ;; + --preview=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; + --preview) + if (( i + 1 >= ${#prescan_args[@]} )); then + echo "Error: --preview requires a value" >&2 + exit 1 + fi + GLOBAL_FLAGS+=("--preview=${prescan_args[$((i + 1))]}") + ;; --experimental-additional-egress-extproc-service=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; @@ -307,6 +322,14 @@ for ((i = 0; i < ${#prescan_args[@]}; i++)); do ;; --cordon-control-plane|--cordon-control-plane=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; --fake-data-plane|--fake-data-plane=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; + --ingress-auth-mode=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; + --ingress-auth-mode) + if (( i + 1 >= ${#prescan_args[@]} )); then + echo "Error: --ingress-auth-mode requires deprecated-insecure or static-mtls" >&2 + exit 1 + fi + GLOBAL_FLAGS+=("--ingress-auth-mode=${prescan_args[$((i + 1))]}") + ;; --rollout-timeout=*) GLOBAL_FLAGS+=("${prescan_args[i]}") ;; --rollout-timeout) if (( i + 1 >= ${#prescan_args[@]} )); then @@ -363,16 +386,18 @@ while [[ "$#" -gt 0 ]]; do # Captured in the pre-scan above; matched here only so the `*)` branch does # not reject them, and so a separated value is consumed with its flag. --atenet-dataplane|--podcert-workers-per-signer|--rollout-timeout|--otlp-endpoint) shift ;; - --cluster-size) shift ;; + --cluster-size|--ingress-auth-mode) shift ;; --experimental-additional-egress-extproc-service) shift ;; --credential-provider) shift ;; --benchmark-worker-count|--benchmark-sandbox-class|--benchmark-actor-memory) shift ;; --atenet-dataplane=*|--podcert-workers-per-signer=*|--rollout-timeout=*|--otlp-endpoint=*) ;; - --cluster-size=*|--cordon-control-plane|--cordon-control-plane=*) ;; --fake-data-plane|--fake-data-plane=*) ;; + --cluster-size=*|--cordon-control-plane|--cordon-control-plane=*|--ingress-auth-mode=*) ;; --experimental-additional-egress-extproc-service=*) ;; --credential-provider=*) ;; --benchmark-worker-count=*|--benchmark-sandbox-class=*|--benchmark-actor-memory=*) ;; + --preview) shift ;; + --preview=*) ;; --deploy-ate-system) ate_setup deploy ate-system "--setup-csi=${SETUP_CSI}" ;; --setup-csi=*) ate_setup setup csi "${SETUP_CSI}" ;; diff --git a/hack/microvm-assets/README.md b/hack/microvm-assets/README.md index 3fe5b35534..c69f80ddc5 100644 --- a/hack/microvm-assets/README.md +++ b/hack/microvm-assets/README.md @@ -1,7 +1,7 @@ # Micro-VM runtime assets + counter demo (kind, fetch-not-bake) -The `microvm` runtime (`cmd/ateom-microvm`, kata + cloud-hypervisor) fetches its -toolchain at runtime — nothing kata-specific is baked into the worker image. ateom drives +The `microvm` runtime (`cmd/ateworker-microvm`, kata + cloud-hypervisor) fetches its +toolchain at runtime — nothing kata-specific is baked into the worker image. ateworker drives the kata-agent directly (no kata shim, no containerd). Each actor container's rootfs is an overlay of a read-only lower (the OCI image, served into the guest over virtio-fs by `virtiofsd`) and a writable upper on a guest tmpfs, so `virtiofsd` is part of the asset diff --git a/hack/microvm-assets/assemble.sh b/hack/microvm-assets/assemble.sh index 7ca4eca5a4..68483b4de8 100755 --- a/hack/microvm-assets/assemble.sh +++ b/hack/microvm-assets/assemble.sh @@ -15,7 +15,7 @@ # limitations under the License. # Assemble the micro-VM (kata + cloud-hypervisor) runtime asset set that -# ateom-microvm fetches at runtime (fetch-not-bake). Run this on a Linux +# ateworker-microvm fetches at runtime (fetch-not-bake). Run this on a Linux # host of the TARGET arch. # # Produces, under $OUT, the four assets named as the SandboxConfig expects: @@ -24,7 +24,7 @@ # paste their sha256 sums into the manifest # (manifests/microvm/sandboxconfig-microvm.yaml.tmpl). # -# ateom drives the kata-agent directly (the kata containerd shim is NOT an asset). The +# ateworker drives the kata-agent directly (the kata containerd shim is NOT an asset). The # actor rootfs is overlay(virtio-fs RO lower + guest-tmpfs upper), so virtiofsd IS an # asset. CH's restore handshake hangs against virtiofsd v1.13.3, which kata bundled up # to and including 4.0.0; kata 4.1.0 bundles v1.14.0, the first release carrying the @@ -99,7 +99,7 @@ tar --zstd -xf kata-static.tar.zst -C kata KROOT="kata/opt/kata" cp "$(readlink -f "${KROOT}/share/kata-containers/vmlinux.container")" "${OUT}/vmlinux" -# rootfs.img is attached over virtio-pmem and mounted with DAX (cmd/ateom-microvm/run.go). +# rootfs.img is attached over virtio-pmem and mounted with DAX (cmd/ateworker-microvm/run.go). # The image must meet the following requirements: # - a size that is a multiple of 2 MiB (cloud-hypervisor) # - partition 1 aligned to the guest page size diff --git a/hack/run-e2e.sh b/hack/run-e2e.sh index 8635aa2968..3c02ebe776 100755 --- a/hack/run-e2e.sh +++ b/hack/run-e2e.sh @@ -46,6 +46,7 @@ Common E2E Flags (passed after -args): --kube-config Path to kubeconfig file --kube-context Kubernetes context to use --storage-class StorageClass to use for external volume tests (default: csi-nfs-sc) + --preview Preview gates the cluster was installed with ("ALL" for all) Common Go Test Flags (passed before -args): -run Run only tests matching regexp diff --git a/hack/setup-csi-hostpath-kind.sh b/hack/setup-csi-hostpath-kind.sh index f0a0a6e95b..c412fc89ce 100755 --- a/hack/setup-csi-hostpath-kind.sh +++ b/hack/setup-csi-hostpath-kind.sh @@ -97,11 +97,11 @@ spec: containers: - name: hostpath volumeMounts: - - name: ateom-dir + - name: ate-base mountPath: /var/lib/ate mountPropagation: Bidirectional volumes: - - name: ateom-dir + - name: ate-base hostPath: path: /var/lib/ate type: DirectoryOrCreate diff --git a/hack/setup-csi-nfs-kind.sh b/hack/setup-csi-nfs-kind.sh index eeec5a61ac..624d824ad9 100755 --- a/hack/setup-csi-nfs-kind.sh +++ b/hack/setup-csi-nfs-kind.sh @@ -47,11 +47,11 @@ spec: containers: - name: nfs volumeMounts: - - name: ateom-dir + - name: ate-base mountPath: /var/lib/ate mountPropagation: Bidirectional volumes: - - name: ateom-dir + - name: ate-base hostPath: path: /var/lib/ate type: DirectoryOrCreate diff --git a/hack/verify/golangci-lint.sh b/hack/verify/golangci-lint.sh index 4ba6a7bcad..196dc54cf7 100755 --- a/hack/verify/golangci-lint.sh +++ b/hack/verify/golangci-lint.sh @@ -23,7 +23,7 @@ cd "${ROOT}" # executable on the developer's machine, then invoke it with GOOS=linux so # the linter analyzes the code the same way it would on the Linux CI # runners. Without the GOOS=linux override, platform-gated packages -# (notably the netlink bindings used by ateom-gvisor) fail to typecheck on +# (notably the netlink bindings used by ateworker-gvisor) fail to typecheck on # macOS and golangci-lint's typecheck fail-stop suppresses all other # findings in the affected files. BIN="$("${ROOT}"/hack/run-tool.sh --print-bin-path golangci-lint)" diff --git a/internal/actorevent/actorevent.go b/internal/actorevent/actorevent.go index 535110ef1d..b189583807 100644 --- a/internal/actorevent/actorevent.go +++ b/internal/actorevent/actorevent.go @@ -89,7 +89,7 @@ var identityKeys = []string{ // Three names. A crash is its own because it has a different severity; there // is no name per state, because ate.actor.state already says which transition -// happened. UsageSampled is the ateoms' measurement record. +// happened. UsageSampled is the workers' measurement record. var ( StateChanged = Event{ Name: "ate.actor.state_changed", diff --git a/internal/actoridjwt/actoridjwt.go b/internal/actoridjwt/actoridjwt.go index a0701ba0c4..b3bb9df14e 100644 --- a/internal/actoridjwt/actoridjwt.go +++ b/internal/actoridjwt/actoridjwt.go @@ -33,10 +33,10 @@ const ( // KindActor is the actor itself. Subject "actor//", // claims under "actor.ate.dev". KindActor - // KindAteomForActor is an ateom, or another Substrate component, acting - // on behalf of the actor. Subject "ateom-for-actor//", - // claims under "ateom-for-actor.ate.dev". - KindAteomForActor + // KindWorkerForActor is a worker, or another Substrate component, acting + // on behalf of the actor. Subject "worker-for-actor//", + // claims under "worker-for-actor.ate.dev". + KindWorkerForActor ) // SubjectPrefix is the first segment of the sub claim for this kind. @@ -44,8 +44,8 @@ func (k Kind) SubjectPrefix() string { switch k { case KindActor: return "actor" - case KindAteomForActor: - return "ateom-for-actor" + case KindWorkerForActor: + return "worker-for-actor" } return "" } @@ -56,8 +56,8 @@ func (k Kind) ClaimName() string { switch k { case KindActor: return "actor.ate.dev" - case KindAteomForActor: - return "ateom-for-actor.ate.dev" + case KindWorkerForActor: + return "worker-for-actor.ate.dev" } return "" } @@ -98,18 +98,18 @@ type WireClaims struct { // Substrate claims, under the claim name for the token's kind. Exactly // one is set. - Actor *WireSubstrateClaims `json:"actor.ate.dev,omitempty"` - AteomForActor *WireSubstrateClaims `json:"ateom-for-actor.ate.dev,omitempty"` + Actor *WireSubstrateClaims `json:"actor.ate.dev,omitempty"` + WorkerForActor *WireSubstrateClaims `json:"worker-for-actor.ate.dev,omitempty"` } // Substrate returns the token's kind and its Substrate claims, or ok false if // the token carries neither claim or both. func (w *WireClaims) Substrate() (kind Kind, claims *WireSubstrateClaims, ok bool) { switch { - case w.Actor != nil && w.AteomForActor == nil: + case w.Actor != nil && w.WorkerForActor == nil: return KindActor, w.Actor, true - case w.AteomForActor != nil && w.Actor == nil: - return KindAteomForActor, w.AteomForActor, true + case w.WorkerForActor != nil && w.Actor == nil: + return KindWorkerForActor, w.WorkerForActor, true } return 0, nil, false } @@ -143,10 +143,10 @@ func ClaimsToWire(claims *Claims) (*WireClaims, error) { switch claims.Kind { case KindActor: wire.Actor = substrate - case KindAteomForActor: - wire.AteomForActor = substrate + case KindWorkerForActor: + wire.WorkerForActor = substrate default: - return nil, fmt.Errorf("token kind %d is not one of KindActor or KindAteomForActor", claims.Kind) + return nil, fmt.Errorf("token kind %d is not one of KindActor or KindWorkerForActor", claims.Kind) } return wire, nil diff --git a/internal/actoridjwt/actoridjwt_test.go b/internal/actoridjwt/actoridjwt_test.go index 9b68bee517..8cd928b844 100644 --- a/internal/actoridjwt/actoridjwt_test.go +++ b/internal/actoridjwt/actoridjwt_test.go @@ -33,8 +33,8 @@ func TestClaimsToWire_ClaimNamePerKind(t *testing.T) { wantClaim string otherClaim string }{ - {KindActor, "actor.ate.dev", "ateom-for-actor.ate.dev"}, - {KindAteomForActor, "ateom-for-actor.ate.dev", "actor.ate.dev"}, + {KindActor, "actor.ate.dev", "worker-for-actor.ate.dev"}, + {KindWorkerForActor, "worker-for-actor.ate.dev", "actor.ate.dev"}, } { claims := base claims.Kind = tc.kind @@ -78,7 +78,7 @@ func TestClaimsToWire_RejectsUnspecifiedKind(t *testing.T) { func TestWireClaims_Substrate(t *testing.T) { actor := &WireSubstrateClaims{Atespace: "a", ActorName: "n", ActorUID: "u"} - ateom := &WireSubstrateClaims{Atespace: "a2", ActorName: "n2", ActorUID: "u2"} + worker := &WireSubstrateClaims{Atespace: "a2", ActorName: "n2", ActorUID: "u2"} for _, tc := range []struct { name string wire WireClaims @@ -87,9 +87,9 @@ func TestWireClaims_Substrate(t *testing.T) { wantClaims *WireSubstrateClaims }{ {name: "neither claim", wire: WireClaims{}}, - {name: "both claims", wire: WireClaims{Actor: actor, AteomForActor: ateom}}, + {name: "both claims", wire: WireClaims{Actor: actor, WorkerForActor: worker}}, {name: "actor", wire: WireClaims{Actor: actor}, wantOK: true, wantKind: KindActor, wantClaims: actor}, - {name: "ateom-for-actor", wire: WireClaims{AteomForActor: ateom}, wantOK: true, wantKind: KindAteomForActor, wantClaims: ateom}, + {name: "worker-for-actor", wire: WireClaims{WorkerForActor: worker}, wantOK: true, wantKind: KindWorkerForActor, wantClaims: worker}, } { t.Run(tc.name, func(t *testing.T) { kind, claims, ok := tc.wire.Substrate() @@ -116,7 +116,7 @@ func TestKind_Names(t *testing.T) { }{ {KindUnspecified, "", ""}, {KindActor, "actor", "actor.ate.dev"}, - {KindAteomForActor, "ateom-for-actor", "ateom-for-actor.ate.dev"}, + {KindWorkerForActor, "worker-for-actor", "worker-for-actor.ate.dev"}, } { if got := tc.kind.SubjectPrefix(); got != tc.prefix { t.Errorf("Kind(%d).SubjectPrefix() = %q, want %q", tc.kind, got, tc.prefix) diff --git a/internal/actorlog/logger.go b/internal/actorlog/logger.go index 47885520c9..ceddc3d254 100644 --- a/internal/actorlog/logger.go +++ b/internal/actorlog/logger.go @@ -13,7 +13,7 @@ // limitations under the License. // Package actorlog provides structured JSON logging for actor sandboxes shared -// by the gVisor and micro-VM ateom runtimes. It forwards an actor container's +// by the gVisor and micro-VM worker runtimes. It forwards an actor container's // stdout/stderr to the worker pod's stdout, annotated with the ate.* identity // labels from internal/ateattr, and emits synthetic actor lifecycle events. // diff --git a/internal/actorlog/logger_test.go b/internal/actorlog/logger_test.go index 8d86caaa49..c9f374f387 100644 --- a/internal/actorlog/logger_test.go +++ b/internal/actorlog/logger_test.go @@ -150,7 +150,7 @@ func TestWrapContainerLogs_JSONInput(t *testing.T) { } // TestWrapContainerLogs_ActorTraceContextPassthrough covers the only way an actor -// log line can join a trace: the actor emitting the spec fields itself. ateom +// log line can join a trace: the actor emitting the spec fields itself. worker // cannot supply them, because one forwarder covers a whole stream. func TestWrapContainerLogs_ActorTraceContextPassthrough(t *testing.T) { const ( diff --git a/internal/ateattr/ateattr.go b/internal/ateattr/ateattr.go index c9c4bce479..ce27d09ee8 100644 --- a/internal/ateattr/ateattr.go +++ b/internal/ateattr/ateattr.go @@ -186,7 +186,7 @@ const ( ) // SandboxOOMKillsKey is memory.events' oom_kill for the sandbox cgroup leaf, on -// the record ateom-gvisor writes when a hosted actor's sandbox has no process +// the record ateworker-gvisor writes when a hosted actor's sandbox has no process // left. Logs only: it is only ever recorded beside actor identity. const SandboxOOMKillsKey = attribute.Key("ate.sandbox.oom_kills") @@ -198,7 +198,7 @@ const ( StatsKindFinal = "final" ) -// Values for StatsSourceKey, mirroring ateompb.StatsSource. The two sources do +// Values for StatsSourceKey, mirroring ateworkerpb.StatsSource. The two sources do // not measure the same thing (the cgroup source charges the sandbox runtime's // overhead along with the workload, the guest-agent source sees only the // workload's containers), so rollups must group by this key rather than sum @@ -352,13 +352,13 @@ func SnapshotFidelityValue(fidelity ateletpb.SnapshotFidelity) string { // with the asset fetch and OCI unpack), so they are independent observations, // not a partition of Total: summing across them is meaningless. const ( - SnapshotPhaseVolumeMount = "volume_mount" - SnapshotPhaseManifestFetch = "manifest_fetch" - SnapshotPhaseSandboxAssets = "sandbox_assets" - SnapshotPhaseDownload = "download" - SnapshotPhaseOCIUnpack = "oci_unpack" - SnapshotPhaseAteomRestore = "ateom_restore" - SnapshotPhaseAteomCheckpoint = "ateom_checkpoint" + SnapshotPhaseVolumeMount = "volume_mount" + SnapshotPhaseManifestFetch = "manifest_fetch" + SnapshotPhaseSandboxAssets = "sandbox_assets" + SnapshotPhaseDownload = "download" + SnapshotPhaseOCIUnpack = "oci_unpack" + SnapshotPhaseWorkerRestore = "worker_restore" + SnapshotPhaseWorkerCheckpoint = "worker_checkpoint" // Persist is one step with two destinations (upload for external, rename // for local); SnapshotKindKey already says which. SnapshotPhasePersist = "persist" diff --git a/internal/ateattr/ateattr_test.go b/internal/ateattr/ateattr_test.go index e4fe0f8baf..b57f8275e1 100644 --- a/internal/ateattr/ateattr_test.go +++ b/internal/ateattr/ateattr_test.go @@ -406,8 +406,8 @@ func TestMetricLabelValues(t *testing.T) { {SnapshotPhaseSandboxAssets, "sandbox_assets"}, {SnapshotPhaseDownload, "download"}, {SnapshotPhaseOCIUnpack, "oci_unpack"}, - {SnapshotPhaseAteomRestore, "ateom_restore"}, - {SnapshotPhaseAteomCheckpoint, "ateom_checkpoint"}, + {SnapshotPhaseWorkerRestore, "worker_restore"}, + {SnapshotPhaseWorkerCheckpoint, "worker_checkpoint"}, {SnapshotPhasePersist, "persist"}, {SnapshotPhaseTotal, "total"}, diff --git a/internal/ateclient/builder.go b/internal/ateclient/builder.go index c8a05b3d69..45268c0dcc 100644 --- a/internal/ateclient/builder.go +++ b/internal/ateclient/builder.go @@ -128,6 +128,7 @@ func (c *Client) Close() { // NewClient creates a new Ate API client. If endpoint is empty, it automatically // port-forwards to the ate-api-server pod in substrate's system namespace. +// extraOpts are appended to the options used to dial the server. func NewClient(ctx context.Context, kubeconfigPath, k8sContext, endpoint, tokenFile string, traceEnabled bool, extraOpts ...grpc.DialOption) (*Client, error) { tp, err := initTracing(ctx, traceEnabled) if err != nil { diff --git a/internal/ateinterceptors/ateinterceptors_test.go b/internal/ateinterceptors/ateinterceptors_test.go index 371f04f782..f19d8bb4c9 100644 --- a/internal/ateinterceptors/ateinterceptors_test.go +++ b/internal/ateinterceptors/ateinterceptors_test.go @@ -171,9 +171,9 @@ func TestInternalServerUnaryInterceptorCodes(t *testing.T) { }, { name: "upstream status becomes Internal without its details", - handlerErr: fmt.Errorf("while calling ateom: %w", statusWithErrorInfo(t, codes.DataLoss, "FAILED_SAVE_SNAPSHOT", nil)), + handlerErr: fmt.Errorf("while calling worker: %w", statusWithErrorInfo(t, codes.DataLoss, "FAILED_SAVE_SNAPSHOT", nil)), wantCode: codes.Internal, - wantMsg: "while calling ateom: rpc error: code = DataLoss desc = boom", + wantMsg: "while calling worker: rpc error: code = DataLoss desc = boom", }, { name: "wrapped upstream status becomes Internal", @@ -368,7 +368,7 @@ func TestServerUnaryInterceptorRequestLogMasksEnvValues(t *testing.T) { }, } - _, err := ServerUnaryInterceptor(context.Background(), req, &grpc.UnaryServerInfo{FullMethod: "/atelet.AteomHerder/Run"}, func(ctx context.Context, req interface{}) (interface{}, error) { + _, err := ServerUnaryInterceptor(context.Background(), req, &grpc.UnaryServerInfo{FullMethod: "/atelet.Atelet/Run"}, func(ctx context.Context, req interface{}) (interface{}, error) { return &ateletpb.RunResponse{}, nil }) if err != nil { diff --git a/internal/ateomnet/dns/dnshandler.go b/internal/ateworker/actornet/dns/dnshandler.go similarity index 99% rename from internal/ateomnet/dns/dnshandler.go rename to internal/ateworker/actornet/dns/dnshandler.go index 103b733b00..665e5a30b0 100644 --- a/internal/ateomnet/dns/dnshandler.go +++ b/internal/ateworker/actornet/dns/dnshandler.go @@ -21,7 +21,7 @@ import ( "net" "time" - "github.com/agent-substrate/substrate/internal/ateomnet/dns/protocol" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns/protocol" "golang.org/x/net/dns/dnsmessage" ) diff --git a/internal/ateomnet/dns/dnshandler_test.go b/internal/ateworker/actornet/dns/dnshandler_test.go similarity index 99% rename from internal/ateomnet/dns/dnshandler_test.go rename to internal/ateworker/actornet/dns/dnshandler_test.go index cb7c052839..1551212e2a 100644 --- a/internal/ateomnet/dns/dnshandler_test.go +++ b/internal/ateworker/actornet/dns/dnshandler_test.go @@ -20,7 +20,7 @@ import ( "net" "testing" - "github.com/agent-substrate/substrate/internal/ateomnet/dns/protocol" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns/protocol" "golang.org/x/net/dns/dnsmessage" ) diff --git a/internal/ateomnet/dns/evaluator.go b/internal/ateworker/actornet/dns/evaluator.go similarity index 100% rename from internal/ateomnet/dns/evaluator.go rename to internal/ateworker/actornet/dns/evaluator.go diff --git a/internal/ateomnet/dns/evaluator_test.go b/internal/ateworker/actornet/dns/evaluator_test.go similarity index 100% rename from internal/ateomnet/dns/evaluator_test.go rename to internal/ateworker/actornet/dns/evaluator_test.go diff --git a/internal/ateomnet/dns/freemap/doc.go b/internal/ateworker/actornet/dns/freemap/doc.go similarity index 100% rename from internal/ateomnet/dns/freemap/doc.go rename to internal/ateworker/actornet/dns/freemap/doc.go diff --git a/internal/ateomnet/dns/freemap/freemap64k.go b/internal/ateworker/actornet/dns/freemap/freemap64k.go similarity index 100% rename from internal/ateomnet/dns/freemap/freemap64k.go rename to internal/ateworker/actornet/dns/freemap/freemap64k.go diff --git a/internal/ateomnet/dns/freemap/freemap64k_test.go b/internal/ateworker/actornet/dns/freemap/freemap64k_test.go similarity index 100% rename from internal/ateomnet/dns/freemap/freemap64k_test.go rename to internal/ateworker/actornet/dns/freemap/freemap64k_test.go diff --git a/internal/ateomnet/dns/limiter.go b/internal/ateworker/actornet/dns/limiter.go similarity index 100% rename from internal/ateomnet/dns/limiter.go rename to internal/ateworker/actornet/dns/limiter.go diff --git a/internal/ateomnet/dns/limiter_test.go b/internal/ateworker/actornet/dns/limiter_test.go similarity index 100% rename from internal/ateomnet/dns/limiter_test.go rename to internal/ateworker/actornet/dns/limiter_test.go diff --git a/internal/ateomnet/dns/pendingreq.go b/internal/ateworker/actornet/dns/pendingreq.go similarity index 97% rename from internal/ateomnet/dns/pendingreq.go rename to internal/ateworker/actornet/dns/pendingreq.go index ba5146adeb..65a04e7c5e 100644 --- a/internal/ateomnet/dns/pendingreq.go +++ b/internal/ateworker/actornet/dns/pendingreq.go @@ -21,8 +21,8 @@ import ( "sync" "time" - "github.com/agent-substrate/substrate/internal/ateomnet/dns/freemap" - "github.com/agent-substrate/substrate/internal/ateomnet/dns/protocol" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns/freemap" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns/protocol" "golang.org/x/net/dns/dnsmessage" ) diff --git a/internal/ateomnet/dns/pendingreq_test.go b/internal/ateworker/actornet/dns/pendingreq_test.go similarity index 100% rename from internal/ateomnet/dns/pendingreq_test.go rename to internal/ateworker/actornet/dns/pendingreq_test.go diff --git a/internal/ateomnet/dns/policy.go b/internal/ateworker/actornet/dns/policy.go similarity index 100% rename from internal/ateomnet/dns/policy.go rename to internal/ateworker/actornet/dns/policy.go diff --git a/internal/ateomnet/dns/policy_test.go b/internal/ateworker/actornet/dns/policy_test.go similarity index 99% rename from internal/ateomnet/dns/policy_test.go rename to internal/ateworker/actornet/dns/policy_test.go index fa1b6f4efa..aec5aabb8e 100644 --- a/internal/ateomnet/dns/policy_test.go +++ b/internal/ateworker/actornet/dns/policy_test.go @@ -25,7 +25,7 @@ import ( "github.com/google/go-cmp/cmp" "golang.org/x/net/dns/dnsmessage" - "github.com/agent-substrate/substrate/internal/ateomnet/dns/protocol" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns/protocol" "github.com/agent-substrate/substrate/internal/proto/ateletpb" ) diff --git a/internal/ateomnet/dns/protocol/doc.go b/internal/ateworker/actornet/dns/protocol/doc.go similarity index 100% rename from internal/ateomnet/dns/protocol/doc.go rename to internal/ateworker/actornet/dns/protocol/doc.go diff --git a/internal/ateomnet/dns/protocol/protocol.go b/internal/ateworker/actornet/dns/protocol/protocol.go similarity index 100% rename from internal/ateomnet/dns/protocol/protocol.go rename to internal/ateworker/actornet/dns/protocol/protocol.go diff --git a/internal/ateomnet/dns/protocol/protocol_test.go b/internal/ateworker/actornet/dns/protocol/protocol_test.go similarity index 100% rename from internal/ateomnet/dns/protocol/protocol_test.go rename to internal/ateworker/actornet/dns/protocol/protocol_test.go diff --git a/internal/ateomnet/dns/protocol/tcp.go b/internal/ateworker/actornet/dns/protocol/tcp.go similarity index 100% rename from internal/ateomnet/dns/protocol/tcp.go rename to internal/ateworker/actornet/dns/protocol/tcp.go diff --git a/internal/ateomnet/dns/protocol/tcp_test.go b/internal/ateworker/actornet/dns/protocol/tcp_test.go similarity index 100% rename from internal/ateomnet/dns/protocol/tcp_test.go rename to internal/ateworker/actornet/dns/protocol/tcp_test.go diff --git a/internal/ateomnet/dns/readme.md b/internal/ateworker/actornet/dns/readme.md similarity index 92% rename from internal/ateomnet/dns/readme.md rename to internal/ateworker/actornet/dns/readme.md index d7e0d3fba9..1b9e3aed75 100644 --- a/internal/ateomnet/dns/readme.md +++ b/internal/ateworker/actornet/dns/readme.md @@ -6,7 +6,7 @@ Actor DNS relays DNS requests from the actor to the external world. ### Relay -- `Relay` -- one per Ateom, holds upstream resolvers, and is used to create the per-Actor `Server`. +- `Relay` -- one per Ateworker, holds upstream resolvers, and is used to create the per-Actor `Server`. - `Relay.Serve()` -- creates a per-Actor `Server` that manages DNS requests for the Actor, checked against the Actor's egress policy (see [Egress policy](#egress-policy)). Returns a `*Server`. ### Server @@ -69,7 +69,7 @@ UDP and TCP call up to the DNS protocol layer (`golang.org/x/net/dns/dnsmessage` `policy.go` and `evaluator.go`: DNS bypasses the egress gateway, so the relay enforces the Actor's egress policy itself. -- `PolicySource` supplies the Actor's `EgressPolicy`. The ateom passes atunnel's per-Actor source, which caches the policy for a few seconds so a policy created, updated, or deleted while the Actor runs takes effect without a restart, and keeps serving the stale cached entry if `atelet` is temporarily unreachable. +- `PolicySource` supplies the Actor's `EgressPolicy`. The ateworker passes atunnel's per-Actor source, which caches the policy for a few seconds so a policy created, updated, or deleted while the Actor runs takes effect without a restart, and keeps serving the stale cached entry if `atelet` is temporarily unreachable. - `evaluator` (`evaluator.go`) compiles the hostname and wildcard patterns from an `EgressPolicy`'s `http`, `https`, and `tls_passthrough` rules and evaluates normalized `QNAME`s via `allowsHostname(qname)`. - `egressPolicyCheck` adapts the source to the `dnsHandler`'s `queryPolicy` hook, caching the compiled `*evaluator` for the current `*ateletpb.EgressPolicy` pointer and consulting it for every query. Fetching uses the `Server`'s context, so `Stop()` cancels a fetch in progress. - `allowQuery` evaluates a single `dnsmessage.Question` against a policy using `evaluator`. diff --git a/internal/ateomnet/dns/relay.go b/internal/ateworker/actornet/dns/relay.go similarity index 100% rename from internal/ateomnet/dns/relay.go rename to internal/ateworker/actornet/dns/relay.go diff --git a/internal/ateomnet/dns/relay_linux.go b/internal/ateworker/actornet/dns/relay_linux.go similarity index 96% rename from internal/ateomnet/dns/relay_linux.go rename to internal/ateworker/actornet/dns/relay_linux.go index 7aa6c121a8..394bf88645 100644 --- a/internal/ateomnet/dns/relay_linux.go +++ b/internal/ateworker/actornet/dns/relay_linux.go @@ -22,7 +22,7 @@ import ( "net" "strconv" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" + "github.com/agent-substrate/substrate/internal/netns" ) // Serve serves UDP and TCP DNS in the sandbox's local gateway namespace. diff --git a/internal/ateomnet/dns/relay_test.go b/internal/ateworker/actornet/dns/relay_test.go similarity index 100% rename from internal/ateomnet/dns/relay_test.go rename to internal/ateworker/actornet/dns/relay_test.go diff --git a/internal/ateomnet/dns/resolvconf.go b/internal/ateworker/actornet/dns/resolvconf.go similarity index 100% rename from internal/ateomnet/dns/resolvconf.go rename to internal/ateworker/actornet/dns/resolvconf.go diff --git a/internal/ateomnet/dns/resolvconf_test.go b/internal/ateworker/actornet/dns/resolvconf_test.go similarity index 100% rename from internal/ateomnet/dns/resolvconf_test.go rename to internal/ateworker/actornet/dns/resolvconf_test.go diff --git a/internal/ateomnet/dns/serve_linux_test.go b/internal/ateworker/actornet/dns/serve_linux_test.go similarity index 95% rename from internal/ateomnet/dns/serve_linux_test.go rename to internal/ateworker/actornet/dns/serve_linux_test.go index 3943070a6c..995e3e3fbc 100644 --- a/internal/ateomnet/dns/serve_linux_test.go +++ b/internal/ateworker/actornet/dns/serve_linux_test.go @@ -21,7 +21,7 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" + "github.com/agent-substrate/substrate/internal/netns" "github.com/agent-substrate/substrate/internal/roottest" ) diff --git a/internal/ateomnet/dns/server.go b/internal/ateworker/actornet/dns/server.go similarity index 100% rename from internal/ateomnet/dns/server.go rename to internal/ateworker/actornet/dns/server.go diff --git a/internal/ateomnet/dns/tcphandler.go b/internal/ateworker/actornet/dns/tcphandler.go similarity index 98% rename from internal/ateomnet/dns/tcphandler.go rename to internal/ateworker/actornet/dns/tcphandler.go index 93ca9883dd..e3eab1c011 100644 --- a/internal/ateomnet/dns/tcphandler.go +++ b/internal/ateworker/actornet/dns/tcphandler.go @@ -24,7 +24,7 @@ import ( "sync" "time" - "github.com/agent-substrate/substrate/internal/ateomnet/dns/protocol" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns/protocol" ) const ( diff --git a/internal/ateomnet/dns/tcphandler_test.go b/internal/ateworker/actornet/dns/tcphandler_test.go similarity index 98% rename from internal/ateomnet/dns/tcphandler_test.go rename to internal/ateworker/actornet/dns/tcphandler_test.go index 6322342bdd..d79bd266f9 100644 --- a/internal/ateomnet/dns/tcphandler_test.go +++ b/internal/ateworker/actornet/dns/tcphandler_test.go @@ -22,7 +22,7 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/internal/ateomnet/dns/protocol" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns/protocol" "golang.org/x/net/dns/dnsmessage" ) diff --git a/internal/ateomnet/dns/udphandler.go b/internal/ateworker/actornet/dns/udphandler.go similarity index 100% rename from internal/ateomnet/dns/udphandler.go rename to internal/ateworker/actornet/dns/udphandler.go diff --git a/internal/ateomnet/dns/udphandler_test.go b/internal/ateworker/actornet/dns/udphandler_test.go similarity index 100% rename from internal/ateomnet/dns/udphandler_test.go rename to internal/ateworker/actornet/dns/udphandler_test.go diff --git a/internal/ateomnet/net.go b/internal/ateworker/actornet/net.go similarity index 96% rename from internal/ateomnet/net.go rename to internal/ateworker/actornet/net.go index 5d5265ced3..96080e8959 100644 --- a/internal/ateomnet/net.go +++ b/internal/ateworker/actornet/net.go @@ -14,8 +14,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateomnet provides shared networking configuration logic for Substrate runtime agents. -package ateomnet +// Package actornet provides shared networking configuration logic for Substrate runtime agents. +package actornet import ( "fmt" diff --git a/internal/ateomnet/sandbox.go b/internal/ateworker/actornet/sandbox.go similarity index 97% rename from internal/ateomnet/sandbox.go rename to internal/ateworker/actornet/sandbox.go index a0b59acae2..3e08da95bd 100644 --- a/internal/ateomnet/sandbox.go +++ b/internal/ateworker/actornet/sandbox.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomnet +package actornet import ( "context" @@ -26,8 +26,8 @@ import ( "net/netip" "sync" - "github.com/agent-substrate/substrate/internal/ateomnet/dns" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns" + "github.com/agent-substrate/substrate/internal/netns" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/google/nftables" @@ -245,8 +245,8 @@ func installEgressRules(ns netns.Handle, egressPort uint16) error { rejectCode uint8 isPrefix func(netip.Prefix) bool }{ - {nftables.TableFamilyIPv4, "ateom-actor", 4, 16, icmpDestinationAdministrativelyProhibited, func(prefix netip.Prefix) bool { return prefix.Addr().Is4() }}, - {nftables.TableFamilyIPv6, "ateom-actor-v6", 16, 24, icmpv6DestinationAdministrativelyProhibited, func(prefix netip.Prefix) bool { return prefix.Addr().Is6() }}, + {nftables.TableFamilyIPv4, "worker-actor", 4, 16, icmpDestinationAdministrativelyProhibited, func(prefix netip.Prefix) bool { return prefix.Addr().Is4() }}, + {nftables.TableFamilyIPv6, "worker-actor-v6", 16, 24, icmpv6DestinationAdministrativelyProhibited, func(prefix netip.Prefix) bool { return prefix.Addr().Is6() }}, } { table := c.AddTable(&nftables.Table{Family: family.tableFamily, Name: family.name}) if family.tableFamily == nftables.TableFamilyIPv4 { @@ -492,7 +492,7 @@ func (s *SandboxSession) Close(ctx context.Context) error { return errs } -// SessionHolder is the one sandbox session a worker is serving, for the ateoms +// SessionHolder is the one sandbox session a worker is serving, for the workers // to share what is otherwise the same locking, replacement and dialing in both. type SessionHolder struct { mu sync.Mutex diff --git a/internal/ateomnet/sandbox_atunnel_linux_test.go b/internal/ateworker/actornet/sandbox_atunnel_linux_test.go similarity index 91% rename from internal/ateomnet/sandbox_atunnel_linux_test.go rename to internal/ateworker/actornet/sandbox_atunnel_linux_test.go index 4b68f8d7ea..3109a8d6da 100644 --- a/internal/ateomnet/sandbox_atunnel_linux_test.go +++ b/internal/ateworker/actornet/sandbox_atunnel_linux_test.go @@ -14,9 +14,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -// These integration tests use an external package so that ateomnet's own test +// These integration tests use an external package so that actornet's own test // binary does not depend on atunnel. -package ateomnet_test +package actornet_test import ( "context" @@ -25,9 +25,9 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/internal/ateomnet" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" + "github.com/agent-substrate/substrate/internal/ateworker/actornet" "github.com/agent-substrate/substrate/internal/atunnel" + "github.com/agent-substrate/substrate/internal/netns" "github.com/agent-substrate/substrate/internal/roottest" ) @@ -37,13 +37,13 @@ func TestSandboxEgressReachesAtunnelOnAnyPort(t *testing.T) { ctx := context.Background() const egressPort = 15001 - n, err := ateomnet.SetupSandboxNetwork(ctx, ateomnet.SandboxNetworkConfig{ + n, err := actornet.SetupSandboxNetwork(ctx, actornet.SandboxNetworkConfig{ ActorUID: "66666666-6666-6666-6666-666666666666", Veth: true, EgressPort: egressPort, }) if err != nil { t.Fatalf("SetupSandboxNetwork: %v", err) } - t.Cleanup(func() { ateomnet.CleanupSandboxNetwork(n) }) + t.Cleanup(func() { actornet.CleanupSandboxNetwork(n) }) listeners, err := netns.Listen(ctx, n.GatewayNetNS, []uint16{egressPort}) if err != nil { @@ -102,7 +102,7 @@ func TestSandboxLinkLocalDoesNotReachAtunnel(t *testing.T) { ctx := context.Background() const egressPort = 15001 - n, err := ateomnet.SetupSandboxNetwork(ctx, ateomnet.SandboxNetworkConfig{ + n, err := actornet.SetupSandboxNetwork(ctx, actornet.SandboxNetworkConfig{ ActorUID: "99999999-9999-9999-9999-999999999999", Veth: true, EgressPort: egressPort, @@ -110,7 +110,7 @@ func TestSandboxLinkLocalDoesNotReachAtunnel(t *testing.T) { if err != nil { t.Fatalf("SetupSandboxNetwork: %v", err) } - t.Cleanup(func() { ateomnet.CleanupSandboxNetwork(n) }) + t.Cleanup(func() { actornet.CleanupSandboxNetwork(n) }) listeners, err := netns.Listen(ctx, n.GatewayNetNS, []uint16{egressPort}) if err != nil { diff --git a/internal/ateomnet/sandbox_linux_test.go b/internal/ateworker/actornet/sandbox_linux_test.go similarity index 99% rename from internal/ateomnet/sandbox_linux_test.go rename to internal/ateworker/actornet/sandbox_linux_test.go index 4a39ac9d12..6308c39b5a 100644 --- a/internal/ateomnet/sandbox_linux_test.go +++ b/internal/ateworker/actornet/sandbox_linux_test.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomnet +package actornet import ( "context" @@ -32,7 +32,7 @@ import ( "github.com/agent-substrate/substrate/internal/nodepath" "github.com/vishvananda/netlink" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" + "github.com/agent-substrate/substrate/internal/netns" "github.com/agent-substrate/substrate/internal/roottest" ) diff --git a/internal/ateomcgroup/actor.go b/internal/ateworker/cgroup/actor.go similarity index 99% rename from internal/ateomcgroup/actor.go rename to internal/ateworker/cgroup/actor.go index 868ae51969..54e4b29f32 100644 --- a/internal/ateomcgroup/actor.go +++ b/internal/ateworker/cgroup/actor.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomcgroup +package cgroup import ( "context" diff --git a/internal/ateomcgroup/actor_linux_test.go b/internal/ateworker/cgroup/actor_linux_test.go similarity index 83% rename from internal/ateomcgroup/actor_linux_test.go rename to internal/ateworker/cgroup/actor_linux_test.go index 068ada3c11..991470cf7e 100644 --- a/internal/ateomcgroup/actor_linux_test.go +++ b/internal/ateworker/cgroup/actor_linux_test.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomcgroup +package cgroup import ( "context" @@ -35,7 +35,7 @@ func TestActorLeafStartsProcessesInside(t *testing.T) { root := ownCgroup(t) self := "/" + strings.TrimPrefix(root, Root) - leaf, err := openActorLeaf(root, "ateomcgroup-test", 500) + leaf, err := openActorLeaf(root, "cgroup-test", 500) if err != nil { t.Skipf("cannot create a cgroup here: %v", err) } @@ -51,10 +51,10 @@ func TestActorLeafStartsProcessesInside(t *testing.T) { if err != nil { t.Fatal(err) } - if want := "0::" + filepath.Join(self, "ateomcgroup-test"); !strings.Contains(string(got), want) { + if want := "0::" + filepath.Join(self, "cgroup-test"); !strings.Contains(string(got), want) { t.Errorf("process cgroup = %q, want %q", got, want) } - if err := removeActorLeaf(root, "ateomcgroup-test"); err != nil { + if err := removeActorLeaf(root, "cgroup-test"); err != nil { t.Errorf("removing the leaf after its process exited: %v", err) } } @@ -63,7 +63,7 @@ func TestActorLeafStartsProcessesInside(t *testing.T) { func TestKillActorLeaf(t *testing.T) { roottest.Require(t, "creates cgroups") root := ownCgroup(t) - leaf, err := openActorLeaf(root, "ateomcgroup-kill-test", 0) + leaf, err := openActorLeaf(root, "cgroup-kill-test", 0) if err != nil { t.Skipf("cannot create a cgroup here: %v", err) } @@ -76,15 +76,15 @@ func TestKillActorLeaf(t *testing.T) { _ = leaf.Close() ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() - if err := KillLeafUnder(ctx, root, "ateomcgroup-kill-test"); err != nil { + if err := KillLeafUnder(ctx, root, "cgroup-kill-test"); err != nil { t.Fatalf("KillLeafUnder: %v", err) } _ = cmd.Wait() - if err := removeActorLeaf(root, "ateomcgroup-kill-test"); err != nil { + if err := removeActorLeaf(root, "cgroup-kill-test"); err != nil { t.Errorf("removing the killed leaf: %v", err) } // Already gone: nothing to kill. - if err := KillLeafUnder(ctx, root, "ateomcgroup-kill-test"); err != nil { + if err := KillLeafUnder(ctx, root, "cgroup-kill-test"); err != nil { t.Errorf("KillLeafUnder on a removed leaf: %v", err) } } diff --git a/internal/ateomcgroup/actor_test.go b/internal/ateworker/cgroup/actor_test.go similarity index 99% rename from internal/ateomcgroup/actor_test.go rename to internal/ateworker/cgroup/actor_test.go index b23e98b055..05e5fa2dae 100644 --- a/internal/ateomcgroup/actor_test.go +++ b/internal/ateworker/cgroup/actor_test.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomcgroup +package cgroup import ( "os" diff --git a/internal/ateomcgroup/ateomcgroup.go b/internal/ateworker/cgroup/cgroup.go similarity index 96% rename from internal/ateomcgroup/ateomcgroup.go rename to internal/ateworker/cgroup/cgroup.go index a18d79be52..776856c394 100644 --- a/internal/ateomcgroup/ateomcgroup.go +++ b/internal/ateworker/cgroup/cgroup.go @@ -14,9 +14,9 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateomcgroup delegates the worker pod's cgroup to the ateom so each +// Package cgroup delegates the worker pod's cgroup to the worker so each // actor can get a leaf of its own. -package ateomcgroup +package cgroup import ( "context" @@ -31,9 +31,9 @@ import ( // Root is the worker pod's cgroup scope inside its private cgroup namespace. const Root = "/sys/fs/cgroup" -// workerLeaf holds the ateom's own processes, since a cgroup that delegates +// workerLeaf holds the worker's own processes, since a cgroup that delegates // controllers may not hold processes itself. -const workerLeaf = "ateom" +const workerLeaf = "worker" // Delegate prepares the worker pod's cgroup so the runtime can create per-actor // leaves under it with real cpu/memory/pids accounting. It reports whether it @@ -46,7 +46,7 @@ const workerLeaf = "ateom" // - The cgroup v2 "no internal processes" rule forbids a cgroup from holding // processes directly while also delegating controllers to children. The pod // scope is not the true cgroup root, so the exemption does not apply: we move -// the worker's own processes into a dedicated "ateom" leaf. +// the worker's own processes into a dedicated "worker" leaf. // - Controllers are only available to children if enabled in the scope's // cgroup.subtree_control. We enable everything the parent delegated to us. // diff --git a/internal/ateomcgroup/ateomcgroup_test.go b/internal/ateworker/cgroup/cgroup_test.go similarity index 98% rename from internal/ateomcgroup/ateomcgroup_test.go rename to internal/ateworker/cgroup/cgroup_test.go index bb0b194259..a5fbb791bc 100644 --- a/internal/ateomcgroup/ateomcgroup_test.go +++ b/internal/ateworker/cgroup/cgroup_test.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomcgroup +package cgroup import "testing" diff --git a/internal/ateomphaselog/phaselog.go b/internal/ateworker/phaselog/phaselog.go similarity index 73% rename from internal/ateomphaselog/phaselog.go rename to internal/ateworker/phaselog/phaselog.go index 58929f7675..85fe959047 100644 --- a/internal/ateomphaselog/phaselog.go +++ b/internal/ateworker/phaselog/phaselog.go @@ -12,12 +12,12 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateomphaselog renders the per-actor "timing breakdown" log record an -// ateom emits for each checkpoint and restore. Every ateom binary times its +// Package phaselog renders the per-actor "timing breakdown" log record an +// worker emits for each checkpoint and restore. Every worker binary times its // own phases, which are implementation details of that runtime; this package -// holds the shape they share so the records of all ateoms carry the same +// holds the shape they share so the records of all workers carry the same // identity, fidelity and key layout, and atelet's own record joins to any of them. -package ateomphaselog +package phaselog import ( "context" @@ -26,43 +26,43 @@ import ( "github.com/agent-substrate/substrate/internal/apierror" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "google.golang.org/grpc/status" ) // The keys the per-phase durations are logged under. They are named like // instruments but deliberately are not ones: the phases are implementation -// details of each ateom, so they stay a developer-facing log record rather -// than becoming metric API. They decompose the ateom_restore / -// ateom_checkpoint phases of atelet's ate.actor.*.duration histograms, whose +// details of each worker, so they stay a developer-facing log record rather +// than becoming metric API. They decompose the worker_restore / +// worker_checkpoint phases of atelet's ate.actor.*.duration histograms, whose // names they extend, and are joined per actor by benchmarking tooling. const ( - RestoreDurationKey = "ateom.actor.restore.duration" - CheckpointDurationKey = "ateom.actor.checkpoint.duration" + RestoreDurationKey = "ateworker.actor.restore.duration" + CheckpointDurationKey = "ateworker.actor.checkpoint.duration" ) // Phase is one timed step of a snapshot operation. A zero duration means the // phase never ran (a VOLUMES checkpoint captures no guest) and is skipped // rather than logged as instant. Name is suffixed onto the duration key; each -// ateom defines its own names, with ateattr.SnapshotPhaseTotal shared so the +// worker defines its own names, with ateattr.SnapshotPhaseTotal shared so the // two layers' records agree on the denominator. type Phase struct { Name string D time.Duration } -// FidelityLogValue maps the ateom wire enum onto the shared fidelity label +// FidelityLogValue maps the worker wire enum onto the shared fidelity label // values, the same way ateattr.SnapshotFidelityValue does for the atelet // enum. An unrecognized fidelity reports as unknown rather than stringified, // so no wire value can widen the value set readers key on. -func FidelityLogValue(fidelity ateompb.SnapshotFidelity) string { +func FidelityLogValue(fidelity ateworkerpb.SnapshotFidelity) string { switch fidelity { - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES: return ateattr.SnapshotFidelityVolumes - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS: return ateattr.SnapshotFidelityRootfs - case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: + case ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY: return ateattr.SnapshotFidelityMemory default: return ateattr.SnapshotFidelityUnknown @@ -79,7 +79,7 @@ func FidelityLogValue(fidelity ateompb.SnapshotFidelity) string { // it completed, marked with error.type (the gRPC code, context errors as // DeadlineExceeded / Canceled) so a reader can leave it out of a latency // distribution. Absence means success. -func SnapshotPhaseAttrs(a resources.ActorAttribution, fidelity ateompb.SnapshotFidelity, durationKey string, err error, phases []Phase) []slog.Attr { +func SnapshotPhaseAttrs(a resources.ActorAttribution, fidelity ateworkerpb.SnapshotFidelity, durationKey string, err error, phases []Phase) []slog.Attr { attrs := ateattr.ActorLogAttrs(a) attrs = append(attrs, slog.String(string(ateattr.SnapshotFidelityKey), FidelityLogValue(fidelity))) if err != nil { @@ -103,6 +103,6 @@ func SnapshotPhaseAttrs(a resources.ActorAttribution, fidelity ateompb.SnapshotF } // LogSnapshotPhases emits the SnapshotPhaseAttrs record under msg. -func LogSnapshotPhases(ctx context.Context, msg string, a resources.ActorAttribution, fidelity ateompb.SnapshotFidelity, durationKey string, err error, phases []Phase) { +func LogSnapshotPhases(ctx context.Context, msg string, a resources.ActorAttribution, fidelity ateworkerpb.SnapshotFidelity, durationKey string, err error, phases []Phase) { slog.LogAttrs(ctx, slog.LevelInfo, msg, SnapshotPhaseAttrs(a, fidelity, durationKey, err, phases)...) } diff --git a/internal/ateomphaselog/phaselog_test.go b/internal/ateworker/phaselog/phaselog_test.go similarity index 78% rename from internal/ateomphaselog/phaselog_test.go rename to internal/ateworker/phaselog/phaselog_test.go index 7df3d0c6ca..157fb821bd 100644 --- a/internal/ateomphaselog/phaselog_test.go +++ b/internal/ateworker/phaselog/phaselog_test.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomphaselog +package phaselog import ( "bytes" @@ -24,7 +24,7 @@ import ( "time" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" ) @@ -53,7 +53,7 @@ func renderPhaseRecord(t *testing.T, attrs []slog.Attr) map[string]any { func TestSnapshotPhaseAttrs(t *testing.T) { t.Parallel() - attrs := SnapshotPhaseAttrs(phaseLogAttribution(), ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, + attrs := SnapshotPhaseAttrs(phaseLogAttribution(), ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, CheckpointDurationKey, nil, []Phase{ {Name: "prep", D: 40 * time.Millisecond}, {Name: "pause", D: 3 * time.Millisecond}, @@ -92,11 +92,11 @@ func TestSnapshotPhaseAttrs(t *testing.T) { // Seconds, not slog.Duration's nanoseconds: the keys extend the atelet // histograms' names, and those declare unit s. for k, want := range map[string]float64{ - "ateom.actor.checkpoint.duration.prep": 0.04, - "ateom.actor.checkpoint.duration.pause": 0.003, - "ateom.actor.checkpoint.duration.snapshot": 0.85, - "ateom.actor.checkpoint.duration.teardown": 0.23, - "ateom.actor.checkpoint.duration.total": 1.25, + "ateworker.actor.checkpoint.duration.prep": 0.04, + "ateworker.actor.checkpoint.duration.pause": 0.003, + "ateworker.actor.checkpoint.duration.snapshot": 0.85, + "ateworker.actor.checkpoint.duration.teardown": 0.23, + "ateworker.actor.checkpoint.duration.total": 1.25, } { got, ok := rec[k].(float64) if !ok { @@ -106,7 +106,7 @@ func TestSnapshotPhaseAttrs(t *testing.T) { } } for _, k := range []string{ - "ateom.actor.checkpoint.duration.durable_dir", + "ateworker.actor.checkpoint.duration.durable_dir", // The phase key names the one step a datapoint timed; this record // carries them all. "ate.snapshot.phase", @@ -140,7 +140,7 @@ func TestSnapshotPhaseAttrsFailure(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() rec := renderPhaseRecord(t, SnapshotPhaseAttrs(phaseLogAttribution(), - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, CheckpointDurationKey, tt.err, + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, CheckpointDurationKey, tt.err, []Phase{ {Name: "prep", D: 40 * time.Millisecond}, {Name: "pause", D: 3 * time.Millisecond}, @@ -151,10 +151,10 @@ func TestSnapshotPhaseAttrsFailure(t *testing.T) { } // The phases that ran before the failure are still on the record; // the ones that never started are not. - if _, ok := rec["ateom.actor.checkpoint.duration.pause"]; !ok { - t.Error("missing ateom.actor.checkpoint.duration.pause") + if _, ok := rec["ateworker.actor.checkpoint.duration.pause"]; !ok { + t.Error("missing ateworker.actor.checkpoint.duration.pause") } - if v, ok := rec["ateom.actor.checkpoint.duration.snapshot"]; ok { + if v, ok := rec["ateworker.actor.checkpoint.duration.snapshot"]; ok { t.Errorf("snapshot phase present with %v, want absent", v) } }) @@ -162,19 +162,19 @@ func TestSnapshotPhaseAttrsFailure(t *testing.T) { } // TestFidelityLogValue pins the mapping onto the shared fidelity values, so the -// ateom and atelet records of one operation agree on the fidelity they carry. +// worker and atelet records of one operation agree on the fidelity they carry. func TestFidelityLogValue(t *testing.T) { t.Parallel() tests := []struct { - scope ateompb.SnapshotFidelity + scope ateworkerpb.SnapshotFidelity want string }{ - {ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateattr.SnapshotFidelityMemory}, - {ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, ateattr.SnapshotFidelityVolumes}, - {ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, ateattr.SnapshotFidelityRootfs}, - {ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, ateattr.SnapshotFidelityUnknown}, - {ateompb.SnapshotFidelity(99), ateattr.SnapshotFidelityUnknown}, + {ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY, ateattr.SnapshotFidelityMemory}, + {ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, ateattr.SnapshotFidelityVolumes}, + {ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS, ateattr.SnapshotFidelityRootfs}, + {ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, ateattr.SnapshotFidelityUnknown}, + {ateworkerpb.SnapshotFidelity(99), ateattr.SnapshotFidelityUnknown}, } for _, tt := range tests { if got := FidelityLogValue(tt.scope); got != tt.want { @@ -184,7 +184,7 @@ func TestFidelityLogValue(t *testing.T) { } // TestLogSnapshotPhases: the record goes out under the caller's message and -// at Info, through the default logger the ateoms write to. +// at Info, through the default logger the workers write to. func TestLogSnapshotPhases(t *testing.T) { var buf bytes.Buffer prev := slog.Default() @@ -192,7 +192,7 @@ func TestLogSnapshotPhases(t *testing.T) { t.Cleanup(func() { slog.SetDefault(prev) }) LogSnapshotPhases(context.Background(), "Restore timing breakdown", phaseLogAttribution(), - ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, RestoreDurationKey, nil, + ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES, RestoreDurationKey, nil, []Phase{{Name: ateattr.SnapshotPhaseTotal, D: 2 * time.Second}}) var rec map[string]any @@ -208,7 +208,7 @@ func TestLogSnapshotPhases(t *testing.T) { if rec["ate.snapshot.fidelity"] != ateattr.SnapshotFidelityVolumes { t.Errorf("ate.snapshot.fidelity = %v, want %q", rec["ate.snapshot.fidelity"], ateattr.SnapshotFidelityVolumes) } - if got := rec["ateom.actor.restore.duration.total"]; got != 2.0 { - t.Errorf("ateom.actor.restore.duration.total = %v, want 2", got) + if got := rec["ateworker.actor.restore.duration.total"]; got != 2.0 { + t.Errorf("ateworker.actor.restore.duration.total = %v, want 2", got) } } diff --git a/internal/ateominit/ateominit.go b/internal/ateworker/pid1/pid1.go similarity index 87% rename from internal/ateominit/ateominit.go rename to internal/ateworker/pid1/pid1.go index 2b4cc0a890..606ac775a2 100644 --- a/internal/ateominit/ateominit.go +++ b/internal/ateworker/pid1/pid1.go @@ -14,13 +14,13 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateominit runs a minimal init as PID 1 so ateom doesn't have to. +// Package pid1 runs a minimal init as PID 1 so worker doesn't have to. // // PID 1 inherits every orphan (e.g. the runsc sandbox once `runsc create` // exits) and must reap them with wait4(-1), which races os/exec for the exit -// status of ateom's own subprocesses. The init re-executes the binary as its +// status of worker's own subprocesses. The init re-executes the binary as its // only child, reaps everything, and exits with the child's status. -package ateominit +package pid1 import ( "fmt" @@ -30,7 +30,7 @@ import ( "golang.org/x/sys/unix" ) -// forwarded are the signals passed through to ateom. +// forwarded are the signals passed through to worker. var forwarded = []os.Signal{ unix.SIGTERM, unix.SIGINT, unix.SIGHUP, unix.SIGQUIT, unix.SIGUSR1, unix.SIGUSR2, } @@ -43,7 +43,7 @@ func RunIfPID1() { } exe, err := os.Executable() if err != nil { - fmt.Fprintf(os.Stderr, "ateominit: finding the executable: %v\n", err) + fmt.Fprintf(os.Stderr, "pid1: finding the executable: %v\n", err) os.Exit(1) } os.Exit(supervise(exe, os.Args)) @@ -54,7 +54,7 @@ func RunIfPID1() { func supervise(path string, argv []string) int { // A no-op as PID 1, but it lets tests run this as an ordinary process. if err := unix.Prctl(unix.PR_SET_CHILD_SUBREAPER, 1, 0, 0, 0); err != nil { - fmt.Fprintf(os.Stderr, "ateominit: setting child subreaper: %v\n", err) + fmt.Fprintf(os.Stderr, "pid1: setting child subreaper: %v\n", err) return 1 } @@ -68,7 +68,7 @@ func supervise(path string, argv []string) int { Files: []*os.File{os.Stdin, os.Stdout, os.Stderr}, }) if err != nil { - fmt.Fprintf(os.Stderr, "ateominit: starting %s: %v\n", path, err) + fmt.Fprintf(os.Stderr, "pid1: starting %s: %v\n", path, err) return 1 } diff --git a/internal/ateominit/ateominit_test.go b/internal/ateworker/pid1/pid1_test.go similarity index 99% rename from internal/ateominit/ateominit_test.go rename to internal/ateworker/pid1/pid1_test.go index 9cca5983f2..8721df9631 100644 --- a/internal/ateominit/ateominit_test.go +++ b/internal/ateworker/pid1/pid1_test.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateominit +package pid1 import ( "os" diff --git a/internal/ateom/register.go b/internal/ateworker/register/register.go similarity index 85% rename from internal/ateom/register.go rename to internal/ateworker/register/register.go index 4457fcba19..3eb8cd0a7f 100644 --- a/internal/ateom/register.go +++ b/internal/ateworker/register/register.go @@ -12,10 +12,10 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateom registers an ateom worker with the control plane through the +// Package register registers a worker with the control plane through the // node-local atelet, reporting its compute capacity and the sandbox runtime // its actors start under. -package ateom +package register import ( "context" @@ -32,26 +32,19 @@ import ( "github.com/agent-substrate/substrate/internal/ateletdial" "github.com/agent-substrate/substrate/internal/hardware" "github.com/agent-substrate/substrate/internal/resources" + "github.com/agent-substrate/substrate/internal/workercapacity" "k8s.io/apimachinery/pkg/api/resource" "github.com/agent-substrate/substrate/internal/proto/ateletpb" ) -// Files the atecontroller projects into the ateom container from the downward -// API, in milli-cores and bytes. -const ( - CapacityMountPath = "/run/ateom-capacity" - CPULimitFile = "cpu_milli" - MemoryLimitFile = "memory_bytes" -) - const ( reportTimeout = 10 * time.Second initialReportBackoff = 500 * time.Millisecond maxReportBackoff = 30 * time.Second ) -// FromFiles combines the downward API compute limits with the ateom's own +// FromFiles combines the downward API compute limits with the worker's own // actor limit. // // A limit that is missing or unparseable is reported as zero, which the control @@ -62,20 +55,20 @@ const ( // in-place Pod vertical scaling (IPPR); capacity is read once at startup. // NOTE: Please do not implement this yet. IPPR needs more general consideration. func FromFiles(actors int) *ateletpb.WorkerResources { - return fromDir(CapacityMountPath, actors) + return fromDir(workercapacity.MountPath, actors) } func fromDir(dir string, actors int) *ateletpb.WorkerResources { return &ateletpb.WorkerResources{ Actors: int32(actors), Resources: cpuMemory( - readLimit(filepath.Join(dir, CPULimitFile)), - readLimit(filepath.Join(dir, MemoryLimitFile)), + readLimit(filepath.Join(dir, workercapacity.CPULimitFile)), + readLimit(filepath.Join(dir, workercapacity.MemoryLimitFile)), ), } } -// defaultRuntime is the runtime actors hosted by this ateom start under: its +// defaultRuntime is the runtime actors hosted by this worker start under: its // sandbox class on this host's hardware. // // TODO: Report the pool's SandboxConfig so atelet can complete the identity @@ -119,7 +112,7 @@ func readLimit(path string) int64 { return value } -// ReportConfig configures how an ateom reaches the node-local atelet and what +// ReportConfig configures how a worker reaches the node-local atelet and what // capacity and sandbox runtime class it registers with. type ReportConfig struct { SocketPath string @@ -129,18 +122,18 @@ type ReportConfig struct { // names atelet's namespace, not this worker's, so it is configured rather // than derived from the downward API. AteletSPIFFEID string - // Actors is how many actors this ateom will host at once. + // Actors is how many actors this worker will host at once. Actors int - // SandboxClass is the class this ateom implements ("gvisor", "microvm"). + // SandboxClass is the class this worker implements ("gvisor", "microvm"). SandboxClass string } -// Report tells the node-local atelet what this ateom can supply and the +// Report tells the node-local atelet what this worker can supply and the // runtime its actors start under, retrying until it is accepted or ctx ends. // // Retrying is what makes a single report durable: atelet only accepts once the // control plane has recorded it, and the Worker record may not exist yet when -// an ateom first comes up. Nothing else reports this, so giving up would leave +// a worker first comes up. Nothing else reports this, so giving up would leave // the Worker holding no capacity and hosting nothing. func Report(ctx context.Context, cfg ReportConfig) error { if cfg.SandboxClass == "" { @@ -195,6 +188,6 @@ func reportOnce(ctx context.Context, socketPath string, tlsConfig *tls.Config, r defer conn.Close() callCtx, cancel := context.WithTimeout(ctx, reportTimeout) defer cancel() - _, err = ateletpb.NewAteomSupportClient(conn).RegisterWorker(callCtx, req) + _, err = ateletpb.NewWorkerSupportClient(conn).RegisterWorker(callCtx, req) return err } diff --git a/internal/ateom/register_test.go b/internal/ateworker/register/register_test.go similarity index 90% rename from internal/ateom/register_test.go rename to internal/ateworker/register/register_test.go index 48b32bcaa0..43cb2690f8 100644 --- a/internal/ateom/register_test.go +++ b/internal/ateworker/register/register_test.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateom +package register import ( "context" @@ -28,9 +28,10 @@ import ( "github.com/agent-substrate/substrate/internal/hardware" "github.com/agent-substrate/substrate/internal/proto/ateletpb" + "github.com/agent-substrate/substrate/internal/workercapacity" ) -// testActors stands in for the ateom's own ceiling, which is a flag in +// testActors stands in for the worker's own ceiling, which is a flag in // production rather than a constant here. const testActors = 7 @@ -49,10 +50,10 @@ func TestFromFiles(t *testing.T) { } { t.Run(tc.name, func(t *testing.T) { dir := t.TempDir() - if err := os.WriteFile(filepath.Join(dir, CPULimitFile), []byte(tc.cpu), 0o600); err != nil { + if err := os.WriteFile(filepath.Join(dir, workercapacity.CPULimitFile), []byte(tc.cpu), 0o600); err != nil { t.Fatalf("writing CPU limit: %v", err) } - if err := os.WriteFile(filepath.Join(dir, MemoryLimitFile), []byte(tc.memory), 0o600); err != nil { + if err := os.WriteFile(filepath.Join(dir, workercapacity.MemoryLimitFile), []byte(tc.memory), 0o600); err != nil { t.Fatalf("writing memory limit: %v", err) } @@ -74,7 +75,7 @@ func TestFromFilesMissing(t *testing.T) { func TestDefaultRuntime(t *testing.T) { got := defaultRuntime("gvisor") - // The class is the ateom's own; the compat identity is the host's, under + // The class is the worker's own; the compat identity is the host's, under // the probe's schema version. want := &ateletpb.SandboxRuntime{ SandboxClass: "gvisor", @@ -144,7 +145,7 @@ func TestReportFailsFastOnBadCredentials(t *testing.T) { } // A runtime without a class is refused by the control plane on every attempt, -// so an ateom that names none must exit rather than retry. +// so a worker that names none must exit rather than retry. func TestReportFailsFastWithoutSandboxClass(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() diff --git a/internal/ateomsuspend/ateomsuspend.go b/internal/ateworker/suspend/suspend.go similarity index 91% rename from internal/ateomsuspend/ateomsuspend.go rename to internal/ateworker/suspend/suspend.go index 2584096d1a..f40bfd5960 100644 --- a/internal/ateomsuspend/ateomsuspend.go +++ b/internal/ateworker/suspend/suspend.go @@ -12,14 +12,14 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateomsuspend asks the control plane, through the node-local atelet, -// to suspend an actor an ateom hosts. +// Package suspend asks the control plane, through the node-local atelet, +// to suspend an actor a worker hosts. // -// This is the upward half of a suspend: the request travels ateom -> atelet -> +// This is the upward half of a suspend: the request travels worker -> atelet -> // ateapi, and the suspend the control plane decides to run travels back down -// the ordinary path as a Checkpoint into the ateom that asked. A caller must +// the ordinary path as a Checkpoint into the worker that asked. A caller must // therefore hold no lock that Checkpoint needs while a request is in flight. -package ateomsuspend +package suspend import ( "context" @@ -66,7 +66,7 @@ func retryBackoff() wait.Backoff { } } -// Config is what an ateom needs to reach the atelet on its node. +// Config is what a worker needs to reach the atelet on its node. type Config struct { SocketPath string CredentialBundlePath string @@ -77,13 +77,13 @@ type Config struct { AteletSPIFFEID string } -// Requester asks for suspends of the actors one ateom hosts. +// Requester asks for suspends of the actors one worker hosts. type Requester struct { socketPath string tlsConfig *tls.Config } -// NewRequester loads the ateom's credentials once, at startup, so a +// NewRequester loads the worker's credentials once, at startup, so a // misconfiguration surfaces there rather than at the first request. func NewRequester(cfg Config) (*Requester, error) { tlsConfig, err := ateletdial.TLSConfig(cfg.CredentialBundlePath, cfg.TrustBundlePath, cfg.AteletSPIFFEID) @@ -94,7 +94,7 @@ func NewRequester(cfg Config) (*Requester, error) { } // Actor names the actor to suspend. The UID pins the request to the -// incarnation the ateom is hosting, so a request cannot outlive its actor and +// incarnation the worker is hosting, so a request cannot outlive its actor and // suspend a recreation that took the same name. type Actor struct { Atespace string @@ -188,7 +188,7 @@ func retryable(err error) bool { func requestOnce(ctx context.Context, conn grpc.ClientConnInterface, actor Actor) error { callCtx, cancel := context.WithTimeout(ctx, requestTimeout) defer cancel() - _, err := ateletpb.NewAteomSupportClient(conn).RequestActorSuspend(callCtx, &ateletpb.RequestActorSuspendRequest{ + _, err := ateletpb.NewWorkerSupportClient(conn).RequestActorSuspend(callCtx, &ateletpb.RequestActorSuspendRequest{ ActorAtespace: actor.Atespace, ActorName: actor.Name, ActorUid: actor.UID, diff --git a/internal/ateomsuspend/ateomsuspend_test.go b/internal/ateworker/suspend/suspend_test.go similarity index 96% rename from internal/ateomsuspend/ateomsuspend_test.go rename to internal/ateworker/suspend/suspend_test.go index 9c49b4479d..9e5b9a7fbf 100644 --- a/internal/ateomsuspend/ateomsuspend_test.go +++ b/internal/ateworker/suspend/suspend_test.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomsuspend +package suspend import ( "context" @@ -40,7 +40,7 @@ import ( // accepts. That is how a control plane that is briefly out of reach and then // comes back is spelled here. type capturingAtelet struct { - ateletpb.UnimplementedAteomSupportServer + ateletpb.UnimplementedWorkerSupportServer errs []error @@ -72,12 +72,12 @@ func testBackoff() wait.Backoff { return wait.Backoff{Steps: retrySteps, Duration: time.Millisecond} } -// dialCapturingAtelet serves an in-process AteomSupport over bufconn, which +// dialCapturingAtelet serves an in-process WorkerSupport over bufconn, which // exercises the request without a socket or certificates. func dialCapturingAtelet(t *testing.T, atelet *capturingAtelet) *grpc.ClientConn { t.Helper() srv := grpc.NewServer() - ateletpb.RegisterAteomSupportServer(srv, atelet) + ateletpb.RegisterWorkerSupportServer(srv, atelet) lis := bufconn.Listen(1 << 20) go func() { if err := srv.Serve(lis); err != nil { @@ -107,7 +107,7 @@ func TestRequestSuspendNamesTheActor(t *testing.T) { t.Fatalf("requestWithRetry() failed: %v", err) } - // The UID pins the request to the incarnation this ateom hosts, so it + // The UID pins the request to the incarnation this worker hosts, so it // cannot outlive the actor and suspend whatever took its name. want := []*ateletpb.RequestActorSuspendRequest{{ ActorAtespace: "team-a", @@ -226,7 +226,7 @@ func TestRetryable(t *testing.T) { } // A misconfiguration must surface at startup rather than at the first request: -// an ateom that cannot reach its atelet can never give a worker slot back, and +// a worker that cannot reach its atelet can never give a worker slot back, and // finding that out only when an actor goes idle hides it for as long as the // actor is busy. func TestNewRequesterFailsOnBadCredentials(t *testing.T) { diff --git a/internal/ateomtunnel/egresspolicy.go b/internal/ateworker/tunnel/egresspolicy.go similarity index 94% rename from internal/ateomtunnel/egresspolicy.go rename to internal/ateworker/tunnel/egresspolicy.go index 9f41847e9e..0e5ff7e898 100644 --- a/internal/ateomtunnel/egresspolicy.go +++ b/internal/ateworker/tunnel/egresspolicy.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomtunnel +package tunnel import ( "context" @@ -27,7 +27,7 @@ import ( "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/ateletdial" - "github.com/agent-substrate/substrate/internal/ateomnet/dns" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns" "github.com/agent-substrate/substrate/internal/nodepath" "github.com/agent-substrate/substrate/internal/proto/ateletpb" "github.com/agent-substrate/substrate/internal/resources" @@ -75,12 +75,12 @@ type actorEgressPolicy struct { func (t *Tunnel) ActorEgressPolicy(actor resources.ActorAttribution) dns.PolicySource { return &actorEgressPolicy{ fetch: func(ctx context.Context) (*ateletpb.EgressPolicy, error) { - conn, err := ateletdial.Dial(nodepath.AteomSupportSocket, t.ateletTLS) + conn, err := ateletdial.Dial(nodepath.WorkerSupportSocket, t.ateletTLS) if err != nil { return nil, err } defer conn.Close() - return getEgressPolicy(ctx, ateletpb.NewAteomSupportClient(conn), actor.Ref) + return getEgressPolicy(ctx, ateletpb.NewWorkerSupportClient(conn), actor.Ref) }, ttl: egressPolicyTTL, now: time.Now, @@ -183,7 +183,7 @@ func (p *actorEgressPolicy) record(ctx context.Context, policy *ateletpb.EgressP // getEgressPolicy asks atelet for actor's egress policy, returning nil if the // actor has none. -func getEgressPolicy(ctx context.Context, client ateletpb.AteomSupportClient, actor resources.ActorRef) (*ateletpb.EgressPolicy, error) { +func getEgressPolicy(ctx context.Context, client ateletpb.WorkerSupportClient, actor resources.ActorRef) (*ateletpb.EgressPolicy, error) { resp, err := client.GetActorEgressPolicy(ctx, &ateletpb.GetActorEgressPolicyRequest{ ActorAtespace: actor.Atespace, ActorName: actor.Name, diff --git a/internal/ateomtunnel/egresspolicy_test.go b/internal/ateworker/tunnel/egresspolicy_test.go similarity index 97% rename from internal/ateomtunnel/egresspolicy_test.go rename to internal/ateworker/tunnel/egresspolicy_test.go index 587f993d6d..33b832989f 100644 --- a/internal/ateomtunnel/egresspolicy_test.go +++ b/internal/ateworker/tunnel/egresspolicy_test.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomtunnel +package tunnel import ( "context" @@ -352,7 +352,7 @@ func TestActorEgressPolicyLogsChanges(t *testing.T) { // fakeSupportClient answers egress policy requests with resp or err. type fakeSupportClient struct { - ateletpb.AteomSupportClient + ateletpb.WorkerSupportClient resp *ateletpb.GetActorEgressPolicyResponse err error got *ateletpb.GetActorEgressPolicyRequest @@ -405,7 +405,7 @@ func TestGetEgressPolicy(t *testing.T) { // fakeAtelet serves an egress policy response as the node's atelet would. type fakeAtelet struct { - ateletpb.UnimplementedAteomSupportServer + ateletpb.UnimplementedWorkerSupportServer mu sync.Mutex resp *ateletpb.GetActorEgressPolicyResponse @@ -425,7 +425,7 @@ func (a *fakeAtelet) respond(resp *ateletpb.GetActorEgressPolicyResponse) { a.resp = resp } -// serveAtelet serves a fake atelet on the AteomSupport socket, presenting a +// serveAtelet serves a fake atelet on the WorkerSupport socket, presenting a // certificate issued by ca to the atelet identity ateletID on the test node. func serveAtelet(t *testing.T, ca testCA, ateletID string) *fakeAtelet { t.Helper() @@ -460,15 +460,15 @@ func serveAtelet(t *testing.T, ca testCA, ateletID string) *fakeAtelet { // t.TempDir() embeds the test name, which can overrun the unix socket // path limit, so the socket gets its own short directory. - socketDir, err := os.MkdirTemp("", "ateomtunnel") + socketDir, err := os.MkdirTemp("", "tunnel") if err != nil { t.Fatal(err) } t.Cleanup(func() { _ = os.RemoveAll(socketDir) }) socket := filepath.Join(socketDir, "support.sock") - old := nodepath.AteomSupportSocket - nodepath.AteomSupportSocket = socket - t.Cleanup(func() { nodepath.AteomSupportSocket = old }) + old := nodepath.WorkerSupportSocket + nodepath.WorkerSupportSocket = socket + t.Cleanup(func() { nodepath.WorkerSupportSocket = old }) lis, err := net.Listen("unix", socket) if err != nil { @@ -481,7 +481,7 @@ func serveAtelet(t *testing.T, ca testCA, ateletID string) *fakeAtelet { ClientCAs: clientCAs, }))) atelet := &fakeAtelet{} - ateletpb.RegisterAteomSupportServer(server, atelet) + ateletpb.RegisterWorkerSupportServer(server, atelet) go func() { _ = server.Serve(lis) }() t.Cleanup(server.Stop) return atelet diff --git a/internal/ateomtunnel/ateomtunnel.go b/internal/ateworker/tunnel/tunnel.go similarity index 94% rename from internal/ateomtunnel/ateomtunnel.go rename to internal/ateworker/tunnel/tunnel.go index a80671a620..ecb2ceb617 100644 --- a/internal/ateomtunnel/ateomtunnel.go +++ b/internal/ateworker/tunnel/tunnel.go @@ -14,10 +14,10 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateomtunnel wires atunnel into an ateom: its flags, listeners, DNS -// relay, and per-actor ingress and egress. Both ateoms build their tunnel here +// Package tunnel wires atunnel into a worker: its flags, listeners, DNS +// relay, and per-actor ingress and egress. Both workers build their tunnel here // so they cannot drift. -package ateomtunnel +package tunnel import ( "context" @@ -30,12 +30,12 @@ import ( "time" "github.com/agent-substrate/substrate/internal/ateletdial" - "github.com/agent-substrate/substrate/internal/ateomnet/dns" + "github.com/agent-substrate/substrate/internal/ateworker/actornet/dns" "github.com/agent-substrate/substrate/internal/atunnel" "github.com/agent-substrate/substrate/internal/egresspolicy" "github.com/agent-substrate/substrate/internal/installdefaults" "github.com/agent-substrate/substrate/internal/nodepath" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/serverboot" ) @@ -48,7 +48,7 @@ type FlagSet interface { StringVar(p *string, name, value, usage string) } -// Config is the atunnel setup an ateom takes from its flags. +// Config is the atunnel setup a worker takes from its flags. type Config struct { // ListenAddress serves actor ingress HTTPS. ListenAddress string @@ -91,7 +91,7 @@ func RegisterFlags(fs FlagSet) *Config { return c } -// Tunnel is the atunnel state one ateom shares across its actors. +// Tunnel is the atunnel state one worker shares across its actors. type Tunnel struct { // Ingress proxies router traffic to the active actors. Ingress *atunnel.Server @@ -193,7 +193,7 @@ type ActorEgress struct { // PrepareEgress mints the actor's certificate and builds its gateway client. A // nil gateway means the actor has no tunneled egress and yields a nil result. -func (t *Tunnel) PrepareEgress(ctx context.Context, actor resources.ActorAttribution, gateway *ateompb.EgressGateway) (*ActorEgress, error) { +func (t *Tunnel) PrepareEgress(ctx context.Context, actor resources.ActorAttribution, gateway *ateworkerpb.EgressGateway) (*ActorEgress, error) { if gateway == nil { return nil, nil } @@ -205,7 +205,7 @@ func (t *Tunnel) PrepareEgress(ctx context.Context, actor resources.ActorAttribu return nil, fmt.Errorf("invalid egress gateway address %q: %w", gateway.GetAddress(), err) } certificateSource, err := atunnel.NewBrokerCertificateSource(atunnel.BrokerConfig{ - SocketPath: nodepath.AteomSupportSocket, + SocketPath: nodepath.WorkerSupportSocket, CredentialBundlePath: t.cfg.CredentialBundle, TrustBundlePath: t.cfg.TrustBundle, ActorAtespace: actor.Ref.Atespace, @@ -218,7 +218,7 @@ func (t *Tunnel) PrepareEgress(ctx context.Context, actor resources.ActorAttribu } // Mint before starting the workload so configured tunneled egress fails // closed. The source retains the private key for mTLS and renewal. - expiresAt, err := certificateSource.MintAteomCertificate(ctx) + expiresAt, err := certificateSource.MintWorkerCertificate(ctx) if err != nil { return nil, fmt.Errorf("while obtaining actor certificate: %w", err) } diff --git a/internal/ateomtunnel/ateomtunnel_test.go b/internal/ateworker/tunnel/tunnel_test.go similarity index 98% rename from internal/ateomtunnel/ateomtunnel_test.go rename to internal/ateworker/tunnel/tunnel_test.go index a25221fae7..4fbbe75f65 100644 --- a/internal/ateomtunnel/ateomtunnel_test.go +++ b/internal/ateworker/tunnel/tunnel_test.go @@ -14,7 +14,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomtunnel +package tunnel import ( "context" @@ -34,7 +34,7 @@ import ( "time" "github.com/agent-substrate/substrate/internal/egresspolicy" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/internal/substratex509" "github.com/spf13/pflag" @@ -313,7 +313,7 @@ func TestPrepareEgress(t *testing.T) { "unbalanced port": "[::1", } { t.Run(name, func(t *testing.T) { - got, err := tunnel.PrepareEgress(context.Background(), actor, &ateompb.EgressGateway{Address: address}) + got, err := tunnel.PrepareEgress(context.Background(), actor, &ateworkerpb.EgressGateway{Address: address}) if err == nil || got != nil { t.Errorf("PrepareEgress(%q) = %v, %v; want an error", address, got, err) } diff --git a/internal/atunnel/client_test.go b/internal/atunnel/client_test.go index f7ffb0d4dd..c4114ea7ac 100644 --- a/internal/atunnel/client_test.go +++ b/internal/atunnel/client_test.go @@ -639,7 +639,7 @@ func newTestClient(t *testing.T, ca *testCA, opts ...ClientOption) *Client { dir := t.TempDir() trustPath := filepath.Join(dir, "trust.pem") certificate := ca.issue(t, - "spiffe://substrate-actor.local/ateom-for-actor/team/actor", + "spiffe://substrate-actor.local/worker-for-actor/team/actor", []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, ) if err := os.WriteFile(trustPath, ca.certPEM, 0o600); err != nil { diff --git a/internal/atunnel/credential.go b/internal/atunnel/credential.go index ed6007a5b3..6a2c861422 100644 --- a/internal/atunnel/credential.go +++ b/internal/atunnel/credential.go @@ -101,9 +101,9 @@ func NewBrokerCertificateSource(cfg BrokerConfig) (*BrokerCertificateSource, err }, nil } -// MintAteomCertificate requests and installs a fresh certificate for the source's existing +// MintWorkerCertificate requests and installs a fresh certificate for the source's existing // actor key. It returns the new expiry for renewal scheduling. -func (s *BrokerCertificateSource) MintAteomCertificate(ctx context.Context) (time.Time, error) { +func (s *BrokerCertificateSource) MintWorkerCertificate(ctx context.Context) (time.Time, error) { csr, err := x509.CreateCertificateRequest(rand.Reader, &x509.CertificateRequest{}, s.privateKey) if err != nil { return time.Time{}, fmt.Errorf("atunnel: create actor CSR: %w", err) @@ -119,7 +119,7 @@ func (s *BrokerCertificateSource) MintAteomCertificate(ctx context.Context) (tim return time.Time{}, err } defer conn.Close() - resp, err := ateletpb.NewAteomSupportClient(conn).MintActorCertificate(ctx, &ateletpb.MintActorCertificateRequest{ + resp, err := ateletpb.NewWorkerSupportClient(conn).MintActorCertificate(ctx, &ateletpb.MintActorCertificateRequest{ ActorAtespace: s.actorAtespace, ActorName: s.actorName, ActorUid: s.actorUID, diff --git a/internal/atunnel/credential_test.go b/internal/atunnel/credential_test.go index 0cb1991be9..381a6ee3b4 100644 --- a/internal/atunnel/credential_test.go +++ b/internal/atunnel/credential_test.go @@ -46,7 +46,7 @@ func TestBrokerCertificateSourceMintsAndReusesKey(t *testing.T) { defer cancel() for range 2 { - if _, err := source.MintAteomCertificate(ctx); err != nil { + if _, err := source.MintWorkerCertificate(ctx); err != nil { t.Fatal(err) } } @@ -59,7 +59,7 @@ func TestBrokerCertificateSourceMintsAndReusesKey(t *testing.T) { if err != nil { t.Fatal(err) } - wantURI := "spiffe://substrate-actor.local/ateom/actor/actor-atespace/actor-name" + wantURI := "spiffe://substrate-actor.local/worker/actor/actor-atespace/actor-name" if len(cert.Leaf.URIs) != 1 || cert.Leaf.URIs[0].String() != wantURI { t.Fatalf("cert.Leaf.URIs = %v, want [%s]", cert.Leaf.URIs, wantURI) } @@ -69,7 +69,7 @@ func TestBrokerCertificateSourceRejectsAteletOnDifferentNode(t *testing.T) { source, _ := newTestBrokerCertificateSource(t, testAteletIdentity("node-b"), time.Hour) ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() - if _, err := source.MintAteomCertificate(ctx); err == nil || !strings.Contains(err.Error(), "not on worker node") { + if _, err := source.MintWorkerCertificate(ctx); err == nil || !strings.Contains(err.Error(), "not on worker node") { t.Fatalf("Mint() error = %v, want node identity rejection", err) } } @@ -78,19 +78,19 @@ func TestBrokerCertificateSourceRejectsExpiredCertificate(t *testing.T) { source, _ := newTestBrokerCertificateSource(t, testAteletIdentity("node-a"), -time.Minute) ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() - if _, err := source.MintAteomCertificate(ctx); err == nil || !strings.Contains(err.Error(), "invalid actor certificate lifetime") { + if _, err := source.MintWorkerCertificate(ctx); err == nil || !strings.Contains(err.Error(), "invalid actor certificate lifetime") { t.Fatalf("Mint() error = %v, want expired certificate rejection", err) } } -type ateomSupportStub struct { - ateletpb.UnimplementedAteomSupportServer +type workerSupportStub struct { + ateletpb.UnimplementedWorkerSupportServer ca *testCA lifetime time.Duration publicKeys chan []byte } -func (s *ateomSupportStub) MintActorCertificate(_ context.Context, req *ateletpb.MintActorCertificateRequest) (*ateletpb.MintActorCertificateResponse, error) { +func (s *workerSupportStub) MintActorCertificate(_ context.Context, req *ateletpb.MintActorCertificateRequest) (*ateletpb.MintActorCertificateResponse, error) { if req.GetActorUid() != "actor-uid" { return nil, status.Error(codes.FailedPrecondition, "unexpected actor UID") } @@ -105,7 +105,7 @@ func (s *ateomSupportStub) MintActorCertificate(_ context.Context, req *ateletpb URIs: []*url.URL{{ Scheme: "spiffe", Host: "substrate-actor.local", - Path: path.Join("ateom", "actor", req.GetActorAtespace(), req.GetActorName()), + Path: path.Join("worker", "actor", req.GetActorAtespace(), req.GetActorName()), }}, NotBefore: now.Add(-time.Minute), NotAfter: now.Add(s.lifetime), @@ -120,18 +120,18 @@ func (s *ateomSupportStub) MintActorCertificate(_ context.Context, req *ateletpb return &ateletpb.MintActorCertificateResponse{ActorCertificates: [][]byte{der}}, nil } -func newTestBrokerCertificateSource(t *testing.T, ateletIdentity *substratex509.PodIdentity, lifetime time.Duration) (*BrokerCertificateSource, *ateomSupportStub) { +func newTestBrokerCertificateSource(t *testing.T, ateletIdentity *substratex509.PodIdentity, lifetime time.Duration) (*BrokerCertificateSource, *workerSupportStub) { t.Helper() ca := newTestCA(t) workerCert := issueTestPodCertificate(t, ca, &substratex509.PodIdentity{ Namespace: "ate-demo", - ServiceAccountName: "ateom", - ServiceAccountUID: "ateom-sa-uid", + ServiceAccountName: "worker", + ServiceAccountUID: "worker-sa-uid", PodName: "worker", PodUID: "worker-uid", NodeName: "node-a", NodeUID: "node-uid", - }, "spiffe://cluster.local/ns/ate-demo/sa/ateom", []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}) + }, "spiffe://cluster.local/ns/ate-demo/sa/worker", []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}) ateletCert := issueTestPodCertificate(t, ca, ateletIdentity, "spiffe://cluster.local/ns/ate-system/sa/atelet", []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}) @@ -163,8 +163,8 @@ func newTestBrokerCertificateSource(t *testing.T, ateletIdentity *substratex509. ClientAuth: tls.RequireAndVerifyClientCert, ClientCAs: clientCAs, }))) - broker := &ateomSupportStub{ca: ca, lifetime: lifetime, publicKeys: make(chan []byte, 2)} - ateletpb.RegisterAteomSupportServer(server, broker) + broker := &workerSupportStub{ca: ca, lifetime: lifetime, publicKeys: make(chan []byte, 2)} + ateletpb.RegisterWorkerSupportServer(server, broker) go func() { _ = server.Serve(listener) }() t.Cleanup(func() { server.Stop() diff --git a/internal/atunnel/egress.go b/internal/atunnel/egress.go index c63678ad56..422ce1d928 100644 --- a/internal/atunnel/egress.go +++ b/internal/atunnel/egress.go @@ -36,7 +36,7 @@ type egressDialer interface { } type actorCertificateSource interface { - MintAteomCertificate(context.Context) (time.Time, error) + MintWorkerCertificate(context.Context) (time.Time, error) } // OriginalDestination returns the address that a transparently intercepted @@ -180,11 +180,11 @@ func (e *Egress) renew(active *egressActivation, expiresAt time.Time) { slog.Time("expiredAt", expiresAt)) expired = true } - nextExpiry, err := active.certificateSource.MintAteomCertificate(active.ctx) + nextExpiry, err := active.certificateSource.MintWorkerCertificate(active.ctx) if err != nil { code := status.Code(err) - // Don't retry on these codes. (No retries in ateom will fix - // Aborted, it indicates that ateom is running an out-of-date + // Don't retry on these codes. (No retries in worker will fix + // Aborted, it indicates that worker is running an out-of-date // actor.) if code == codes.Aborted || code == codes.FailedPrecondition || code == codes.PermissionDenied { e.mu.Lock() diff --git a/internal/atunnel/egress_test.go b/internal/atunnel/egress_test.go index ebc724d66f..75de28f6d0 100644 --- a/internal/atunnel/egress_test.go +++ b/internal/atunnel/egress_test.go @@ -388,7 +388,7 @@ func TestEgressEndToEnd(t *testing.T) { t.Errorf("gateway connection has type %T, want *tls.Conn", conn) } else { peer := tlsConn.ConnectionState().PeerCertificates[0] - if len(peer.URIs) != 1 || peer.URIs[0].String() != "spiffe://substrate-actor.local/ateom-for-actor/team/actor" { + if len(peer.URIs) != 1 || peer.URIs[0].String() != "spiffe://substrate-actor.local/worker-for-actor/team/actor" { t.Errorf("client identity = %v, want actor SPIFFE ID", peer.URIs) } } @@ -541,7 +541,7 @@ type fakeActorCertificateSource struct { release <-chan struct{} } -func (s fakeActorCertificateSource) MintAteomCertificate(context.Context) (time.Time, error) { +func (s fakeActorCertificateSource) MintWorkerCertificate(context.Context) (time.Time, error) { if s.calls != nil { s.calls.Add(1) } diff --git a/internal/atunnel/ingress.go b/internal/atunnel/ingress.go index 358a348a76..e921725438 100644 --- a/internal/atunnel/ingress.go +++ b/internal/atunnel/ingress.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package atunnel carries actor ingress and egress through an ateom worker pod. +// Package atunnel carries actor ingress and egress through a worker pod. package atunnel import ( diff --git a/internal/atunnel/original_dst_linux_test.go b/internal/atunnel/original_dst_linux_test.go index 7e6c24eac8..1e71bc95f4 100644 --- a/internal/atunnel/original_dst_linux_test.go +++ b/internal/atunnel/original_dst_linux_test.go @@ -33,7 +33,7 @@ import ( "github.com/vishvananda/netlink" "golang.org/x/sys/unix" - "github.com/agent-substrate/substrate/internal/ateomnet/netns" + "github.com/agent-substrate/substrate/internal/netns" "github.com/agent-substrate/substrate/internal/roottest" ) diff --git a/internal/benchmarking/glutton/glutton.go b/internal/benchmarking/glutton/glutton.go index 17e438bfbd..1f978e26f8 100644 --- a/internal/benchmarking/glutton/glutton.go +++ b/internal/benchmarking/glutton/glutton.go @@ -30,7 +30,7 @@ const ( ModeHTTP = "http" ) -// Routes the HTTP-mode mux serves. ReadyzRoute is served in both modes: ateom +// Routes the HTTP-mode mux serves. ReadyzRoute is served in both modes: worker // blocks RestoreWorkload until it answers 200, so ResumeActor cannot report // success before the listener is reachable. The fake in ./fake aliases these, // which is what keeps the stand-in and the real mux from drifting apart. diff --git a/internal/cdi/cdi.go b/internal/cdi/cdi.go index 973f2827dd..62e00404c7 100644 --- a/internal/cdi/cdi.go +++ b/internal/cdi/cdi.go @@ -15,15 +15,15 @@ // Package cdi reads a Container Device Interface spec and resolves what a named // set of devices asks a container runtime to do. // -// An ateom cannot leave this to the container runtime. containerd applies CDI to -// the pod's containers, and an actor's containers are created by the ateom +// A worker cannot leave this to the container runtime. containerd applies CDI to +// the pod's containers, and an actor's containers are created by the worker // underneath one of them, so whatever hands devices to actors reads the spec // itself. That holds for any sandbox, and under DRA too, which names allocated // devices as CDI device IDs. // // Applying the edits is a separate step and is not generic: it depends on how the // sandbox gets an OCI spec and whether it shares the host's device nodes. See -// cmd/ateom-gvisor/internal/cdiinject for the gVisor one. +// cmd/ateworker-gvisor/internal/cdiinject for the gVisor one. package cdi import ( diff --git a/internal/e2e/README.md b/internal/e2e/README.md index 0045c221a5..7b72641665 100644 --- a/internal/e2e/README.md +++ b/internal/e2e/README.md @@ -70,11 +70,25 @@ it to `1` to measure single-CPU Envoy performance. |------------------------|-----------------------------|---------------------------|---------|------------| | `e2e.envoyConcurrency` | `E2E_ATE_ENVOY_CONCURRENCY` | `--e2e-envoy-concurrency` | `0` | (global) | +## Preview features + +Tests of a preview feature start with `e2e.RequirePreview(t, preview.Gate...)`, +which skips them unless the `--preview` flag names that gate. It takes the same +syntax as `ATE_PREVIEW` and should match what the cluster was installed with: + +```shell +$ hack/run-e2e-kind.sh -v -args --no-color --preview=ALL +``` + +Unset or empty means no gates. CI must pass it, even if only as empty, so a +lane that forgets cannot silently skip every preview test; see +[preview.go](preview.go). + ## After a failure A suite deletes the namespaces it created only when it passed. A failed run keeps them, because the failure is usually explained inside a worker pod (the -ateom logs, and for a micro-VM worker the guest's console tail), and deleting +worker logs, and for a micro-VM worker the guest's console tail), and deleting the namespace takes those pods with it: ```shell diff --git a/internal/e2e/atenet_dataplane.go b/internal/e2e/atenet_dataplane.go index 63438a0d29..38235eb892 100644 --- a/internal/e2e/atenet_dataplane.go +++ b/internal/e2e/atenet_dataplane.go @@ -141,10 +141,10 @@ func (agentGatewayAtenetDataplane) NewParkingObserver(context.Context) (ParkingO } func (agentGatewayAtenetDataplane) IsRetryableParkingBudgetExhaustion(status int, body string) bool { - return status == http.StatusGatewayTimeout && strings.Contains(body, "request timed out") + return status == http.StatusServiceUnavailable && strings.Contains(body, "no worker has room for the actor") } -func (agentGatewayAtenetDataplane) ParkingBudgetStatus() int { return http.StatusGatewayTimeout } +func (agentGatewayAtenetDataplane) ParkingBudgetStatus() int { return http.StatusServiceUnavailable } func (agentGatewayAtenetDataplane) EgressUpgradeDeniedBody() string { return "HTTP upgrades, including CONNECT, are denied for actor egress" diff --git a/internal/e2e/atenet_dataplane_test.go b/internal/e2e/atenet_dataplane_test.go index 9d3b7c6d45..a38f5965a1 100644 --- a/internal/e2e/atenet_dataplane_test.go +++ b/internal/e2e/atenet_dataplane_test.go @@ -19,6 +19,30 @@ import ( "testing" ) +func TestAgentGatewayParkingCapacityResponse(t *testing.T) { + t.Setenv(AtenetDataplaneEnv, "agentgateway") + dataplane := CurrentAtenetDataplane() + if got := dataplane.ParkingBudgetStatus(); got != http.StatusServiceUnavailable { + t.Errorf("ParkingBudgetStatus() = %d, want 503", got) + } + for _, tt := range []struct { + name string + status int + body string + want bool + }{ + {"capacity exhausted", http.StatusServiceUnavailable, `actor "parked-a" unavailable: no worker has room for the actor`, true}, + {"API unavailable", http.StatusServiceUnavailable, `actor "parked-a" unavailable`, false}, + {"request timeout", http.StatusGatewayTimeout, `actor "parked-a" request timed out`, false}, + } { + t.Run(tt.name, func(t *testing.T) { + if got := dataplane.IsRetryableParkingBudgetExhaustion(tt.status, tt.body); got != tt.want { + t.Errorf("IsRetryableParkingBudgetExhaustion(%d, %q) = %v, want %v", tt.status, tt.body, got, tt.want) + } + }) + } +} + func TestAtenetDataplaneEgressPolicyDenial(t *testing.T) { t.Run("envoy", func(t *testing.T) { t.Setenv(AtenetDataplaneEnv, "") diff --git a/internal/e2e/clients.go b/internal/e2e/clients.go index c007f3f60a..4495812700 100644 --- a/internal/e2e/clients.go +++ b/internal/e2e/clients.go @@ -69,7 +69,7 @@ func NewClients(ctx context.Context) (*Clients, error) { connectCtx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() - apiClient, err := ateclient.NewClient(connectCtx, KubeConfig, KubeContext, "", "", false) + apiClient, err := ateclient.NewClient(connectCtx, KubeConfig, KubeContext, "", "", false, previewDialOptions()...) if err != nil { return nil, fmt.Errorf("NewClient: %w", err) } diff --git a/internal/e2e/collector_metrics_test.go b/internal/e2e/collector_metrics_test.go index b0efb72843..2cdb46e6f9 100644 --- a/internal/e2e/collector_metrics_test.go +++ b/internal/e2e/collector_metrics_test.go @@ -124,7 +124,7 @@ const sampleNode = "kind-worker2" const sampleTargetInfo = `# HELP target_info Target metadata # TYPE target_info gauge target_info{job="atelet",instance="pod-uid-1",k8s_namespace_name="ate-system",k8s_pod_name="atelet-abc",k8s_node_name="` + sampleNode + `"} 1 -target_info{job="ateom-gvisor",instance="pod-uid-2",k8s_namespace_name="ate-system",k8s_pod_name="wp-xyz",k8s_node_name="` + sampleNode + `"} 1 +target_info{job="ateworker-gvisor",instance="pod-uid-2",k8s_namespace_name="ate-system",k8s_pod_name="wp-xyz",k8s_node_name="` + sampleNode + `"} 1 target_info{job="atecontroller",instance="pod-uid-3",k8s_namespace_name="ate-system",k8s_pod_name="ctrl-def"} 1 ate_workerpool_workers{job="atelet",ate_workerpool_name="pool-a"} 3 ` @@ -138,7 +138,7 @@ func TestTargetInfoLabel(t *testing.T) { want string }{ {"node on the first resource", sampleTargetInfo, "atelet", "k8s_node_name", sampleNode}, - {"node on a later resource", sampleTargetInfo, "ateom-gvisor", "k8s_node_name", sampleNode}, + {"node on a later resource", sampleTargetInfo, "ateworker-gvisor", "k8s_node_name", sampleNode}, {"label absent for that service", sampleTargetInfo, "atecontroller", "k8s_node_name", ""}, {"service absent", sampleTargetInfo, "atenet", "k8s_node_name", ""}, {"other resource attributes still readable", sampleTargetInfo, "atelet", "k8s_pod_name", "atelet-abc"}, diff --git a/internal/e2e/fixture.go b/internal/e2e/fixture.go index e3cf5167d4..b2767cbc67 100644 --- a/internal/e2e/fixture.go +++ b/internal/e2e/fixture.go @@ -43,7 +43,6 @@ func substrateTemplateSubstitutions(bucket, name string, trustBundles []string) // manifests/ate-install ships; config_name is required, so the // fixtures name it explicitly. "${TEMPLATE_SANDBOX_CONFIG}": "sandboxConfig:\n sandboxClass: gvisor\n configName: gvisor-default", - "${TEMPLATE_RESOURCES}": "", // Off unless the caller opts in; see WithTrustBundle. "${TEMPLATE_TRUST_BUNDLE}": "", } @@ -64,11 +63,6 @@ func substrateTemplateSubstitutions(bucket, name string, trustBundles []string) // The cluster-wide SandboxConfig hack/install-microvm-deps.sh installs; // a missing or stale one fails loudly at template creation. blocks["${TEMPLATE_SANDBOX_CONFIG}"] = "sandboxConfig:\n sandboxClass: microvm\n configName: microvm" - // Only for fixtures that declare no limits of their own. Without them the - // guest boots at ateom's default size (2GiB), and several of those - // do not fit beside the demo pools on CI's single kind node. These size - // the VM itself — see internal/sizing. Quantities are strings. - blocks["${TEMPLATE_RESOURCES}"] = "resources:\n limits:\n - name: cpu\n quantity: \"1\"\n - name: memory\n quantity: 512Mi" return inline, blocks } diff --git a/internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl b/internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl index a762f0e2e7..e783dce184 100644 --- a/internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl +++ b/internal/e2e/fixtures/capabilities/capabilities-templates.yaml.tmpl @@ -36,7 +36,12 @@ containers: path: /healthz port: 80 timeoutSeconds: 60 -${TEMPLATE_RESOURCES} +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi ${TEMPLATE_SANDBOX_CONFIG} snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-caps-default${FIXTURE_SUFFIX}/ @@ -68,7 +73,12 @@ containers: path: /healthz port: 80 timeoutSeconds: 60 -${TEMPLATE_RESOURCES} +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi ${TEMPLATE_SANDBOX_CONFIG} snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-caps-exact${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/capabilities/capabilities.yaml.tmpl b/internal/e2e/fixtures/capabilities/capabilities.yaml.tmpl index 5ead73d0cc..a983c78490 100644 --- a/internal/e2e/fixtures/capabilities/capabilities.yaml.tmpl +++ b/internal/e2e/fixtures/capabilities/capabilities.yaml.tmpl @@ -38,5 +38,5 @@ spec: # One worker per template for its golden snapshot, plus headroom for the # actors the suite resumes. replicas: 4 - workerImage: ${ATEOM_IMAGE} + workerImage: ${WORKER_IMAGE} ${WORKERPOOL_RUNTIME} diff --git a/internal/e2e/fixtures/probe/probe-multiactor.yaml.tmpl b/internal/e2e/fixtures/probe/probe-multiactor.yaml.tmpl index 9ed0b9b412..2d07d62b05 100644 --- a/internal/e2e/fixtures/probe/probe-multiactor.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-multiactor.yaml.tmpl @@ -30,7 +30,7 @@ metadata: workload: probe-multiactor spec: replicas: 1 - workerImage: ${ATEOM_IMAGE} + workerImage: ${WORKER_IMAGE} template: # A worker's limits are its capacity, which placement weighs along with # the actor count. Two actors' worth of CPU: the pair fits, a third does not. diff --git a/internal/e2e/fixtures/probe/probe-parking.yaml.tmpl b/internal/e2e/fixtures/probe/probe-parking.yaml.tmpl index e006625536..4dcfc54cce 100644 --- a/internal/e2e/fixtures/probe/probe-parking.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-parking.yaml.tmpl @@ -34,7 +34,7 @@ metadata: workload: probe-parking spec: replicas: 1 - workerImage: ${ATEOM_IMAGE} + workerImage: ${WORKER_IMAGE} template: # One worker holding one actor: the limits are what it advertises as # capacity, and they are one actor's worth of CPU, so the suite's second diff --git a/internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl b/internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl index 06f9261ebb..1638fc4c9a 100644 --- a/internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-sized-template.yaml.tmpl @@ -15,9 +15,7 @@ # Sized variant of the probe template: a protojson-shaped # ateapipb.ActorTemplate declaring its own resources.limits, so the sizing e2e # suite can assert the actor's sandbox is shaped to those limits. Reuses the -# probe image (it serves the /resources endpoint). Unlike -# probe-template.yaml.tmpl this carries no ${TEMPLATE_RESOURCES} placeholder — -# the limits are the thing under test. +# probe image (it serves the /resources endpoint). metadata: atespace: ate-e2e${FIXTURE_SUFFIX} diff --git a/internal/e2e/fixtures/probe/probe-sized.yaml.tmpl b/internal/e2e/fixtures/probe/probe-sized.yaml.tmpl index 99bbc9899a..1528d71383 100644 --- a/internal/e2e/fixtures/probe/probe-sized.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-sized.yaml.tmpl @@ -35,5 +35,5 @@ metadata: workload: probe-sized spec: replicas: 3 - workerImage: ${ATEOM_IMAGE} + workerImage: ${WORKER_IMAGE} ${WORKERPOOL_RUNTIME} diff --git a/internal/e2e/fixtures/probe/probe-template.yaml.tmpl b/internal/e2e/fixtures/probe/probe-template.yaml.tmpl index 74194b836a..da362a14cb 100644 --- a/internal/e2e/fixtures/probe/probe-template.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe-template.yaml.tmpl @@ -34,8 +34,8 @@ containers: mountPath: /run/ate # The probe binary binds :80 immediately, so this gates actor start on a # readiness signal rather than a guess, and carries a non-default - # timeoutSeconds so e2e covers the value crossing ateapi -> atelet -> ateom - # instead of only the ateom's built-in default. + # timeoutSeconds so e2e covers the value crossing ateapi -> atelet -> ateworker + # instead of only the ateworker's built-in default. wakeupProbe: httpGet: path: /healthz @@ -56,7 +56,12 @@ volumes: # Only for suites that pass WithTrustBundle: the bundle is a cluster # singleton, so a suite that merely needs a probe must not depend on it. ${TEMPLATE_TRUST_BUNDLE} -${TEMPLATE_RESOURCES} +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi ${TEMPLATE_SANDBOX_CONFIG} snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-probe${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/probe/probe.yaml.tmpl b/internal/e2e/fixtures/probe/probe.yaml.tmpl index 3c08324a18..984925b566 100644 --- a/internal/e2e/fixtures/probe/probe.yaml.tmpl +++ b/internal/e2e/fixtures/probe/probe.yaml.tmpl @@ -41,5 +41,5 @@ metadata: workload: probe${FIXTURE_SUFFIX} spec: replicas: 3 - workerImage: ${ATEOM_IMAGE} + workerImage: ${WORKER_IMAGE} ${WORKERPOOL_RUNTIME} diff --git a/internal/e2e/fixtures/security/atespace-a-pool.yaml.tmpl b/internal/e2e/fixtures/security/atespace-a-pool.yaml.tmpl index 32fd9bc1b1..8b38267899 100644 --- a/internal/e2e/fixtures/security/atespace-a-pool.yaml.tmpl +++ b/internal/e2e/fixtures/security/atespace-a-pool.yaml.tmpl @@ -46,5 +46,5 @@ metadata: workload: ate-e2e-security spec: replicas: 1 - workerImage: ${ATEOM_IMAGE} + workerImage: ${WORKER_IMAGE} ${WORKERPOOL_RUNTIME} diff --git a/internal/e2e/fixtures/security/atespace-template.yaml.tmpl b/internal/e2e/fixtures/security/atespace-template.yaml.tmpl index a54d995349..f635ddcc5f 100644 --- a/internal/e2e/fixtures/security/atespace-template.yaml.tmpl +++ b/internal/e2e/fixtures/security/atespace-template.yaml.tmpl @@ -70,7 +70,12 @@ volumes: path: actor-uid - name: data durableDir: {} -${TEMPLATE_RESOURCES} +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi ${TEMPLATE_SANDBOX_CONFIG} snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/testserver/grpcecho-template.yaml.tmpl b/internal/e2e/fixtures/testserver/grpcecho-template.yaml.tmpl index cbc3368909..8f81a7081d 100644 --- a/internal/e2e/fixtures/testserver/grpcecho-template.yaml.tmpl +++ b/internal/e2e/fixtures/testserver/grpcecho-template.yaml.tmpl @@ -45,7 +45,12 @@ containers: path: /readyz port: 8080 timeoutSeconds: 60 -${TEMPLATE_RESOURCES} +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi ${TEMPLATE_SANDBOX_CONFIG} snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-grpcecho${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/testserver/grpcecho.yaml.tmpl b/internal/e2e/fixtures/testserver/grpcecho.yaml.tmpl index 9b22779fb7..18b5de884e 100644 --- a/internal/e2e/fixtures/testserver/grpcecho.yaml.tmpl +++ b/internal/e2e/fixtures/testserver/grpcecho.yaml.tmpl @@ -39,5 +39,5 @@ spec: # One worker for the template's golden snapshot, plus headroom for the actors # the suite resumes. replicas: 2 - workerImage: ${ATEOM_IMAGE} + workerImage: ${WORKER_IMAGE} ${WORKERPOOL_RUNTIME} diff --git a/internal/e2e/fixtures/testserver/http.go b/internal/e2e/fixtures/testserver/http.go index 32e57ce5b7..6da26f7ad0 100644 --- a/internal/e2e/fixtures/testserver/http.go +++ b/internal/e2e/fixtures/testserver/http.go @@ -19,6 +19,7 @@ import ( "fmt" "io" "log" + "net" "net/http" "os" "strings" @@ -34,6 +35,17 @@ type echoedHeaders struct { Headers map[string]string `json:"headers"` } +// dialResult is the /dial response body. +type dialResult struct { + Address string `json:"address"` + // OK reports whether the TCP connection was established. + OK bool `json:"ok"` + Error string `json:"error,omitempty"` +} + +// defaultDialTimeout is the /dial timeout when the request sets none. +const defaultDialTimeout = 3 * time.Second + // newHTTPHandler is the HTTP/1.1 origin an Actor's egress lands on. It exists so // a test can assert the destination port is recovered from SO_ORIGINAL_DST // rather than defaulted from the URL scheme: the actor fetches its /healthz on a @@ -42,6 +54,8 @@ type echoedHeaders struct { // response inside the egress tunnel. /echoheaders returns the request headers as // they arrived, which is how a suite sees what the gateway put on the wire. // The /fetch handler for e2e tests that need test requests with bodies. +// /dial reports whether this pod can open a TCP connection to an address; see +// dial. func newHTTPHandler(body, authorizationFile string) http.Handler { mux := http.NewServeMux() mux.HandleFunc("/healthz", func(w http.ResponseWriter, _ *http.Request) { @@ -70,9 +84,45 @@ func newHTTPHandler(body, authorizationFile string) http.Handler { } w.WriteHeader(http.StatusOK) }) + mux.HandleFunc("/dial", dial) return mux } +// dial opens a TCP connection to ?address= within ?timeout= +// (default 3s) and closes it again. A suite uses it to check what this pod +// can reach, such as whether a NetworkPolicy lets the pod's traffic through. +// A malformed request gets a 400 rather than ok=false, so a mistake in the +// caller is not mistaken for an unreachable address. +func dial(w http.ResponseWriter, r *http.Request) { + address := r.URL.Query().Get("address") + if _, port, err := net.SplitHostPort(address); err != nil || port == "" { + http.Error(w, fmt.Sprintf("address parameter %q is not :", address), http.StatusBadRequest) + return + } + timeout := defaultDialTimeout + if raw := r.URL.Query().Get("timeout"); raw != "" { + parsed, err := time.ParseDuration(raw) + if err != nil || parsed <= 0 { + http.Error(w, fmt.Sprintf("timeout parameter %q is not a positive duration", raw), http.StatusBadRequest) + return + } + timeout = parsed + } + + result := dialResult{Address: address} + dialer := net.Dialer{Timeout: timeout} + if conn, err := dialer.DialContext(r.Context(), "tcp", address); err != nil { + result.Error = err.Error() + } else { + result.OK = true + _ = conn.Close() + } + w.Header().Set("Content-Type", "application/json") + if err := json.NewEncoder(w).Encode(result); err != nil { + log.Printf("testserver http: encoding /dial response: %v", err) + } +} + func newHTTPCmd() *cobra.Command { var ( authorizationFile string @@ -83,7 +133,7 @@ func newHTTPCmd() *cobra.Command { ) cmd := &cobra.Command{ Use: "http", - Short: "Serve an HTTP/1.1 origin answering /healthz, /echoheaders and POST /fetch.", + Short: "Serve an HTTP/1.1 origin answering /healthz, /echoheaders, POST /fetch and /dial.", Args: cobra.NoArgs, RunE: func(_ *cobra.Command, _ []string) error { server := &http.Server{ diff --git a/internal/e2e/fixtures/testserver/http_test.go b/internal/e2e/fixtures/testserver/http_test.go index 88875a4327..27569aa90c 100644 --- a/internal/e2e/fixtures/testserver/http_test.go +++ b/internal/e2e/fixtures/testserver/http_test.go @@ -21,6 +21,7 @@ import ( "io" "net/http" "net/http/httptest" + "net/url" "os" "path/filepath" "strings" @@ -232,6 +233,78 @@ func TestHTTPWithoutCertificateServesOnlyCleartext(t *testing.T) { } } +// getDial calls the origin's /dial with query and returns the status and, for a +// 200, the decoded result. +func getDial(t *testing.T, originURL, query string) (int, dialResult) { + t.Helper() + // Its own transport with keep-alives off, so each call opens one connection + // and closes it after the response. + client := &http.Client{Timeout: 5 * time.Second, Transport: &http.Transport{DisableKeepAlives: true}} + response, err := client.Get(originURL + "/dial?" + query) + if err != nil { + t.Fatalf("GET /dial?%s: %v", query, err) + } + defer response.Body.Close() + var result dialResult + if response.StatusCode == http.StatusOK { + if err := json.NewDecoder(response.Body).Decode(&result); err != nil { + t.Fatalf("decoding /dial?%s: %v", query, err) + } + } + return response.StatusCode, result +} + +func TestHTTPDialReachesListener(t *testing.T) { + server := startOriginServer(t, newHTTPHandler("", ""), "", "") + target := listenLoopback(t) + t.Cleanup(func() { target.Close() }) + + address := target.Addr().String() + status, result := getDial(t, server.httpURL, "address="+url.QueryEscape(address)) + if status != http.StatusOK { + t.Fatalf("GET /dial status = %d, want %d", status, http.StatusOK) + } + if !result.OK || result.Error != "" { + t.Errorf("dial %s = ok %t, error %q; want ok with no error", address, result.OK, result.Error) + } + if result.Address != address { + t.Errorf("dial result address = %q, want %q", result.Address, address) + } +} + +func TestHTTPDialReportsUnreachableAddress(t *testing.T) { + server := startOriginServer(t, newHTTPHandler("", ""), "", "") + // A port that was just listening and is now closed refuses the connection. + closed := listenLoopback(t) + address := closed.Addr().String() + closed.Close() + + status, result := getDial(t, server.httpURL, "address="+url.QueryEscape(address)+"&timeout=1s") + if status != http.StatusOK { + t.Fatalf("GET /dial status = %d, want %d: an unreachable address is a result, not a request error", status, http.StatusOK) + } + if result.OK || result.Error == "" { + t.Errorf("dial %s = ok %t, error %q; want not ok with the reason", address, result.OK, result.Error) + } +} + +func TestHTTPDialRejectsMalformedRequest(t *testing.T) { + server := startOriginServer(t, newHTTPHandler("", ""), "", "") + for name, query := range map[string]string{ + "missing address": "", + "address w/o port": "address=127.0.0.1", + "empty port": "address=127.0.0.1:", + "bad timeout": "address=127.0.0.1:80&timeout=soon", + "zero timeout": "address=127.0.0.1:80&timeout=0s", + } { + t.Run(name, func(t *testing.T) { + if status, _ := getDial(t, server.httpURL, query); status != http.StatusBadRequest { + t.Errorf("GET /dial?%s status = %d, want %d", query, status, http.StatusBadRequest) + } + }) + } +} + func TestCredentialHandler(t *testing.T) { path := filepath.Join(t.TempDir(), "token") if err := os.WriteFile(path, []byte("expected-token"), 0600); err != nil { diff --git a/internal/e2e/fixtures/testserver/main.go b/internal/e2e/fixtures/testserver/main.go index 177c89e0c1..c17ae28af9 100644 --- a/internal/e2e/fixtures/testserver/main.go +++ b/internal/e2e/fixtures/testserver/main.go @@ -18,7 +18,7 @@ // than a new image to build and push: // // testserver grpc --listen=:50051 a cleartext HTTP/2 gRPC echo origin -// testserver http --listen=:8080 an HTTP(S) origin serving /healthz and /echoheaders +// testserver http --listen=:8080 an HTTP(S) origin serving /healthz, /echoheaders, POST /fetch and /dial // testserver egressprobe --listen=:8080 a client that drives the egress gateway // testserver websocket --listen=:8080 a websocket server that responds to PINGs // diff --git a/internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl b/internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl index 590e63b297..18617008f3 100644 --- a/internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl +++ b/internal/e2e/fixtures/testserver/websocket-template.yaml.tmpl @@ -31,7 +31,12 @@ containers: path: /readyz port: 80 timeoutSeconds: 30 -${TEMPLATE_RESOURCES} +resources: + limits: + - name: cpu + quantity: "1" + - name: memory + quantity: 512Mi ${TEMPLATE_SANDBOX_CONFIG} snapshotConfig: storageLocation: gs://${BUCKET_NAME}/ate-e2e-websocket${FIXTURE_SUFFIX}/ diff --git a/internal/e2e/fixtures/testserver/websocket.yaml.tmpl b/internal/e2e/fixtures/testserver/websocket.yaml.tmpl index 4c70c40df8..b5a716b983 100644 --- a/internal/e2e/fixtures/testserver/websocket.yaml.tmpl +++ b/internal/e2e/fixtures/testserver/websocket.yaml.tmpl @@ -32,5 +32,5 @@ metadata: workload: websocket${FIXTURE_SUFFIX} spec: replicas: 1 - workerImage: ${ATEOM_IMAGE} + workerImage: ${WORKER_IMAGE} ${WORKERPOOL_RUNTIME} diff --git a/internal/e2e/namespace.go b/internal/e2e/namespace.go index 15dda05b89..72bbcd9f29 100644 --- a/internal/e2e/namespace.go +++ b/internal/e2e/namespace.go @@ -109,7 +109,7 @@ func CreateNamespace(t *testing.T) *Namespace { // single-node kind cluster shows up as worker pods that never get scheduled. // // A failed test keeps its namespace, for the same reason RetainNamespaces does: - // the worker pods are where the ateom (and micro-VM guest console) logs live, + // the worker pods are where the worker (and micro-VM guest console) logs live, // and they are gone the moment the namespace is. t.Cleanup(func() { if t.Failed() { @@ -165,7 +165,7 @@ func deleteNamespace(name string) { // RetainNamespaces leaves the registered namespaces in the cluster and reports // them, instead of deleting them. Used when the suite failed: the namespaces' -// worker pods hold the ateom (and, for micro-VM workers, guest) logs a +// worker pods hold the worker (and, for micro-VM workers, guest) logs a // post-mortem needs, and they are deleted long before anyone can read them. // // Passing tests have already released theirs, so what is left is the failures' diff --git a/internal/e2e/preview.go b/internal/e2e/preview.go new file mode 100644 index 0000000000..de8068af1a --- /dev/null +++ b/internal/e2e/preview.go @@ -0,0 +1,83 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "fmt" + "os" + "strings" + "testing" + + "google.golang.org/grpc" + + "github.com/agent-substrate/substrate/internal/preview" + + "github.com/spf13/pflag" +) + +// previewFlag lists the preview gates the install under test enabled, in the +// same syntax as ate-setup's ATE_PREVIEW: comma- or space-separated gate +// names, or "ALL" for all of them. Empty means none. +var previewFlag string + +// previewRequired reports whether --preview must be passed. A lane that +// forgets it would otherwise skip every preview test and still pass, so CI +// must declare its preview mode, even if only as empty. +func previewRequired() bool { + return os.Getenv("CI") == "true" +} + +// previewGates returns the gates in value. set reports whether --preview was +// passed at all. +func previewGates(value string, set bool) ([]string, error) { + if !set && previewRequired() { + return nil, fmt.Errorf("--preview must be passed in CI (empty for no preview gates)") + } + return strings.FieldsFunc(value, func(r rune) bool { + return r == ',' || r == ' ' || r == '\t' || r == '\n' + }), nil +} + +// initPreview sets the process-wide preview gates from --preview. +func initPreview() error { + gates, err := previewGates(previewFlag, pflag.CommandLine.Changed("preview")) + if err != nil { + return err + } + if err := preview.Init(gates...); err != nil { + return fmt.Errorf("--preview: %w", err) + } + return nil +} + +// previewDialOptions returns the options for the ATE API client to opt in to +// preview features, if the install under test enabled any. +func previewDialOptions() []grpc.DialOption { + for _, enabled := range preview.AsMap() { + if enabled { + return preview.DialOptionsWithOptIn() + } + } + return nil +} + +// RequirePreview skips t unless the install under test enabled gate. Tests of +// a preview feature call it first, so a run without the gate skips them. +func RequirePreview(t testing.TB, gate preview.Gate) { + t.Helper() + if !preview.IsEnabled(gate) { + t.Skipf("preview gate %q is not enabled (pass --preview)", gate) + } +} diff --git a/internal/e2e/preview_test.go b/internal/e2e/preview_test.go new file mode 100644 index 0000000000..0efb5b7f08 --- /dev/null +++ b/internal/e2e/preview_test.go @@ -0,0 +1,50 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package e2e + +import ( + "slices" + "testing" + + "k8s.io/utils/ptr" +) + +func TestPreviewGates(t *testing.T) { + for _, tc := range []struct { + name string + ci string + value *string + want []string + wantErr bool + }{ + {name: "unset locally"}, + {name: "unset in CI", ci: "true", wantErr: true}, + {name: "empty in CI", ci: "true", value: ptr.To("")}, + {name: "ALL", ci: "true", value: ptr.To("ALL"), want: []string{"ALL"}}, + {name: "list", value: ptr.To("A, B C"), want: []string{"A", "B", "C"}}, + } { + t.Run(tc.name, func(t *testing.T) { + t.Setenv("CI", tc.ci) + + got, err := previewGates(ptr.Deref(tc.value, ""), tc.value != nil) + if gotErr := err != nil; gotErr != tc.wantErr { + t.Fatalf("previewGates() error = %v, wantErr %v", err, tc.wantErr) + } + if !tc.wantErr && !slices.Equal(got, tc.want) { + t.Errorf("previewGates() = %q, want %q", got, tc.want) + } + }) + } +} diff --git a/internal/e2e/sandbox.go b/internal/e2e/sandbox.go index db5eb8b89d..702b88629b 100644 --- a/internal/e2e/sandbox.go +++ b/internal/e2e/sandbox.go @@ -40,7 +40,7 @@ func IsMicroVM() bool { return SandboxClass() == SandboxClassMicroVM } // Fixture identifies an installed WorkerPool + ActorTemplate pair (both carry // the same name) that suites either create Actors from directly or copy the -// resolved runtime — sandbox class, ateom image, container images — out of. +// resolved runtime — sandbox class, worker image, container images — out of. type Fixture struct { Namespace string Name string @@ -52,7 +52,7 @@ type Fixture struct { // SubstrateFixture identifies an installed substrate ActorTemplate (the proto // resource created through the ate API, not the CRD) plus the CRD WorkerPool // backing it. Suites copy the resolved runtime — container images, sandbox -// config, sandbox size — out of the template, and the ateom image and sandbox +// config, sandbox size — out of the template, and the worker image and sandbox // class out of the pool. type SubstrateFixture struct { // Atespace and Name locate the ActorTemplate for GetActorTemplate. @@ -177,8 +177,8 @@ func RenderFixtureManifest(t *testing.T, relPath, bucket, name string) string { // RenderFixtureManifest treats differently. func fixtureSubstitutions(bucket, name string) (inline, blocks map[string]string) { inline = map[string]string{ - "${BUCKET_NAME}": bucket, - "${ATEOM_IMAGE}": "ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor", + "${BUCKET_NAME}": bucket, + "${WORKER_IMAGE}": "ko://github.com/agent-substrate/substrate/cmd/ateworker-gvisor", // The manifest-side half of FixtureName: it suffixes the fixture's // namespace, and with it the snapshot prefix underneath. "${FIXTURE_SUFFIX}": "-" + name, @@ -186,7 +186,6 @@ func fixtureSubstitutions(bucket, name string) (inline, blocks map[string]string blocks = map[string]string{ "${WORKERPOOL_RUNTIME}": " sandboxClasses:\n - name: gvisor", "${TEMPLATE_SANDBOX_CLASS}": "", - "${TEMPLATE_RESOURCES}": "", // Off unless the caller opts in; see WithTrustBundle. "${TEMPLATE_TRUST_BUNDLE}": "", } @@ -194,7 +193,7 @@ func fixtureSubstitutions(bucket, name string) (inline, blocks map[string]string return inline, blocks } - inline["${ATEOM_IMAGE}"] = "ko://github.com/agent-substrate/substrate/cmd/ateom-microvm" + inline["${WORKER_IMAGE}"] = "ko://github.com/agent-substrate/substrate/cmd/ateworker-microvm" inline["${FIXTURE_SUFFIX}"] = "-" + SandboxClassMicroVM + "-" + name // The micro-VM ActorTemplates name the cluster-wide SandboxConfig // hack/install-microvm-deps.sh installs (configName: microvm), so a @@ -203,10 +202,5 @@ func fixtureSubstitutions(bucket, name string) (inline, blocks map[string]string // Must match the WorkerPool's: a snapshot is not portable across sandbox // classes, so only same-class pools are eligible to run these actors. blocks["${TEMPLATE_SANDBOX_CLASS}"] = " sandboxClass: microvm" - // Only for fixtures that declare no limits of their own. Without them the - // guest boots at ateom's default size (2GiB), and several of those do - // not fit beside the demo pools on CI's single kind node. These size the VM - // itself — see internal/sizing. - blocks["${TEMPLATE_RESOURCES}"] = " resources:\n limits:\n cpu: \"1\"\n memory: 512Mi" return inline, blocks } diff --git a/internal/e2e/sandbox_test.go b/internal/e2e/sandbox_test.go index c035a88039..d04f1058cc 100644 --- a/internal/e2e/sandbox_test.go +++ b/internal/e2e/sandbox_test.go @@ -129,8 +129,8 @@ func TestRenderSubstrateFixtures_GVisor(t *testing.T) { for _, fixture := range substrateFixtures { t.Run(fixture.manifests.Pool, func(t *testing.T) { pool := renderPool(t, fixture.manifests.Pool) - if !strings.HasSuffix(pool.Spec.WorkerImage, "/cmd/ateom-gvisor") { - t.Errorf("WorkerPool workerImage = %q, want the gVisor ateom", pool.Spec.WorkerImage) + if !strings.HasSuffix(pool.Spec.WorkerImage, "/cmd/ateworker-gvisor") { + t.Errorf("WorkerPool workerImage = %q, want the gVisor worker", pool.Spec.WorkerImage) } if pool.Spec.DefaultSandboxClass() != v1alpha1.SandboxClassGvisor { t.Errorf("WorkerPool runtime = class %q, want gvisor", pool.Spec.DefaultSandboxClass()) @@ -178,8 +178,8 @@ func TestRenderSubstrateFixtures_MicroVM(t *testing.T) { for _, fixture := range substrateFixtures { t.Run(fixture.manifests.Pool, func(t *testing.T) { pool := renderPool(t, fixture.manifests.Pool) - if !strings.HasSuffix(pool.Spec.WorkerImage, "/cmd/ateom-microvm") { - t.Errorf("WorkerPool workerImage = %q, want the micro-VM ateom", pool.Spec.WorkerImage) + if !strings.HasSuffix(pool.Spec.WorkerImage, "/cmd/ateworker-microvm") { + t.Errorf("WorkerPool workerImage = %q, want the micro-VM worker", pool.Spec.WorkerImage) } if pool.Spec.DefaultSandboxClass() != SandboxClassMicroVM { t.Errorf("WorkerPool runtime = class %q, want microvm", pool.Spec.DefaultSandboxClass()) @@ -199,11 +199,9 @@ func TestRenderSubstrateFixtures_MicroVM(t *testing.T) { if got := tmpl.GetSandboxConfig().GetConfigName(); got != "microvm" { t.Errorf("template %s configName = %q, want microvm", name, got) } - // Undeclared limits boot the guest at ateom's default size - // (2GiB), which does not fit beside the demo pools on one kind - // node. + // The API rejects a template without limits. if memoryLimit(tmpl) == "" { - t.Errorf("template %s declares no memory limit, so the guest would boot at the kata default", name) + t.Errorf("template %s declares no memory limit", name) } location := tmpl.GetSnapshotConfig().GetStorageLocation() if want := "-microvm-render/"; !strings.HasSuffix(location, want) { diff --git a/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go b/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go index 70e6b5b823..236ae0f6b7 100644 --- a/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go +++ b/internal/e2e/suites/combinedvolumes/combinedvolumes_test.go @@ -31,6 +31,7 @@ import ( "time" "github.com/agent-substrate/substrate/internal/e2e" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "github.com/google/go-containerregistry/pkg/authn" @@ -154,13 +155,18 @@ func buildFixtureImage(t *testing.T, repo string) string { return fmt.Sprintf("%s@%s", tag.Context().Name(), digest) } -// storageClassOrEmpty returns the configured StorageClass if the cluster has -// one, and "" if it does not. The CSI driver is optional, so a missing class +// storageClassOrEmpty returns the configured StorageClass if external volumes +// are enabled and the cluster has the class, and "" if not. External volumes +// are a preview feature and the CSI driver is optional, so either missing // drops the external volume from the template instead of failing every test // in the suite. func storageClassOrEmpty(ctx context.Context, t *testing.T, clients *e2e.Clients) string { t.Helper() + if !preview.IsEnabled(preview.GateExternalVolumes) { + t.Logf("preview gate %q is not enabled; the external-volume cases will be skipped", preview.GateExternalVolumes) + return "" + } if _, err := clients.K8s.StorageV1().StorageClasses().Get(ctx, e2e.StorageClass, metav1.GetOptions{}); err != nil { t.Logf("StorageClass %q not found (%v); the external-volume case will be skipped", e2e.StorageClass, err) return "" @@ -360,6 +366,14 @@ func TestCombinedVolumes(t *testing.T) { payloadPath := mountPath + "/" + payloadName + requireExternalVolume := func(t *testing.T) { + t.Helper() + e2e.RequirePreview(t, preview.GateExternalVolumes) + if storageClass == "" { + t.Skipf("StorageClass %q is not installed", e2e.StorageClass) + } + } + t.Run("DeliversImageContents", func(t *testing.T) { requireContent(ctx, t, router, actorRef, payloadPath, payloadContent) }) @@ -385,9 +399,7 @@ func TestCombinedVolumes(t *testing.T) { }) t.Run("SameExternalVolumeAtTwoPathsSharesWrites", func(t *testing.T) { - if storageClass == "" { - t.Skipf("StorageClass %q is not installed", e2e.StorageClass) - } + requireExternalVolume(t) requireSharedWrite(ctx, t, router, actorRef, extPathA+"/multi.txt", extPathB+"/multi.txt") }) @@ -410,9 +422,7 @@ func TestCombinedVolumes(t *testing.T) { // Suspend detached the external volume; the resume must reattach it // once and restore both of its mounts. t.Run("ExternalVolumeReattached", func(t *testing.T) { - if storageClass == "" { - t.Skipf("StorageClass %q is not installed", e2e.StorageClass) - } + requireExternalVolume(t) requireContentAtBoth(resumeCtx, t, router, actorRef, extPathA+"/multi.txt", extPathB+"/multi.txt", probeWrittenContent) }) }) @@ -454,9 +464,7 @@ func TestCombinedVolumes(t *testing.T) { // it. The actor comes back with its memory and durable dir at the // snapshot but this volume exactly as the discarded execution left it. t.Run("ExternalVolumeNotReverted", func(t *testing.T) { - if storageClass == "" { - t.Skipf("StorageClass %q is not installed", e2e.StorageClass) - } + requireExternalVolume(t) requireContentAtBoth(resumeCtx, t, router, actorRef, extPathA+"/"+postSnapshotName, extPathB+"/"+postSnapshotName, probeWrittenContent) }) }) diff --git a/internal/e2e/suites/demo/demo_test.go b/internal/e2e/suites/demo/demo_test.go index f45cd2d741..48becbc6a4 100644 --- a/internal/e2e/suites/demo/demo_test.go +++ b/internal/e2e/suites/demo/demo_test.go @@ -30,6 +30,7 @@ import ( "github.com/agent-substrate/substrate/internal/atenet" "github.com/agent-substrate/substrate/internal/e2e" "github.com/agent-substrate/substrate/internal/hardware" + "github.com/agent-substrate/substrate/internal/preview" "github.com/agent-substrate/substrate/internal/resources" atev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -427,6 +428,7 @@ func TestMultipleDurableDirLifecycle(t *testing.T) { } func TestExternalVolumeLifecycle(t *testing.T) { + e2e.RequirePreview(t, preview.GateExternalVolumes) t.Parallel() tests := []struct { @@ -454,6 +456,7 @@ func TestExternalVolumeLifecycle(t *testing.T) { } func TestDeleteActorAnyStateWithExternalVolume(t *testing.T) { + e2e.RequirePreview(t, preview.GateExternalVolumes) t.Parallel() ctx := context.Background() clients := e2e.GetClients() @@ -517,6 +520,7 @@ func TestDeleteActorAnyStateWithExternalVolume(t *testing.T) { } func TestExternalVolume_NodeMigration(t *testing.T) { + e2e.RequirePreview(t, preview.GateExternalVolumes) t.Parallel() if e2e.IsMicroVM() { t.Skip("Skipping TestExternalVolume_NodeMigration for microVM environment") @@ -676,7 +680,8 @@ func runActorLifecycleTestCase(t *testing.T, prefix string, createTemplate func( t.Logf("Successfully created Actor: %s", createResp.GetMetadata().GetName()) defer func() { clients.SubstrateAPI.DeleteActor(ctx, &ateapipb.DeleteActorRequest{ - Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID}, + Actor: &ateapipb.ObjectRef{Atespace: demoAtespace, Name: actorID}, + AnyState: true, }) }() diff --git a/internal/e2e/suites/demo/termination_test.go b/internal/e2e/suites/demo/termination_test.go index fdb7c72df9..763179cc96 100644 --- a/internal/e2e/suites/demo/termination_test.go +++ b/internal/e2e/suites/demo/termination_test.go @@ -88,7 +88,7 @@ func TestGracefulWorkerTermination(t *testing.T) { } t.Logf("Actor %q bound to worker pod %s/%s", actorID, podNS, podName) - // Evict the worker pod. The kubelet sends SIGTERM to ateom, which propagates + // Evict the worker pod. The kubelet sends SIGTERM to worker, which propagates // it into the sandbox; the control plane marks the worker DRAINING on the // DeletionTimestamp watch event and cleans up when the pod is finally gone. if err := clients.K8s.CoreV1().Pods(podNS).Delete(ctx, podName, metav1.DeleteOptions{}); err != nil { @@ -196,7 +196,7 @@ func TestGracefulWorkerTerminationSuspend(t *testing.T) { } t.Logf("Actor %q bound to worker pod %s/%s", actorID, podNS, podName) - // Evict the worker pod. The kubelet sends SIGTERM to ateom, which propagates + // Evict the worker pod. The kubelet sends SIGTERM to worker, which propagates // it into the sandbox; the container hangs for 30s. if err := clients.K8s.CoreV1().Pods(podNS).Delete(ctx, podName, metav1.DeleteOptions{}); err != nil { t.Fatalf("failed to delete worker pod %s/%s: %v", podNS, podName, err) diff --git a/internal/e2e/suites/egressauthz/actoridentity_test.go b/internal/e2e/suites/egressauthz/actoridentity_test.go index 15acd5e2a4..5543ed931d 100644 --- a/internal/e2e/suites/egressauthz/actoridentity_test.go +++ b/internal/e2e/suites/egressauthz/actoridentity_test.go @@ -101,7 +101,7 @@ func mintActorCredential(t *testing.T, ca *localca.CA, atespace, actorName strin URIs: []*url.URL{{ Scheme: "spiffe", Host: "substrate-actor.local", - Path: path.Join("ateom-for-actor", atespace, actorName), + Path: path.Join("worker-for-actor", atespace, actorName), }}, NotBefore: time.Now().Add(-5 * time.Minute), NotAfter: time.Now().Add(actorCertificateLifetime), diff --git a/internal/e2e/suites/metrics/metrics_test.go b/internal/e2e/suites/metrics/metrics_test.go index 7ab059e946..737e2257e1 100644 --- a/internal/e2e/suites/metrics/metrics_test.go +++ b/internal/e2e/suites/metrics/metrics_test.go @@ -98,7 +98,7 @@ func TestPlatformMetricsEmitted(t *testing.T) { // The actor lifecycle events the steps above must have produced. The crash // is the one state only ateapi can report, so it is what proves the events - // carry more than the ateom plane could. + // carry more than the worker plane could. wantStates := []string{ ateattr.ActorStateResuming, ateattr.ActorStateRunning, @@ -110,7 +110,7 @@ func TestPlatformMetricsEmitted(t *testing.T) { dataplane := e2e.CurrentAtenetDataplane() prefixes := dataplane.PlatformMetricPrefixes(e2e.PlatformMetricPrefixes) var missing []string - var ateomSeen, controllerSeen, routeDurationSeen, lifecycleSeen bool + var workerSeen, controllerSeen, routeDurationSeen, lifecycleSeen bool var missingStates []string var lastLabelErr error for time.Now().Before(deadline) { @@ -125,7 +125,7 @@ func TestPlatformMetricsEmitted(t *testing.T) { } missingStates = e2e.StatesNotAdvanced(lifecycleBaseline, e2e.LifecycleEventCounts(scrape), wantStates) lifecycleSeen = len(missingStates) == 0 - ateomSeen = e2e.CollectorHasService(scrape, "ateom-gvisor", "ateom-microvm") + workerSeen = e2e.CollectorHasService(scrape, "ateworker-gvisor", "ateworker-microvm") // atecontroller bridges controller-runtime's Prometheus registry onto its OTLP // reader, so the reconcile families are what prove the bridge, not just that // some series arrived. Substring, not prefix: the collector's Prometheus @@ -133,7 +133,7 @@ func TestPlatformMetricsEmitted(t *testing.T) { controllerSeen = e2e.CollectorHasService(scrape, "atecontroller") && strings.Contains(scrape, "controller_runtime_") - if len(missing) == 0 && ateomSeen && controllerSeen && routeDurationSeen && lifecycleSeen { + if len(missing) == 0 && workerSeen && controllerSeen && routeDurationSeen && lifecycleSeen { var errs []string // Verify ate_workerpool_desired_workers carries required namespaced attributes. @@ -253,11 +253,11 @@ func TestPlatformMetricsEmitted(t *testing.T) { } if lastLabelErr != nil { - t.Fatalf("platform telemetry validation failed: missing metrics %v, missing lifecycle states %v, ateom pushed=%v, atecontroller pushed=%v, error detail: %v", - missing, missingStates, ateomSeen, controllerSeen, lastLabelErr) + t.Fatalf("platform telemetry validation failed: missing metrics %v, missing lifecycle states %v, worker pushed=%v, atecontroller pushed=%v, error detail: %v", + missing, missingStates, workerSeen, controllerSeen, lastLabelErr) } - t.Fatalf("platform telemetry validation failed: collector missing metrics %v, missing lifecycle states %v, AgentGateway route duration seen=%v, ateom pushed=%v, atecontroller pushed=%v", - missing, missingStates, routeDurationSeen, ateomSeen, controllerSeen) + t.Fatalf("platform telemetry validation failed: collector missing metrics %v, missing lifecycle states %v, AgentGateway route duration seen=%v, worker pushed=%v, atecontroller pushed=%v", + missing, missingStates, routeDurationSeen, workerSeen, controllerSeen) } func triggerActorCrash(t *testing.T, ctx context.Context, clients *e2e.Clients, actorID string) { @@ -328,12 +328,12 @@ func validateSnapshotPhaseLabels(scrape string) error { } // validateNodeAttribution guards k8s.node.name on the node-scoped resources. -// atelet takes it from the Downward API in its DaemonSet, ateom from the worker +// atelet takes it from the Downward API in its DaemonSet, worker from the worker // pod ate-controller builds. No deployed collector runs k8sattributes, so an // unexpanded or dropped attribute has nothing downstream to restore it. func validateNodeAttribution(scrape string) error { var checked int - for _, svc := range []string{"atelet", "ateom-gvisor", "ateom-microvm"} { + for _, svc := range []string{"atelet", "ateworker-gvisor", "ateworker-microvm"} { if !e2e.CollectorHasService(scrape, svc) { continue } @@ -342,7 +342,7 @@ func validateNodeAttribution(scrape string) error { return fmt.Errorf("%s published telemetry with no k8s_node_name on its target_info; OTEL_RESOURCE_ATTRIBUTES lost k8s.node.name or its $(NODE_NAME) did not expand", svc) } } - // Only one ateom runtime runs per cluster, so the set is checked as it is + // Only one worker runtime runs per cluster, so the set is checked as it is // found; zero of them means the assertion above proved nothing. if checked == 0 { return fmt.Errorf("no node-scoped service reached the collector, so k8s_node_name went unchecked") diff --git a/internal/e2e/suites/networking/accesslog_test.go b/internal/e2e/suites/networking/accesslog_test.go index f355ba6641..3bc5c50b1e 100644 --- a/internal/e2e/suites/networking/accesslog_test.go +++ b/internal/e2e/suites/networking/accesslog_test.go @@ -115,19 +115,19 @@ func TestProtocolGatewayAccessLogMatches(t *testing.T) { { name: "envoy success", container: "envoy", - line: `authority=10.96.0.7:80 code=200 peer_san="spiffe://substrate-actor.local/ateom-for-actor/networking-e2e/ws-1"`, + line: `authority=10.96.0.7:80 code=200 peer_san="spiffe://substrate-actor.local/worker-for-actor/networking-e2e/ws-1"`, want: true, }, { name: "envoy wrong actor identity", container: "envoy", - line: `authority=10.96.0.7:80 code=200 peer_san="spiffe://substrate-actor.local/ateom-for-actor/networking-e2e/ws-2"`, + line: `authority=10.96.0.7:80 code=200 peer_san="spiffe://substrate-actor.local/worker-for-actor/networking-e2e/ws-2"`, want: false, }, { name: "envoy wrong atespace identity", container: "envoy", - line: `authority=10.96.0.7:80 code=200 peer_san="spiffe://substrate-actor.local/ateom-for-actor/other/ws-1"`, + line: `authority=10.96.0.7:80 code=200 peer_san="spiffe://substrate-actor.local/worker-for-actor/other/ws-1"`, want: false, }, { diff --git a/internal/e2e/suites/networking/networking_test.go b/internal/e2e/suites/networking/networking_test.go index 088631868a..5a0eceaefe 100644 --- a/internal/e2e/suites/networking/networking_test.go +++ b/internal/e2e/suites/networking/networking_test.go @@ -374,7 +374,7 @@ func assertEgressGatewayConnect(t *testing.T, ctx context.Context, since metav1. if !strings.HasSuffix(authority, ":"+port) { continue } - spiffeSlug := "/ateom-for-actor/" + atespace + "/" + actorName + spiffeSlug := "/worker-for-actor/" + atespace + "/" + actorName if !strings.Contains(line.text, spiffeSlug) { continue } @@ -963,7 +963,7 @@ func postEgressWithStatus(t *testing.T, ctx context.Context, router *e2e.RouterC func assertProtocolGateway(t *testing.T, ctx context.Context, since metav1.Time, actorName, authority string) { t.Helper() actorRef := resources.ActorRef{Atespace: networkingAtespace, Name: actorName} - identity := resources.AteomForActorSPIFFEID(actorRef).String() + identity := resources.WorkerForActorSPIFFEID(actorRef).String() // The authenticated CONNECT record identifies the actor and destination. // Callers separately verify the HTTPS response using only the gateway's CA, // so a successful TLS handshake proves interception rather than passthrough. @@ -988,7 +988,7 @@ func protocolGatewayAccessLogMatches(entry gatewayAccessLogLine, actorRef resour return false } for peer := range strings.SplitSeq(peers, ",") { - if peer == resources.AteomForActorSPIFFEID(actorRef).String() { + if peer == resources.WorkerForActorSPIFFEID(actorRef).String() { return true } } diff --git a/internal/e2e/suites/networkpolicy/networkpolicy_test.go b/internal/e2e/suites/networkpolicy/networkpolicy_test.go index ff021982c7..3f9efb1b6d 100644 --- a/internal/e2e/suites/networkpolicy/networkpolicy_test.go +++ b/internal/e2e/suites/networkpolicy/networkpolicy_test.go @@ -16,10 +16,13 @@ package networkpolicy import ( "context" + "encoding/json" "fmt" "io" + "net" "net/http" - "os/exec" + "net/url" + "strconv" "strings" "testing" "time" @@ -28,8 +31,11 @@ import ( networkingv1 "k8s.io/api/networking/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" + "github.com/agent-substrate/substrate/internal/ateclient" "github.com/agent-substrate/substrate/internal/e2e" + "github.com/agent-substrate/substrate/internal/portforward" "github.com/agent-substrate/substrate/internal/resources" "github.com/agent-substrate/substrate/pkg/api/v1alpha1" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" @@ -52,7 +58,7 @@ func TestNetworkPolicyLifecycleAndReconciliation(t *testing.T) { }, Spec: v1alpha1.WorkerPoolSpec{ Replicas: 1, - WorkerImage: "ateom:v1", + WorkerImage: "worker:v1", SandboxClasses: []v1alpha1.WorkerPoolSandboxClass{{Name: v1alpha1.SandboxClassGvisor}}, }, } @@ -207,7 +213,6 @@ func TestNetworkPolicyDataPlaneEnforcement(t *testing.T) { // === Negative Data Plane Verification (Unauthorized Ingress Blocking) === t.Log("=== Verifying unauthorized ingress blocking ===") - rogueNsObj := e2e.CreateNamespace(t) // Find the IP address of the worker pod backing our WorkerPool pods, err := clients.K8s.CoreV1().Pods(nsObj.Name).List(ctx, metav1.ListOptions{ @@ -228,46 +233,121 @@ func TestNetworkPolicyDataPlaneEnforcement(t *testing.T) { t.Fatalf("no running worker pod with IP found for worker pool %q", poolName) } - // Deploy an unauthorized probe pod in an external test namespace - probePod := &corev1.Pod{ - ObjectMeta: metav1.ObjectMeta{ - Name: "unauthorized-probe", - Namespace: rogueNsObj.Name, - }, - Spec: corev1.PodSpec{ - Containers: []corev1.Container{ - { - Name: "probe", - Image: "busybox@sha256:1487d0af5f52b4ba31c7e465126ee2123fe3f2305d638e7827681e7cf6c83d5e", - Command: []string{"/bin/sleep", "3600"}, - }, - }, - RestartPolicy: corev1.RestartPolicyNever, - }, + // The probe runs in a new namespace, which the policy does not allow in. + // The control pod runs in the worker's namespace, but no policy selects + // it. If the probe can reach the control pod, a failed dial to the worker + // is caused by the policy, not by a probe that cannot connect at all. + probe := e2e.DeployServerPod(t, ctx, testserverPod("unauthorized-probe", "")) + control := e2e.DeployServerPod(t, ctx, testserverPod("netpol-control", nsObj.Name)) + controlPod, err := clients.K8s.CoreV1().Pods(control.Namespace).Get(ctx, control.Name, metav1.GetOptions{}) + if err != nil || controlPod.Status.PodIP == "" { + t.Fatalf("failed to get the IP of control pod %s/%s: %v", control.Namespace, control.Name, err) } - t.Logf("Creating unauthorized probe pod %s/%s...", rogueNsObj.Name, probePod.Name) - if _, err := clients.K8s.CoreV1().Pods(rogueNsObj.Name).Create(ctx, probePod, metav1.CreateOptions{}); err != nil { - t.Fatalf("failed to create probe pod: %v", err) + pc := newProbeClient(t, ctx, probe) + + controlAddress := net.JoinHostPort(controlPod.Status.PodIP, strconv.Itoa(testserverPort)) + t.Logf("Dialing control pod %s/%s at %s from probe pod %s/%s...", control.Namespace, control.Name, controlAddress, probe.Namespace, probe.Name) + var result dialResult + controlDeadline := time.Now().Add(30 * time.Second) + for { + result = pc.dial(t, ctx, controlAddress) + if result.OK || time.Now().After(controlDeadline) { + break + } + time.Sleep(1 * time.Second) + } + if !result.OK { + t.Fatalf("Probe control failed: probe pod %s/%s cannot reach control pod %s, which no NetworkPolicy selects (%s), so a blocked dial to the worker would prove nothing", probe.Namespace, probe.Name, controlAddress, result.Error) + } + + workerAddress := net.JoinHostPort(workerIP, workerReadyzPort) + t.Logf("Attempting unauthorized connection from %s/%s to worker %s...", probe.Namespace, probe.Name, workerAddress) + result = pc.dial(t, ctx, workerAddress) + if result.OK { + t.Fatalf("Negative Data Plane Verification failed: unauthorized probe pod %s/%s successfully connected to worker pod %s. NetworkPolicy did not block the connection!", probe.Namespace, probe.Name, workerAddress) + } + t.Logf("Negative Data Plane Verification PASSED: Unauthorized connection attempt from %s/%s to %s was blocked as expected (%s)", probe.Namespace, probe.Name, workerAddress, result.Error) +} + +const ( + // testserverPort is the port the probe and control pods listen on. + testserverPort = 8080 + // workerReadyzPort is the port of worker's /readyz server. + workerReadyzPort = "8080" +) + +// testserverPod describes a testserver http pod, used for both the probe, +// which serves /dial, and the control, which is the dial target. An empty +// namespace makes DeployServerPod create a new one. +func testserverPod(name, namespace string) e2e.ServerPod { + return e2e.ServerPod{ + Name: name, + Namespace: namespace, + ImportPath: "github.com/agent-substrate/substrate/internal/e2e/fixtures/testserver", + Args: []string{"http"}, + Port: testserverPort, } - waitForPodRunning(ctx, t, clients, rogueNsObj.Name, probePod.Name) +} - // Attempt direct connection to worker pod IP from unauthorized probe pod - execArgs := []string{"exec", "-n", rogueNsObj.Name, probePod.Name} - if e2e.KubeContext != "" { - execArgs = append([]string{"--context=" + e2e.KubeContext}, execArgs...) +// dialResult mirrors testserver's /dial response. +type dialResult struct { + Address string `json:"address"` + OK bool `json:"ok"` + Error string `json:"error"` +} + +// probeClient calls a testserver pod's /dial over a port-forward, so the +// connection is made from that pod. +type probeClient struct { + baseURL string + http *http.Client +} + +func newProbeClient(t *testing.T, ctx context.Context, probe e2e.Server) *probeClient { + t.Helper() + config, err := ateclient.LoadKubeConfig(e2e.KubeConfig, e2e.KubeContext) + if err != nil { + t.Fatalf("loading kubeconfig: %v", err) } - if e2e.KubeConfig != "" { - execArgs = append([]string{"--kubeconfig=" + e2e.KubeConfig}, execArgs...) + clientset, err := kubernetes.NewForConfig(config) + if err != nil { + t.Fatalf("creating k8s client for port-forward: %v", err) + } + localPort, stop, err := portforward.ServicePortForward(ctx, config, clientset, probe.Namespace, probe.Name, testserverPort) + if err != nil { + t.Fatalf("port-forwarding %s/%s: %v", probe.Namespace, probe.Name, err) } - execArgs = append(execArgs, "--", "sh", "-c", fmt.Sprintf("wget -T 3 -q -O - http://%s:8080/ || nc -z -w 3 %s 8080", workerIP, workerIP)) + t.Cleanup(stop) + return &probeClient{ + baseURL: fmt.Sprintf("http://127.0.0.1:%d", localPort), + http: &http.Client{Timeout: 30 * time.Second}, + } +} - t.Logf("Executing unauthorized connection attempt from %s/%s to %s:8080...", rogueNsObj.Name, probePod.Name, workerIP) - cmd := exec.Command("kubectl", execArgs...) - output, err := cmd.CombinedOutput() - if err == nil { - t.Fatalf("Negative Data Plane Verification failed: unauthorized probe pod successfully connected to worker pod %s:8080 (output: %s). NetworkPolicy did not block the connection!", workerIP, string(output)) +// dial asks the probe to open a TCP connection to address. A refused or +// timed-out connection comes back as a result with OK false. Anything else +// going wrong fails the test, so a broken probe is not mistaken for a blocked +// connection. +func (c *probeClient) dial(t *testing.T, ctx context.Context, address string) dialResult { + t.Helper() + req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+"/dial?address="+url.QueryEscape(address), nil) + if err != nil { + t.Fatalf("building probe request for %s: %v", address, err) + } + resp, err := c.http.Do(req) + if err != nil { + t.Fatalf("calling probe /dial for %s: %v", address, err) + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + body, _ := io.ReadAll(io.LimitReader(resp.Body, 4<<10)) + t.Fatalf("probe /dial for %s returned %d: %s", address, resp.StatusCode, strings.TrimSpace(string(body))) + } + var out dialResult + if err := json.NewDecoder(resp.Body).Decode(&out); err != nil { + t.Fatalf("decoding probe /dial response for %s: %v", address, err) } - t.Logf("Negative Data Plane Verification PASSED: Unauthorized connection attempt from %s/%s to %s:8080 was blocked as expected (err: %v)", rogueNsObj.Name, probePod.Name, workerIP, err) + return out } // setupDemoCounterTemplate provisions the per-test WorkerPool and substrate @@ -303,17 +383,3 @@ func waitForActorRunning(ctx context.Context, t *testing.T, clients *e2e.Clients } t.Fatalf("timed out waiting for actor %q to reach RUNNING state", actorName) } - -func waitForPodRunning(ctx context.Context, t *testing.T, clients *e2e.Clients, namespace, podName string) { - t.Helper() - deadline := time.Now().Add(60 * time.Second) - for time.Now().Before(deadline) { - pod, err := clients.K8s.CoreV1().Pods(namespace).Get(ctx, podName, metav1.GetOptions{}) - if err == nil && pod.Status.Phase == corev1.PodRunning { - t.Logf("Pod %s/%s is RUNNING", namespace, podName) - return - } - time.Sleep(1 * time.Second) - } - t.Fatalf("timed out waiting for pod %s/%s to reach RUNNING status", namespace, podName) -} diff --git a/internal/e2e/suites/sandboxconfig/classdefault_test.go b/internal/e2e/suites/sandboxconfig/classdefault_test.go index 1d97bb8f5e..27bba75312 100644 --- a/internal/e2e/suites/sandboxconfig/classdefault_test.go +++ b/internal/e2e/suites/sandboxconfig/classdefault_test.go @@ -85,7 +85,7 @@ func TestWorkerPoolFollowsClassDefault(t *testing.T) { Spec: atev1alpha1.WorkerPoolSpec{ Replicas: 0, // Never pulled: the pool runs no workers. - WorkerImage: "ateom:unused", + WorkerImage: "worker:unused", SandboxClasses: []atev1alpha1.WorkerPoolSandboxClass{{Name: class}}, }, }, metav1.CreateOptions{}) diff --git a/internal/e2e/suites/security/README.md b/internal/e2e/suites/security/README.md index 0c06735843..e41410f208 100644 --- a/internal/e2e/suites/security/README.md +++ b/internal/e2e/suites/security/README.md @@ -41,7 +41,7 @@ Note the setup (victim run/write/suspend, attacker reuse) runs regardless — th kubectl get ns -l ate.dev/e2e -kubectl -n logs -c ateom +kubectl -n logs -c ateworker KIND_CLUSTER_NAME=kind hack/cleanup-e2e.sh # reclaim when done @@ -105,13 +105,13 @@ The shared worker pod on the host # The pod hosting both actors in turn kubectl get pods -n ate-e2e-sec-a -o wide -# ateom logs — the sandbox lifecycle, checkpoints, resumes -kubectl logs -n ate-e2e-sec-a -l app=ateom -c ateom -f +# worker logs — the sandbox lifecycle, checkpoints, resumes +kubectl logs -n ate-e2e-sec-a -l app=ateworker -c ateworker -f # The on-host per-actor layout the host-state probes assert against. # Two UID-keyed durable dirs (victim's + attacker's) live side by side here: -POD=$(kubectl get pods -n ate-e2e-sec-a -l app=ateom -o name | head -1) -kubectl exec -n ate-e2e-sec-a "$POD" -c ateom -- ls -la /var/lib/ateom-gvisor/actors +POD=$(kubectl get pods -n ate-e2e-sec-a -l app=ateworker -o name | head -1) +kubectl exec -n ate-e2e-sec-a "$POD" -c ateworker -- ls -la /var/lib/ateworker-gvisor/actors Control plane / snapshots (for the tag-scope subtest) kubectl ate get tags -a ate-e2e-sec-a # atespace-a-tag appears during the last subtest diff --git a/internal/e2e/template.go b/internal/e2e/template.go index 4bc24dc076..5339e2b89c 100644 --- a/internal/e2e/template.go +++ b/internal/e2e/template.go @@ -60,7 +60,7 @@ func CreateSubstrateCounterTemplate(ctx context.Context, t *testing.T, clients * } // CreateSubstrateTemplateFrom creates a per-test WorkerPool CRD plus a -// substrate ActorTemplate copying the resolved runtime (sandbox config, ateom +// substrate ActorTemplate copying the resolved runtime (sandbox config, worker // image, container images, sandbox size) from the installed fixture src. It // registers cleanup of the template (which does not ride the k8s namespace GC // the CRD templates did) and blocks until the golden snapshot exists. @@ -107,9 +107,7 @@ func CreateSubstrateTemplateFrom(ctx context.Context, t *testing.T, clients *Cli Metadata: &ateapipb.ResourceMetadata{Atespace: opts.Atespace, Name: opts.Name}, WorkerSelector: &ateapipb.Selector{MatchLabels: opts.Labels}, Containers: srcTmpl.GetContainers(), - // The source's limits size the sandbox. Copying them matters most on - // micro-VM, where an ActorTemplate that declares none boots the guest - // at ateom's default guest size (2GiB) instead of the demo's 512Mi. + // The source's limits size the sandbox, and the API requires them. Resources: srcTmpl.GetResources(), // The source carries the sandbox_class/config_name pair for the // class under test. diff --git a/internal/e2e/testmain.go b/internal/e2e/testmain.go index bcdaba98a2..6e97235f9c 100644 --- a/internal/e2e/testmain.go +++ b/internal/e2e/testmain.go @@ -60,6 +60,7 @@ func bindFlags() { pflag.StringVar(&KubeConfig, "kube-config", "", "Location of the kubeconfig") pflag.StringVar(&KubeContext, "kube-context", "", "Kubernetes context to use") pflag.StringVar(&StorageClass, "storage-class", "csi-nfs-sc", "StorageClass to use for external volume tests") + pflag.StringVar(&previewFlag, "preview", "", `Preview gates the install under test enabled, as in ATE_PREVIEW ("ALL" for all, "" for none); required in CI`) } // RunTestMain should be used to run your e2e test suite. @@ -80,6 +81,11 @@ func RunTestMain(m *testing.M) int { return 0 } + if err := initPreview(); err != nil { + fmt.Println(Colorf("%v\n", err)) + return 1 + } + return runAndCleanup(m) } @@ -102,7 +108,7 @@ func runAndCleanup(m *testing.M) int { } // Namespaces are deleted only on success. A failed run keeps them: the actor - // lives in a worker pod there, and its ateom logs (for a micro-VM worker, the + // lives in a worker pod there, and its worker logs (for a micro-VM worker, the // guest's console tail too) are the only record of why it failed. Deleting the // namespace takes those pods with it before anyone — a developer or CI's // post-failure log dump — can read them. diff --git a/internal/egresspolicy/egresspolicy.go b/internal/egresspolicy/egresspolicy.go index 3655d15261..dd9ae88cf2 100644 --- a/internal/egresspolicy/egresspolicy.go +++ b/internal/egresspolicy/egresspolicy.go @@ -442,7 +442,7 @@ func lowerASCII(s string) string { } // DNSEgressPolicyEnforceInAtunnel is the --dns-egress-policy mode that -// enforces an Actor's EgressPolicy on DNS queries in the ateom DNS relay. +// enforces an Actor's EgressPolicy on DNS queries in the worker DNS relay. const DNSEgressPolicyEnforceInAtunnel = "enforce-in-atunnel" // ValidateDNSEgressPolicy checks the --dns-egress-policy flag value. diff --git a/internal/imagecache/README.md b/internal/imagecache/README.md index 1b81f4c272..4d9e5cc8db 100644 --- a/internal/imagecache/README.md +++ b/internal/imagecache/README.md @@ -37,13 +37,13 @@ What it buys, concretely: The design is shaped by an existing substrate boundary: **atelet runs as plain root with every Linux capability dropped** ("atelet does no mounts" — -see `manifests/ate-install/atelet.yaml`), while the **ateom worker pods are +see `manifests/ate-install/atelet.yaml`), while the **worker pods are privileged** and own all mounts on the node. The module is split accordingly: | Half | Runs in | Files | Needs | |---|---|---|---| | Store: pull, parse, unpack, record | atelet | `imagecache.go`, `unpack.go`, `spec.go` (portable) | nothing but file I/O | -| Consumer: finalize, mount, unmount | ateom-gvisor / ateom-microvm | `bundle_linux.go` (`//go:build linux`) | `CAP_MKNOD`, `CAP_SYS_ADMIN` | +| Consumer: finalize, mount, unmount | ateworker-gvisor / ateworker-microvm | `bundle_linux.go` (`//go:build linux`) | `CAP_MKNOD`, `CAP_SYS_ADMIN` | The two halves communicate through the filesystem only: the shared cache directory (on the `/var/lib/ate` hostPath, so the same absolute @@ -112,7 +112,7 @@ directories bottom-first plus any `ExtraDirs` (in-rootfs bind-mount targets, e.g. the actor identity mount at `/run/ate`), and creates the empty bundle-local `rootfs/`, `upper/`, and `work/` directories. -## Compose path (ateom: `SetupBundleRootfs`) +## Compose path (ateworker: `SetupBundleRootfs`) Called immediately before `runsc create`/`runsc restore` (gVisor) and before staging the virtio-fs lower (micro-VM): @@ -120,7 +120,7 @@ staging the virtio-fs lower (micro-VM): 1. **`FinalizeLayer`** for each referenced layer — materializes the recorded whiteouts as 0:0 char devices (`mknod`) and opaque dirs as `trusted.overlay.opaque=y` xattrs. Once per layer node-wide; idempotent - and safe under concurrent ateom pods (`EEXIST` tolerated, marker written + and safe under concurrent worker pods (`EEXIST` tolerated, marker written last). Paths from `whiteouts.json` are re-validated, so a crafted file cannot escape the layer tree. 2. **Mount** an overlay at `/rootfs`: `lowerdir` is the layer chain @@ -162,8 +162,8 @@ keeps building its own tmpfs upper, as before. **Teardown**: `UnmountAllUnder(bundleDir)` lazily detaches every mount below an actor's bundle directory (via `/proc/self/mountinfo`) before atelet wipes -it — called from the checkpoint cleanup path in ateom-gvisor and -`teardownActor` in ateom-microvm. +it — called from the checkpoint cleanup path in ateworker-gvisor and +`teardownActor` in ateworker-microvm. ## Garbage collection @@ -196,9 +196,9 @@ percentage as `df` reports it. 1. **Root set** (`Store.InUse`): scan every bundle's `rootfs-overlay.json` under the actors dir (`WithActorsDir`). - Overlay mounts live in the ateom pods' mount namespaces, so atelet + Overlay mounts live in the worker pods' mount namespaces, so atelet cannot see them in its own `/proc/mounts`; the bundle specs are - written by atelet itself before any ateom is asked to mount and + written by atelet itself before any ateworker is asked to mount and removed only after unmount, so they are the authoritative "actively mounted" set. A spec roots its image digest, each layer dir it names, and its *exact* layer set — the last also roots the diff --git a/internal/imagecache/bundle_linux.go b/internal/imagecache/bundle_linux.go index f4edfd4052..118cf0c9e0 100644 --- a/internal/imagecache/bundle_linux.go +++ b/internal/imagecache/bundle_linux.go @@ -15,7 +15,7 @@ // limitations under the License. // The consumer half of the image cache: everything in this file runs in the -// privileged ateom pods (which own all mounts on the node), never in atelet. +// privileged worker pods (which own all mounts on the node), never in atelet. package imagecache @@ -228,11 +228,11 @@ func fsContextLog(fsfd int) string { // FinalizeLayer materializes the whiteout state recorded at unpack time: // 0:0 char devices for whiteouts and trusted.overlay.opaque=y on opaque -// dirs. This runs in ateom rather than atelet because mknod needs CAP_MKNOD +// dirs. This runs in worker rather than atelet because mknod needs CAP_MKNOD // and trusted.* xattrs need CAP_SYS_ADMIN, both of which atelet deliberately // drops. // -// Idempotent and safe under concurrent callers (multiple ateom pods share +// Idempotent and safe under concurrent callers (multiple worker pods share // the node's pool): EEXIST from mknod is success, setxattr is naturally // idempotent, and the marker is written last. func FinalizeLayer(layerDir string) error { diff --git a/internal/imagecache/gc.go b/internal/imagecache/gc.go index bdc1d6d192..b17e02f643 100644 --- a/internal/imagecache/gc.go +++ b/internal/imagecache/gc.go @@ -23,7 +23,7 @@ package imagecache // is evictable unless vetoed by, in order: // // 1. the root set — bundle overlay specs under the actors dir, the same -// authority that hands out mounts (atelet cannot see the ateoms' mount +// authority that hands out mounts (atelet cannot see the workers' mount // namespaces); // 2. min-age — records and layers younger than minAge are never touched, // covering the pull → spec-write → mount window; diff --git a/internal/imagecache/gc_test.go b/internal/imagecache/gc_test.go index 8b0b6c8b90..5f6e1146d7 100644 --- a/internal/imagecache/gc_test.go +++ b/internal/imagecache/gc_test.go @@ -564,7 +564,7 @@ func TestConcurrentEnsureImageAndEvict(t *testing.T) { } // The returned image must reference live, complete layer dirs. Its // record must be fresh (either the hit's touch or the re-pull wrote - // it), which is what protects it until an ateom would mount it. + // it), which is what protects it until a worker would mount it. for _, dir := range img.LayerDirs { if _, err := os.Stat(filepath.Join(dir, layerFSDirName, "f")); err != nil { t.Fatalf("iteration %d: returned layer dir unusable: %v", i, err) diff --git a/internal/imagecache/imagecache.go b/internal/imagecache/imagecache.go index 16793503ca..d6601ef2c0 100644 --- a/internal/imagecache/imagecache.go +++ b/internal/imagecache/imagecache.go @@ -14,10 +14,10 @@ // Package imagecache implements the node-local OCI image cache: a // content-addressed pool of unpacked image layers shared by every actor on -// the node, plus the per-bundle overlay spec that tells the ateom runtimes +// the node, plus the per-bundle overlay spec that tells the worker runtimes // how to compose an actor rootfs from cached layers. // -// The work is split along the existing atelet/ateom privilege boundary: +// The work is split along the existing atelet/worker privilege boundary: // // - atelet (plain root, all capabilities dropped) pulls layers and unpacks // them into the pool (Store.EnsureImage), and writes a rootfs-overlay.json @@ -25,19 +25,19 @@ // recorded in per-layer metadata rather than materialized, because // overlayfs whiteouts are char devices (CAP_MKNOD) with trusted.* xattrs // for opaque dirs (CAP_SYS_ADMIN). -// - ateom (privileged; it already owns every mount on the node) finalizes +// - worker (privileged; it already owns every mount on the node) finalizes // layers — materializing the recorded whiteout state, once per layer — // and mounts the overlay rootfs (SetupBundleRootfs) just before // `runsc create` / staging the micro-VM virtio-fs lower. // // On-disk layout under the cache root (a directory on the BasePath hostPath, -// so the same absolute paths resolve in atelet and every ateom pod): +// so the same absolute paths resolve in atelet and every worker pod): // // version layout version marker // layers/sha256// // fs/ the unpacked layer tree (overlay lowerdir) // whiteouts.json whiteout state recorded at unpack time -// finalized marker written by FinalizeLayer (ateom) +// finalized marker written by FinalizeLayer (worker) // manifests/sha256/.json // image config + ordered diffID list // @@ -197,7 +197,7 @@ type Store struct { // minAge vetoes eviction of any layer or image record younger than this, // covering the window between a pull (or cache-hit stat) and the bundle - // spec write / ateom mount that roots it. + // spec write / worker mount that roots it. minAge time.Duration // pullTimeout bounds each pull. Pulls run detached from the contexts of @@ -495,7 +495,7 @@ func (s *Store) EnsureImage(ctx context.Context, ref string) (_ *Image, err erro // with respect to eviction's record removal (removeStaleRecord holds // hitMu exclusive). Refreshing the mtime also renews the min-age veto, so // an image in active use cannot age into eviction between this stat and -// the ateom's mount. +// the worker's mount. func (s *Store) cachedImageHit(digest v1.Hash) (*Image, error) { s.hitMu.RLock() defer s.hitMu.RUnlock() diff --git a/internal/imagecache/mountinfo_test.go b/internal/imagecache/mountinfo_test.go index fdaf6db256..cc7dfa492c 100644 --- a/internal/imagecache/mountinfo_test.go +++ b/internal/imagecache/mountinfo_test.go @@ -23,25 +23,25 @@ import ( func TestMountPointsIn(t *testing.T) { mountinfo := strings.Join([]string{ `22 26 0:20 / /sys rw,nosuid,nodev shared:7 - sysfs sysfs rw`, - `40 21 8:1 / /var/lib/ateom-gvisor/actors/a1/bundles/task/rootfs rw shared:1 - overlay overlay rw,lowerdir=/x`, - `41 21 8:1 / /var/lib/ateom-gvisor/actors/a1/bundles/pause/rootfs rw - overlay overlay rw`, - `42 21 8:1 / /var/lib/ateom-gvisor/actors/a2/bundles/task/rootfs rw - overlay overlay rw`, + `40 21 8:1 / /var/lib/ateworker-gvisor/actors/a1/bundles/task/rootfs rw shared:1 - overlay overlay rw,lowerdir=/x`, + `41 21 8:1 / /var/lib/ateworker-gvisor/actors/a1/bundles/pause/rootfs rw - overlay overlay rw`, + `42 21 8:1 / /var/lib/ateworker-gvisor/actors/a2/bundles/task/rootfs rw - overlay overlay rw`, // Path with an escaped space; mountinfo octal-escapes it. - `43 21 8:1 / /var/lib/ateom-gvisor/actors/a1/bundles/odd\040name/rootfs rw - overlay overlay rw`, + `43 21 8:1 / /var/lib/ateworker-gvisor/actors/a1/bundles/odd\040name/rootfs rw - overlay overlay rw`, // Prefix-sibling directory that must NOT match a1's subtree. - `44 21 8:1 / /var/lib/ateom-gvisor/actors/a1-sibling/bundles/x/rootfs rw - overlay overlay rw`, + `44 21 8:1 / /var/lib/ateworker-gvisor/actors/a1-sibling/bundles/x/rootfs rw - overlay overlay rw`, `malformed line`, ``, }, "\n") - got, err := mountPointsIn(strings.NewReader(mountinfo), "/var/lib/ateom-gvisor/actors/a1/bundles") + got, err := mountPointsIn(strings.NewReader(mountinfo), "/var/lib/ateworker-gvisor/actors/a1/bundles") if err != nil { t.Fatalf("mountPointsIn: %v", err) } want := []string{ - "/var/lib/ateom-gvisor/actors/a1/bundles/task/rootfs", - "/var/lib/ateom-gvisor/actors/a1/bundles/pause/rootfs", - "/var/lib/ateom-gvisor/actors/a1/bundles/odd name/rootfs", + "/var/lib/ateworker-gvisor/actors/a1/bundles/task/rootfs", + "/var/lib/ateworker-gvisor/actors/a1/bundles/pause/rootfs", + "/var/lib/ateworker-gvisor/actors/a1/bundles/odd name/rootfs", } if !slices.Equal(got, want) { t.Errorf("mount points = %v, want %v", got, want) diff --git a/internal/imagecache/spec.go b/internal/imagecache/spec.go index e1e0873440..e2ee2d450a 100644 --- a/internal/imagecache/spec.go +++ b/internal/imagecache/spec.go @@ -25,7 +25,7 @@ import ( ) // ImageVolumeMountPath is where one image volume is composed inside a -// bundle. The ateom mounts the volume there (see setupImageVolumes) and binds +// bundle. The worker mounts the volume there (see setupImageVolumes) and binds // it into the container. func ImageVolumeMountPath(bundlePath, volumeName string) string { return filepath.Join(bundlePath, "volumes", volumeName) @@ -38,7 +38,7 @@ func ImageVolumeMountPath(bundlePath, volumeName string) string { const OverlaySpecFileName = "rootfs-overlay.json" // OverlaySpec is the contract between atelet (which cannot mount) and the -// ateom runtimes (which mount the rootfs overlay just before running the +// worker runtimes (which mount the rootfs overlay just before running the // workload). The overlay's mountpoint, upperdir, and workdir are always the // bundle-local rootfs/, upper/, and work/ directories — derived from the // bundle path by the consumer rather than trusted from the file. diff --git a/internal/imagecache/unpack.go b/internal/imagecache/unpack.go index 81e64cf6c5..9b95ccd462 100644 --- a/internal/imagecache/unpack.go +++ b/internal/imagecache/unpack.go @@ -332,7 +332,7 @@ func unpackLayer(ctx context.Context, tarData io.Reader, root *os.Root) (*whiteo // restoreModTime sets the atime and mtime of an unpacked path to the mtime its // tar entry records, so that every node unpacks a layer to the same tree (see -// the deterministic-unpack note in cmd/ateom-microvm/restore.go). +// the deterministic-unpack note in cmd/ateworker-microvm/restore.go). func restoreModTime(root *os.Root, name string, modTime time.Time) error { if modTime.IsZero() { return nil diff --git a/internal/installdefaults/installdefaults_test.go b/internal/installdefaults/installdefaults_test.go index b6c89860a0..165265abc2 100644 --- a/internal/installdefaults/installdefaults_test.go +++ b/internal/installdefaults/installdefaults_test.go @@ -47,7 +47,7 @@ func TestAteletSPIFFEID(t *testing.T) { } func TestRouterSPIFFEID(t *testing.T) { - // Matches the --atunnel-client-identity default the ateom binaries ship + // Matches the --atunnel-client-identity default the worker binaries ship // with, which is what actor ingress authenticates the router against. const want = "spiffe://cluster.local/ns/ate-system/sa/atenet-router" if got := RouterSPIFFEID(SystemNamespace); got != want { diff --git a/internal/ateomnet/netns/dial.go b/internal/netns/dial.go similarity index 100% rename from internal/ateomnet/netns/dial.go rename to internal/netns/dial.go diff --git a/internal/ateomnet/netns/dial_linux_test.go b/internal/netns/dial_linux_test.go similarity index 100% rename from internal/ateomnet/netns/dial_linux_test.go rename to internal/netns/dial_linux_test.go diff --git a/internal/ateomnet/netns/doc.go b/internal/netns/doc.go similarity index 100% rename from internal/ateomnet/netns/doc.go rename to internal/netns/doc.go diff --git a/internal/ateomnet/netns/netns.go b/internal/netns/netns.go similarity index 100% rename from internal/ateomnet/netns/netns.go rename to internal/netns/netns.go diff --git a/internal/ateomnet/netns/netns_linux_test.go b/internal/netns/netns_linux_test.go similarity index 94% rename from internal/ateomnet/netns/netns_linux_test.go rename to internal/netns/netns_linux_test.go index 80ecde8d5d..a2a38b5321 100644 --- a/internal/ateomnet/netns/netns_linux_test.go +++ b/internal/netns/netns_linux_test.go @@ -28,7 +28,7 @@ import ( ) func TestNamedNetNSRejectsInvalidNames(t *testing.T) { - for _, name := range []string{"", ".", "..", "/absolute", "../outside", "nested/name", "ateom-actor:uid/../../outside", "nul\x00name"} { + for _, name := range []string{"", ".", "..", "/absolute", "../outside", "nested/name", "worker-actor:uid/../../outside", "nul\x00name"} { t.Run(name, func(t *testing.T) { if err := RemoveNamed(name); !errors.Is(err, os.ErrInvalid) { t.Fatalf("RemoveNamed(%q): got %v, want invalid name", name, err) @@ -46,8 +46,8 @@ func TestNamedNetNSRejectsInvalidNames(t *testing.T) { func TestRemoveNamedDoesNotFollowSymlinks(t *testing.T) { roottest.Require(t, "creates network namespaces") - const targetName = "ateomnet-symlink-target-test" - const linkName = "ateomnet-symlink-test" + const targetName = "actornet-symlink-target-test" + const linkName = "actornet-symlink-test" targetPath := "/run/netns/" + targetName linkPath := "/run/netns/" + linkName target, err := CreateNamed(targetName) @@ -83,7 +83,7 @@ func TestCreateNamedReplacesALeftover(t *testing.T) { roottest.Require(t, "creates network namespaces") for _, state := range []string{"mounted", "unmounted"} { t.Run(state, func(t *testing.T) { - name := "ateomnet-leftover-test-" + state + name := "actornet-leftover-test-" + state path := "/run/netns/" + name t.Cleanup(func() { _ = RemoveNamed(name) }) diff --git a/internal/ateomnet/netns/sysctl.go b/internal/netns/sysctl.go similarity index 100% rename from internal/ateomnet/netns/sysctl.go rename to internal/netns/sysctl.go diff --git a/internal/ateomnet/netns/sysctl_linux_test.go b/internal/netns/sysctl_linux_test.go similarity index 98% rename from internal/ateomnet/netns/sysctl_linux_test.go rename to internal/netns/sysctl_linux_test.go index 01cae6eb0f..28820847c5 100644 --- a/internal/ateomnet/netns/sysctl_linux_test.go +++ b/internal/netns/sysctl_linux_test.go @@ -32,7 +32,7 @@ import ( // and /proc/sys is left as it was found. Remounting it read-only on the way // out would break every later write. func TestSetSysctlReportsAnUnrelatedError(t *testing.T) { - err := setSysctl("net/ipv4/ateomnet_no_such_sysctl", "0") + err := setSysctl("net/ipv4/workernet_no_such_sysctl", "0") if !errors.Is(err, unix.ENOENT) { t.Fatalf("setSysctl() on a missing key: got %v, want ENOENT", err) } diff --git a/internal/ateomnet/netns/sysctl_root_linux_test.go b/internal/netns/sysctl_root_linux_test.go similarity index 100% rename from internal/ateomnet/netns/sysctl_root_linux_test.go rename to internal/netns/sysctl_root_linux_test.go diff --git a/internal/nodepath/nodepath.go b/internal/nodepath/nodepath.go index f03b2f7f29..5743f40310 100644 --- a/internal/nodepath/nodepath.go +++ b/internal/nodepath/nodepath.go @@ -13,7 +13,7 @@ // limitations under the License. // Package nodepath holds the node-level paths more than one component needs: -// the host directory atelet and the ateoms both mount, the sockets and netns +// the host directory atelet and the workers both mount, the sockets and netns // they find each other by, the per-actor parent directory, and where atelet // stages runtime binaries. package nodepath @@ -21,7 +21,7 @@ package nodepath import "path/filepath" // BasePath is the root shared folder on the host filesystem, mounted at the -// same path into the atelet and ateom containers. +// same path into the atelet and worker containers. const BasePath = "/var/lib/ate" // ActorsDir is the parent of the per-actor directories atelet prepares. @@ -30,20 +30,20 @@ var ActorsDir = filepath.Join(BasePath, "actors") // StaticFilesDir holds things like runsc binaries. var StaticFilesDir = filepath.Join(BasePath, "static-files") -// AteomSupportSocket is the node-local atelet socket used by atunnel +// WorkerSupportSocket is the node-local atelet socket used by atunnel // to request credentials for the worker's current actor assignment. -var AteomSupportSocket = filepath.Join(BasePath, "ateom-support.sock") +var WorkerSupportSocket = filepath.Join(BasePath, "worker-support.sock") // AteletOTLPSocketPath is the node-scoped unix socket atelet serves the OTLP // relay on (see internal/otlprelay). It is node-scoped rather than per-pod -// because every ateom on the node pushes into the same relay: atelet is a +// because every worker on the node pushes into the same relay: atelet is a // DaemonSet, so one socket collapses N per-pod collector connections into one // per-node connection. // // It sits directly under BasePath, which is the host directory already mounted -// at the same path into atelet and into every ateom pod, so no new volume is -// needed for ateom to reach it. Note that BasePath is mounted writable -// (workerpool_apply.go) and shared with AteomSupportSocket and the image +// at the same path into atelet and into every worker pod, so no new volume is +// needed for worker to reach it. Note that BasePath is mounted writable +// (workerpool_apply.go) and shared with WorkerSupportSocket and the image // cache, so a worker pod can unlink or replace this socket. Confining // atelet-owned sockets to a subdirectory mounted read-only would be an // improvement, but it is a property of the whole BasePath mount rather than of @@ -56,27 +56,27 @@ func AteletOTLPSocketPath() string { ) } -// AteomsDir is the parent of every per-ateom directory, so listing it is how -// a scraper with no prior knowledge discovers the node's ateoms. atelet -// creates AteomPath(podUID) once it writes the pod's sandbox versions file and -// removes it when the pod is deleted; the ateom also MkdirAlls it at boot. A -// directory may exist before its ateom listens on its socket. -func AteomsDir() string { - return filepath.Join(BasePath, "ateoms") +// WorkersDir is the parent of every per-worker directory, so listing it is how +// a scraper with no prior knowledge discovers the node's workers. atelet +// creates WorkerPath(podUID) once it writes the pod's sandbox versions file and +// removes it when the pod is deleted; the worker also MkdirAlls it at boot. A +// directory may exist before its worker listens on its socket. +func WorkersDir() string { + return filepath.Join(BasePath, "workers") } -func AteomPath(podUID string) string { - return filepath.Join(AteomsDir(), podUID) +func WorkerPath(podUID string) string { + return filepath.Join(WorkersDir(), podUID) } -func AteomSocketPath(podUID string) string { +func WorkerSocketPath(podUID string) string { return filepath.Join( - AteomPath(podUID), - "ateom.sock", + WorkerPath(podUID), + "worker.sock", ) } -// SandboxVersionsFileName is the base name of the file in AteomPath(podUID) +// SandboxVersionsFileName is the base name of the file in WorkerPath(podUID) // where atelet keeps the sandbox versions of a worker's pool, a // sandboxversions.File. A runtime it lists may not have its assets on the // node yet. @@ -84,7 +84,7 @@ const SandboxVersionsFileName = "sandbox-versions.json" // ActorNetNSName names an actor's sandbox network namespace. func ActorNetNSName(actorUID string) string { - return "ateom-actor:" + actorUID + return "worker-actor:" + actorUID } // ActorNetNSPath is the mount path of the actor's named namespace. diff --git a/internal/nodepath/nodepath_test.go b/internal/nodepath/nodepath_test.go index caf5073716..28a84cbdc1 100644 --- a/internal/nodepath/nodepath_test.go +++ b/internal/nodepath/nodepath_test.go @@ -19,20 +19,20 @@ import ( "testing" ) -func TestAteomPath(t *testing.T) { +func TestWorkerPath(t *testing.T) { podUID := "123e4567-e89b-12d3-a456-426614174000" - path := AteomPath(podUID) - expectedSuffix := "/ateoms/" + podUID + path := WorkerPath(podUID) + expectedSuffix := "/workers/" + podUID if !strings.HasSuffix(path, expectedSuffix) { t.Errorf("expected path to end with %s, got %s", expectedSuffix, path) } } -func TestAteomSocketPathLimits(t *testing.T) { +func TestWorkerSocketPathLimits(t *testing.T) { podUID := "123e4567-e89b-12d3-a456-426614174000" - sockPath := AteomSocketPath(podUID) + sockPath := WorkerSocketPath(podUID) // Unix domain socket path limit is 107 bytes (108 with NUL terminator) const maxUnixSocketLen = 107 @@ -41,7 +41,7 @@ func TestAteomSocketPathLimits(t *testing.T) { } // Verify it is deterministic - sockPath2 := AteomSocketPath(podUID) + sockPath2 := WorkerSocketPath(podUID) if sockPath != sockPath2 { t.Errorf("expected deterministic socket paths, got %q and %q", sockPath, sockPath2) } @@ -57,24 +57,24 @@ func TestAteletOTLPSocketPath(t *testing.T) { } // It must sit under BasePath: that is the host directory already mounted at - // the same path into atelet and into every ateom pod, which is the whole + // the same path into atelet and into every worker pod, which is the whole // reason the relay needs no new volume. if !strings.HasPrefix(sockPath, BasePath+"/") { - t.Errorf("AteletOTLPSocketPath() = %q, want it under %q so ateom and atelet see the same file", sockPath, BasePath) + t.Errorf("AteletOTLPSocketPath() = %q, want it under %q so worker and atelet see the same file", sockPath, BasePath) } - // Node-scoped, so it must not collide with any per-pod ateom socket. - if other := AteomSocketPath("123e4567-e89b-12d3-a456-426614174000"); sockPath == other { - t.Errorf("AteletOTLPSocketPath() collides with AteomSocketPath: %q", sockPath) + // Node-scoped, so it must not collide with any per-pod worker socket. + if other := WorkerSocketPath("123e4567-e89b-12d3-a456-426614174000"); sockPath == other { + t.Errorf("AteletOTLPSocketPath() collides with WorkerSocketPath: %q", sockPath) } } -func TestAteomPathUniqueness(t *testing.T) { +func TestWorkerPathUniqueness(t *testing.T) { uid1 := "123e4567-e89b-12d3-a456-426614174000" uid2 := "987f6543-e21b-32d1-b654-246614174111" - path1 := AteomPath(uid1) - path2 := AteomPath(uid2) + path1 := WorkerPath(uid1) + path2 := WorkerPath(uid2) if path1 == path2 { t.Errorf("expected different paths for different pod UIDs, got %q", path1) diff --git a/internal/ocispec/gvisor.go b/internal/ocispec/gvisor.go index f73de3b182..135e44ae7e 100644 --- a/internal/ocispec/gvisor.go +++ b/internal/ocispec/gvisor.go @@ -18,7 +18,7 @@ import ( "path/filepath" "slices" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/sizing" "github.com/opencontainers/runtime-spec/specs-go" ) @@ -34,7 +34,7 @@ const resolvConf = "/etc/resolv.conf" // GVisorOptions describes the gVisor-specific context of one actor container. type GVisorOptions struct { ActorUID string - ActorDirs *ateompb.ActorDirs + ActorDirs *ateworkerpb.ActorDirs ContainerName string // DurableVolumes are declared on the sandbox (pause) spec only. DurableVolumes []string diff --git a/internal/ocispec/gvisor_test.go b/internal/ocispec/gvisor_test.go index 50f8ca15b9..84d0ec04e8 100644 --- a/internal/ocispec/gvisor_test.go +++ b/internal/ocispec/gvisor_test.go @@ -63,8 +63,8 @@ func TestShapeGVisorBindsTheNamedResolvConf(t *testing.T) { {name: "default is the worker pod's", wantSource: "/etc/resolv.conf"}, { name: "a sandbox may name its own", - resolvConf: "/var/lib/ateom-gvisor/actors/a/resolv.conf", - wantSource: "/var/lib/ateom-gvisor/actors/a/resolv.conf", + resolvConf: "/var/lib/ateworker-gvisor/actors/a/resolv.conf", + wantSource: "/var/lib/ateworker-gvisor/actors/a/resolv.conf", }, } { t.Run(tc.name, func(t *testing.T) { diff --git a/internal/ocispec/microvm.go b/internal/ocispec/microvm.go index fd01412b90..33b307020c 100644 --- a/internal/ocispec/microvm.go +++ b/internal/ocispec/microvm.go @@ -22,7 +22,7 @@ import ( "strings" "github.com/agent-substrate/substrate/internal/imagecache" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/opencontainers/runtime-spec/specs-go" ) @@ -43,7 +43,7 @@ const ( type MicroVMOptions struct { // ActorDirs are the actor's directories; volume bind sources under them are // staged into the share. - ActorDirs *ateompb.ActorDirs + ActorDirs *ateworkerpb.ActorDirs ContainerID string } @@ -112,7 +112,7 @@ func mergeKataResources(from *specs.LinuxResources) *specs.LinuxResources { } // guestVolumeSource maps a volume's host directory to its guest path. -func guestVolumeSource(hostPath string, actorDirs *ateompb.ActorDirs, containerID string) (string, error) { +func guestVolumeSource(hostPath string, actorDirs *ateworkerpb.ActorDirs, containerID string) (string, error) { for _, staged := range []struct{ host, guest string }{ {actorDirs.GetDurableDirVolumeMountsDir(), path.Join(GuestSharedDir, ShareDurable)}, {actorDirs.GetVolumesDir(), path.Join(GuestSharedDir, ShareCSI)}, diff --git a/internal/ocispec/microvm_test.go b/internal/ocispec/microvm_test.go index bfde5f6595..c791bbf25f 100644 --- a/internal/ocispec/microvm_test.go +++ b/internal/ocispec/microvm_test.go @@ -17,7 +17,7 @@ package ocispec import ( "testing" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/opencontainers/runtime-spec/specs-go" ) @@ -121,9 +121,9 @@ func eqInt64Ptr(a, b *int64) bool { // unbound the container. func TestShapeMicroVM_KeepsDeclaredContainerLimits(t *testing.T) { const declared = 64 * 1024 * 1024 - spec := Build(Options{Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{ + spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{ Args: []string{"/app"}, - Resources: &ateompb.ResourceLimits{MemoryBytes: declared}, + Resources: &ateworkerpb.ResourceLimits{MemoryBytes: declared}, }}}) if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerID: "app"}); err != nil { t.Fatalf("ShapeMicroVM() = %v", err) @@ -140,7 +140,7 @@ func TestShapeMicroVM_KeepsDeclaredContainerLimits(t *testing.T) { // A container that declares nothing must stay unbounded inside the guest: guest // RAM is the real ceiling, and a cap equal to the whole guest can never bind. func TestShapeMicroVM_LeavesUndeclaredContainerUnlimited(t *testing.T) { - spec := Build(Options{Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app"}}}}) + spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}}}) if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerID: "app"}); err != nil { t.Fatalf("ShapeMicroVM() = %v", err) } diff --git a/internal/ocispec/ocispec.go b/internal/ocispec/ocispec.go index dbaec1cf85..7c40caa656 100644 --- a/internal/ocispec/ocispec.go +++ b/internal/ocispec/ocispec.go @@ -13,7 +13,7 @@ // limitations under the License. // Package ocispec builds the runtime-neutral OCI spec for actor bundles, which -// each ateom shapes for its runtime. +// each worker shapes for its runtime. package ocispec import ( @@ -25,7 +25,7 @@ import ( "github.com/agent-substrate/substrate/internal/imagecache" "github.com/agent-substrate/substrate/internal/nodepath" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/opencontainers/runtime-spec/specs-go" ) @@ -41,9 +41,9 @@ type Options struct { ActorUID string // ActorDirs are the actor's directories the bind mount sources are joined // from, one subdirectory per volume name. - ActorDirs *ateompb.ActorDirs + ActorDirs *ateworkerpb.ActorDirs // Container carries the process and its volume mounts. - Container *ateompb.Container + Container *ateworkerpb.Container } const ( @@ -60,7 +60,7 @@ const ( // ociResources maps the resolved limits onto the OCI spec's linux.resources. // Returns nil when nothing is set, so the spec carries no linux.resources at // all for a container that declares no limits. -func ociResources(r *ateompb.ResourceLimits) *specs.LinuxResources { +func ociResources(r *ateworkerpb.ResourceLimits) *specs.LinuxResources { if r == nil || (r.GetMemoryBytes() <= 0 && r.GetCpuMillis() <= 0) { return nil } diff --git a/internal/ocispec/ocispec_test.go b/internal/ocispec/ocispec_test.go index 2d06ed5c3d..4fec74c022 100644 --- a/internal/ocispec/ocispec_test.go +++ b/internal/ocispec/ocispec_test.go @@ -23,7 +23,7 @@ import ( "github.com/agent-substrate/substrate/internal/imagecache" "github.com/agent-substrate/substrate/internal/nodepath" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/opencontainers/runtime-spec/specs-go" ) @@ -48,22 +48,22 @@ func TestBuild_VolumeMounts(t *testing.T) { bundle = "/node/actors/a/bundles/app" ) spec := Build(Options{ - ActorDirs: &ateompb.ActorDirs{ + ActorDirs: &ateworkerpb.ActorDirs{ OciBundleDir: "/node/actors/a/bundles", DurableDirVolumeMountsDir: durableDir, VolumesDir: volumesDir, SystemInfoVolumeRootsDir: sysInfoDir, }, - Container: &ateompb.Container{ + Container: &ateworkerpb.Container{ Name: "app", - ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app"}}, - DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{ + ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}, + DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{ {VolumeName: "data", MountPath: "/var/data"}, {VolumeName: "data", MountPath: "/home/counter"}, }, - SystemInfoVolumeMounts: []*ateompb.SystemInfoVolumeMount{{VolumeName: "sysinfo", MountPath: "/run/ate"}}, - CsiVolumeMounts: []*ateompb.VolumeMount{{VolumeName: "csi", MountPath: "/mnt/csi"}}, - ImageVolumeMounts: []*ateompb.ImageVolumeMount{{VolumeName: "agent", MountPath: "/ate"}}, + SystemInfoVolumeMounts: []*ateworkerpb.SystemInfoVolumeMount{{VolumeName: "sysinfo", MountPath: "/run/ate"}}, + CsiVolumeMounts: []*ateworkerpb.VolumeMount{{VolumeName: "csi", MountPath: "/mnt/csi"}}, + ImageVolumeMounts: []*ateworkerpb.ImageVolumeMount{{VolumeName: "agent", MountPath: "/ate"}}, }, }) @@ -91,9 +91,9 @@ func TestBuild_VolumeMounts(t *testing.T) { } } -// The container joins the network namespace ateom creates for the actor. +// The container joins the network namespace worker creates for the actor. func TestBuild_JoinsTheActorNetNS(t *testing.T) { - spec := Build(Options{ActorUID: "uid-a", Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app"}}}}) + spec := Build(Options{ActorUID: "uid-a", Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}}}) i := slices.IndexFunc(spec.Linux.Namespaces, func(ns specs.LinuxNamespace) bool { return ns.Type == specs.NetworkNamespace }) if i < 0 { t.Fatal("spec has no network namespace") @@ -106,7 +106,7 @@ func TestBuild_JoinsTheActorNetNS(t *testing.T) { // The resolved set lands in bounding, effective and permitted only. func TestBuild_Capabilities(t *testing.T) { want := []string{"CAP_CHOWN", "CAP_KILL"} - spec := Build(Options{Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app"}, Capabilities: want}}}) + spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}, Capabilities: want}}}) caps := spec.Process.Capabilities if caps == nil { @@ -139,7 +139,7 @@ func TestBuild_Capabilities(t *testing.T) { // The pause container gets no capabilities. func TestBuild_NoCapabilitiesForPause(t *testing.T) { - spec := Build(Options{Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/pause"}}}}) + spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/pause"}}}}) caps := spec.Process.Capabilities if caps == nil { @@ -163,7 +163,7 @@ func TestBuild_NoCapabilitiesForPause(t *testing.T) { func TestSave(t *testing.T) { bundle := t.TempDir() - want := Build(Options{ActorUID: "uid-a", Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app"}}}}) + want := Build(Options{ActorUID: "uid-a", Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}}}) if err := Save(bundle, want); err != nil { t.Fatalf("Save() = %v", err) } @@ -184,11 +184,11 @@ func TestOCIResources(t *testing.T) { if got := ociResources(nil); got != nil { t.Errorf("ociResources(nil) = %v, want nil", got) } - if got := ociResources(&ateompb.ResourceLimits{}); got != nil { + if got := ociResources(&ateworkerpb.ResourceLimits{}); got != nil { t.Errorf("ociResources(zero) = %v, want nil so the spec is unchanged", got) } - got := ociResources(&ateompb.ResourceLimits{MemoryBytes: 268435456, CpuMillis: 200}) + got := ociResources(&ateworkerpb.ResourceLimits{MemoryBytes: 268435456, CpuMillis: 200}) if got == nil { t.Fatal("ociResources() = nil, want limits") } @@ -215,7 +215,7 @@ func TestOCIResources(t *testing.T) { // EINVAL at container create. func TestOCIResources_ClampsQuotaToKernelMinimum(t *testing.T) { for _, millis := range []int64{1, 5, 9} { - got := ociResources(&ateompb.ResourceLimits{CpuMillis: millis}) + got := ociResources(&ateworkerpb.ResourceLimits{CpuMillis: millis}) if got == nil || got.CPU == nil || got.CPU.Quota == nil { t.Fatalf("cpu=%dm: ociResources() = %v, want a quota", millis, got) } @@ -225,7 +225,7 @@ func TestOCIResources_ClampsQuotaToKernelMinimum(t *testing.T) { } } // At the floor the quota is exact, not clamped. - got := ociResources(&ateompb.ResourceLimits{CpuMillis: 10}) + got := ociResources(&ateworkerpb.ResourceLimits{CpuMillis: 10}) if *got.CPU.Quota != cpuQuotaMinUS { t.Errorf("cpu=10m: quota = %d, want exactly %d", *got.CPU.Quota, cpuQuotaMinUS) } @@ -234,23 +234,23 @@ func TestOCIResources_ClampsQuotaToKernelMinimum(t *testing.T) { // A negative limit must not produce a non-nil but empty LinuxResources, which // would put a bare "resources": {} into the spec. func TestOCIResources_NegativeIsUnset(t *testing.T) { - if got := ociResources(&ateompb.ResourceLimits{MemoryBytes: -1, CpuMillis: -1}); got != nil { + if got := ociResources(&ateworkerpb.ResourceLimits{MemoryBytes: -1, CpuMillis: -1}); got != nil { t.Errorf("ociResources(negative) = %+v, want nil", got) } } // A container without limits carries no linux.resources at all. func TestBuild_NoResourcesLeavesLinuxUntouched(t *testing.T) { - spec := Build(Options{Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app"}}}}) + spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}}}) if spec.Linux.Resources != nil { t.Errorf("Linux.Resources = %v, want nil when no limits are declared", spec.Linux.Resources) } } func TestBuild_ResourcesApplied(t *testing.T) { - spec := Build(Options{Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{ + spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{ Args: []string{"/app"}, - Resources: &ateompb.ResourceLimits{MemoryBytes: 67108864}, + Resources: &ateworkerpb.ResourceLimits{MemoryBytes: 67108864}, }}}) if spec.Linux.Resources == nil || spec.Linux.Resources.Memory == nil { t.Fatalf("Linux.Resources = %v, want a memory limit", spec.Linux.Resources) diff --git a/internal/ocispec/parity_test.go b/internal/ocispec/parity_test.go index 293c26a35b..809c67ea03 100644 --- a/internal/ocispec/parity_test.go +++ b/internal/ocispec/parity_test.go @@ -20,14 +20,14 @@ import ( "strings" "testing" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/sizing" "github.com/opencontainers/runtime-spec/specs-go" ) const testActorUID = "actor_uid" -var parityActorDirs = &ateompb.ActorDirs{ +var parityActorDirs = &ateworkerpb.ActorDirs{ RootDir: "/node/actors/a", OciBundleDir: "/node/actors/a/bundles", DurableDirVolumeMountsDir: "/node/actors/a/durable-dir", @@ -39,13 +39,13 @@ var parityActorDirs = &ateompb.ActorDirs{ var parityOptions = Options{ ActorUID: testActorUID, ActorDirs: parityActorDirs, - Container: &ateompb.Container{ + Container: &ateworkerpb.Container{ Name: "app", - ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app"}}, - DurableDirVolumeMounts: []*ateompb.DurableDirVolumeMount{{VolumeName: "data", MountPath: "/var/data"}}, - SystemInfoVolumeMounts: []*ateompb.SystemInfoVolumeMount{{VolumeName: "sysinfo", MountPath: "/run/ate"}}, - CsiVolumeMounts: []*ateompb.VolumeMount{{VolumeName: "csi", MountPath: "/mnt/csi"}}, - ImageVolumeMounts: []*ateompb.ImageVolumeMount{{VolumeName: "agent", MountPath: "/ate"}}, + ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}, + DurableDirVolumeMounts: []*ateworkerpb.DurableDirVolumeMount{{VolumeName: "data", MountPath: "/var/data"}}, + SystemInfoVolumeMounts: []*ateworkerpb.SystemInfoVolumeMount{{VolumeName: "sysinfo", MountPath: "/run/ate"}}, + CsiVolumeMounts: []*ateworkerpb.VolumeMount{{VolumeName: "csi", MountPath: "/mnt/csi"}}, + ImageVolumeMounts: []*ateworkerpb.ImageVolumeMount{{VolumeName: "agent", MountPath: "/ate"}}, }, } @@ -131,7 +131,7 @@ func TestShapeMicroVM_TranslatesSourcesIntoTheShare(t *testing.T) { // ShapeMicroVM errors on a bind that is not staged into the share. func TestShapeMicroVM_UnstagedSourceIsAnError(t *testing.T) { - spec := Build(Options{Container: &ateompb.Container{ContainerSpec: &ateompb.ContainerSpec{Args: []string{"/app"}}}}) + spec := Build(Options{Container: &ateworkerpb.Container{ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}}}}) spec.Mounts = append(spec.Mounts, specs.Mount{Destination: "/mnt/new", Type: "bind", Source: "/var/lib/ate/new-kind/x"}) if err := ShapeMicroVM(spec, MicroVMOptions{ActorDirs: parityActorDirs, ContainerID: "app"}); err == nil { t.Fatal("ShapeMicroVM() = nil, want an error for a bind that is not staged into the share") diff --git a/internal/oidcdiscovery/discovery.go b/internal/oidcdiscovery/discovery.go index 27e9a2b175..4b0898afb8 100644 --- a/internal/oidcdiscovery/discovery.go +++ b/internal/oidcdiscovery/discovery.go @@ -58,6 +58,6 @@ func DiscoveryDocument(issuer string, algorithms []string) ([]byte, error) { ResponseTypesSupported: []string{"id_token"}, SubjectTypesSupported: []string{"public"}, IDTokenSigningAlgValuesSupported: algs, - ClaimsSupported: []string{"iss", "sub", "aud", "exp", "nbf", "iat", "jti", "actor.ate.dev", "ateom-for-actor.ate.dev"}, + ClaimsSupported: []string{"iss", "sub", "aud", "exp", "nbf", "iat", "jti", "actor.ate.dev", "worker-for-actor.ate.dev"}, }) } diff --git a/internal/oidcdiscovery/discovery_test.go b/internal/oidcdiscovery/discovery_test.go index e7c54d70d9..f308c5038f 100644 --- a/internal/oidcdiscovery/discovery_test.go +++ b/internal/oidcdiscovery/discovery_test.go @@ -32,7 +32,7 @@ func TestDiscoveryDocument(t *testing.T) { `"response_types_supported":["id_token"],` + `"subject_types_supported":["public"],` + `"id_token_signing_alg_values_supported":["RS256"],` + - `"claims_supported":["iss","sub","aud","exp","nbf","iat","jti","actor.ate.dev","ateom-for-actor.ate.dev"]}` + `"claims_supported":["iss","sub","aud","exp","nbf","iat","jti","actor.ate.dev","worker-for-actor.ate.dev"]}` if diff := cmp.Diff(want, string(got)); diff != "" { t.Errorf("DiscoveryDocument mismatch (-want +got):\n%s", diff) } diff --git a/internal/otlprelay/client.go b/internal/otlprelay/client.go index fef9906669..55b0466e8a 100644 --- a/internal/otlprelay/client.go +++ b/internal/otlprelay/client.go @@ -26,14 +26,14 @@ import ( "google.golang.org/grpc/credentials/insecure" ) -// Dial opens the ateom half of the relay: a gRPC connection over atelet's unix +// Dial opens the worker half of the relay: a gRPC connection over atelet's unix // socket, to be handed to the OTLP exporters via serverboot's ExporterConn. // // It returns (nil, nil) when sockPath is empty or absent, which the caller reads // as "export directly instead". The existence check is what makes the fallback // deterministic at startup: grpc.NewClient is lazy, so a connection to a missing // socket would be created happily and only fail later, per export, with the -// telemetry already lost. Losing spans is not worth failing ateom over either, +// telemetry already lost. Losing spans is not worth failing worker over either, // hence a fallback rather than an error. // // The connection is plaintext by design. A unix socket cannot leave the node, so diff --git a/internal/otlprelay/e2e_test.go b/internal/otlprelay/e2e_test.go index 81cd8a995d..b355b6c1be 100644 --- a/internal/otlprelay/e2e_test.go +++ b/internal/otlprelay/e2e_test.go @@ -23,7 +23,7 @@ import ( "github.com/agent-substrate/substrate/internal/serverboot" ) -// TestEndToEndThroughServerboot exercises the whole path an ateom span actually +// TestEndToEndThroughServerboot exercises the whole path a worker span actually // takes, rather than the relay in isolation: // // serverboot.InitTracing → OTLP exporter → unix socket → relay → collector @@ -32,7 +32,7 @@ import ( // still pass if TracingOptions.ExporterConn were wired up wrong and the exporter // quietly kept dialing OTEL_EXPORTER_OTLP_ENDPOINT. This one would not: the // endpoint variable points at the fake collector *through* the relay only, and -// the assertion is that the span arrived carrying ateom's own service.name. +// the assertion is that the span arrived carrying worker's own service.name. // // Run it on its own to watch the hop happen: // @@ -55,7 +55,7 @@ func TestEndToEndThroughServerboot(t *testing.T) { } defer conn.Close() - const serviceName = "ateom-microvm" + const serviceName = "ateworker-microvm" tp, err := serverboot.InitTracing(context.Background(), serverboot.TracingOptions{ ServiceName: serviceName, // Ratio 1.0: this test asserts on delivery, not on sampling. @@ -108,7 +108,7 @@ func TestEndToEndThroughServerboot(t *testing.T) { } t.Logf("collector received span %q from service %q (relay=%q)", gotSpan, gotService, gotRelay) - // The point of forwarding the request verbatim: the span is still ateom's, + // The point of forwarding the request verbatim: the span is still worker's, // not atelet's. if gotService != serviceName { t.Errorf("span arrived with service.name %q, want %q; the relay must not re-attribute it", gotService, serviceName) @@ -127,11 +127,11 @@ func TestEndToEndThroughServerboot(t *testing.T) { const relayAttrKey = "ate.otlp.relay" // TestEndToEndFallsBackToDirect is the other half of TestEndToEndThroughServerboot: -// the ateom asked for the relay, atelet was not serving one, and the exporter +// the worker asked for the relay, atelet was not serving one, and the exporter // must fall back to the network path rather than dropping telemetry. // -// This is the case the ateoms degrade into instead of exiting (see the Dial call -// in cmd/ateom-*/main.go), so it needs to be more than a nil check: the span has +// This is the case the workers degrade into instead of exiting (see the Dial call +// in cmd/ateworker-*/main.go), so it needs to be more than a nil check: the span has // to reach the collector, and it has to be distinguishable from a relayed one at // query time — hence the "direct" attribute. func TestEndToEndFallsBackToDirect(t *testing.T) { @@ -152,7 +152,7 @@ func TestEndToEndFallsBackToDirect(t *testing.T) { t.Fatal("Dial returned a connection for a socket that does not exist") } - const serviceName = "ateom-microvm" + const serviceName = "ateworker-microvm" tp, err := serverboot.InitTracing(context.Background(), serverboot.TracingOptions{ ServiceName: serviceName, Sampling: serverboot.ParentRatioSampling(1.0), diff --git a/internal/otlprelay/relay.go b/internal/otlprelay/relay.go index f5a3cbaedf..327605d5b5 100644 --- a/internal/otlprelay/relay.go +++ b/internal/otlprelay/relay.go @@ -12,11 +12,11 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package otlprelay carries ateom's OTLP telemetry to the collector over a unix +// Package otlprelay carries worker's OTLP telemetry to the collector over a unix // socket served by atelet, so a worker pod needs no network path of its own to // export spans, metrics, and log records. // -// Motivation. ateom runs inside the worker pod that hosts the actor, and until +// Motivation. worker runs inside the worker pod that hosts the actor, and until // now exported OTLP straight to the collector over the pod's network (the // endpoint is injected by atecontroller, see workerpool_apply.go). That has four // costs the relay removes: @@ -26,9 +26,9 @@ // reachable to anything that escapes the sandbox. A unix socket cannot leave // the node, so the pod can be denied network egress entirely. // - Connection count. Worker pods are heavily oversubscribed, so a node runs -// many ateoms, each holding its own gRPC connection to the collector. They +// many workers, each holding its own gRPC connection to the collector. They // collapse into atelet's single per-node connection. -// - Interference. ateom installs a transparent redirect of actor egress to its +// - Interference. worker installs a transparent redirect of actor egress to its // own atunnel listener; its own outbound traffic has to stay clear of the // rules it installs. A unix socket is not IP traffic and cannot be caught. // - Shutdown loss. Teardown frees the actor's network and then the pod goes @@ -36,12 +36,12 @@ // in the batch processor. atelet outlives the worker pod. // // The relay forwards the OTLP request message verbatim rather than decoding it -// into SDK records and re-exporting. Verbatim pass-through keeps each ateom's +// into SDK records and re-exporting. Verbatim pass-through keeps each worker's // own resource (service.name, service.instance.id, pod attributes) intact, so -// its spans stay attributed to ateom instead of being absorbed into atelet's. -// Restricting this pass-through to verified ateom sources ensures that future +// its spans stay attributed to worker instead of being absorbed into atelet's. +// Restricting this pass-through to verified worker sources ensures that future // actor telemetry requiring identity rewrites (#761) will be added as an -// explicit rewriting path alongside this forwarder; see ateomServices. +// explicit rewriting path alongside this forwarder; see workerServices. // // Verbatim applies to the payload, not to the call around it. The request's // metadata is dropped and replaced with the headers atelet resolves from its own @@ -106,11 +106,11 @@ const ( otlpDefaultPort = "4317" // socketMode keeps the relay socket private to root: both atelet and the - // ateom worker pods run as root (runAsUser: 0). The socket lives inside + // worker pods run as root (runAsUser: 0). The socket lives inside // BasePath, a root-owned host directory. socketMode = 0o600 - // maxRecvMsgSize bounds a single Export payload. One misbehaving ateom + // maxRecvMsgSize bounds a single Export payload. One misbehaving worker // should not be able to make atelet allocate without limit; the OTel SDK's // batch processor emits far smaller messages than this. maxRecvMsgSize = 16 << 20 // 16 MiB @@ -124,35 +124,35 @@ type Server struct { sockPath string } -// ateomServices are the only sources this relay carries, keyed by the +// workerServices are the only sources this relay carries, keyed by the // service.name their resource declares — which the OTEL_* environment can -// override out from under an ateom; see sourceGate for what that looks like. +// override out from under a worker; see sourceGate for what that looks like. // Mirrors the serviceName constants in -// cmd/ateom-gvisor and cmd/ateom-microvm, which are package main and cannot be -// imported; TestAteomServicesMatchTheAteomBinaries guards the duplication. +// cmd/ateworker-gvisor and cmd/ateworker-microvm, which are package main and cannot be +// imported; TestWorkerServicesMatchTheWorkerBinaries guards the duplication. // // This allowlist is a protocol contract rather than a security boundary: // service.name is client-provided, so a compromised process could claim an -// ateom name. Its purpose is to prevent accidental misuse (e.g. an actor SDK +// worker name. Its purpose is to prevent accidental misuse (e.g. an actor SDK // pointed at the socket) and keep the pass-through contract explicit for #761. // Peer authentication, if needed, would require per-pod sockets or UDS peer // credentials (SO_PEERCRED) tied to #741. -var ateomServices = map[string]bool{ - "ateom-gvisor": true, - "ateom-microvm": true, +var workerServices = map[string]bool{ + "ateworker-gvisor": true, + "ateworker-microvm": true, } -// sourceGate applies the ateomServices allowlist and reports the first +// sourceGate applies the workerServices allowlist and reports the first // rejection of each service.name. // // The log matters because of how this failure presents. service.name is // whatever the resource declares, and resource.WithFromEnv() runs last in // serverboot.newResource, so OTEL_SERVICE_NAME or an OTEL_RESOURCE_ATTRIBUTES -// entry set on a worker pod overrides the ateom's own name. The relay then +// entry set on a worker pod overrides the worker's own name. The relay then // refuses every export with PermissionDenied, which the OTel SDK does not retry // — it drops the batch and reports through the SDK error handler. Telemetry from -// that ateom simply stops, with nothing on the collector side to say why. One -// line per distinct name on the node makes it greppable; the ateom itself is +// that worker simply stops, with nothing on the collector side to say why. One +// line per distinct name on the node makes it greppable; the worker itself is // unaffected, so this is a diagnosability problem rather than an outage. // // Dedup is keyed by name with an LRU cache bounded at 256 entries to prevent @@ -173,7 +173,7 @@ func newSourceGate() *sourceGate { // unidentified source is the one the relay cannot vouch for. func (g *sourceGate) check(ctx context.Context, r *resourcepb.Resource) error { name := resourceServiceName(r) - if ateomServices[name] { + if workerServices[name] { return nil } if g.logged == nil { @@ -181,19 +181,19 @@ func (g *sourceGate) check(ctx context.Context, r *resourcepb.Resource) error { } if _, seen := g.logged.Get(name); !seen { g.logged.Add(name, struct{}{}) - slog.WarnContext(ctx, "OTLP relay rejected telemetry from an unrecognized source; it is being dropped, not retried. If this is an ateom, check whether OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES on the worker pod is overriding its service.name", + slog.WarnContext(ctx, "OTLP relay rejected telemetry from an unrecognized source; it is being dropped, not retried. If this is a worker, check whether OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES on the worker pod is overriding its service.name", slog.String("service.name", name), slog.Any("allowed", allowedServices()), slog.String("note", "logged once per distinct service.name")) } return status.Errorf(codes.PermissionDenied, - "the OTLP relay carries ateom telemetry only, got service.name %q; a source whose identity has to be rewritten (#761) must not be forwarded verbatim. If this is an ateom, an OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES override on the worker pod would produce exactly this", + "the OTLP relay carries worker telemetry only, got service.name %q; a source whose identity has to be rewritten (#761) must not be forwarded verbatim. If this is a worker, an OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES override on the worker pod would produce exactly this", name) } func allowedServices() []string { - names := make([]string, 0, len(ateomServices)) - for name := range ateomServices { + names := make([]string, 0, len(workerServices)) + for name := range workerServices { names = append(names, name) } sort.Strings(names) @@ -210,7 +210,7 @@ func resourceServiceName(r *resourcepb.Resource) string { } // upstreamContext builds the metadata for the upstream call from atelet's own -// configuration, dropping whatever the ateom sent. +// configuration, dropping whatever the worker sent. // // The upstream leg is atelet's connection to the collector, so its credentials // belong to atelet: the relay resolves OTEL_EXPORTER_OTLP_HEADERS from its own @@ -218,11 +218,11 @@ func resourceServiceName(r *resourcepb.Resource) string { // the client's headers instead would let anything that reached the socket choose // what atelet presents to the collector — a header set is not telemetry to be // passed through verbatim the way the resource is, and unlike service.name (see -// ateomServices) it is not merely claimed identity but an actual credential. +// workerServices) it is not merely claimed identity but an actual credential. // // Nothing is allow-listed through. atecontroller injects only // OTEL_EXPORTER_OTLP_ENDPOINT into worker pods (workerpool_apply.go), so no -// ateom has a header to lose today; add an allowlist here, not a blanket +// worker has a header to lose today; add an allowlist here, not a blanket // forward, if one ever needs to reach the collector. // // The incoming metadata is dropped by simply not copying it: gRPC never @@ -301,7 +301,7 @@ type traceRelay struct { // headers atelet presents to the collector; see upstreamContext. Resolved // once at construction: they come from atelet's environment, not the call. headers metadata.MD - // gate is shared with metricRelay and logRelay so a misnamed ateom is + // gate is shared with metricRelay and logRelay so a misnamed worker is // reported once, not once per signal. gate *sourceGate } @@ -362,7 +362,7 @@ func (l *logRelay) Export(ctx context.Context, req *collogspb.ExportLogsServiceR // afterwards, which is why this errors rather than falling back. func validateSocketPath(sockPath string) error { if !filepath.IsAbs(sockPath) { - return fmt.Errorf("the OTLP relay socket path %q is relative; it must be absolute, since atelet and ateom would otherwise resolve it against different working directories", sockPath) + return fmt.Errorf("the OTLP relay socket path %q is relative; it must be absolute, since atelet and the worker would otherwise resolve it against different working directories", sockPath) } return nil } @@ -370,8 +370,8 @@ func validateSocketPath(sockPath string) error { // NewServer builds a relay that forwards to the collector named by the standard // OTLP endpoint environment variables. It returns (nil, nil) when sockPath is // empty (the relay is switched off) or when no endpoint is configured: a relay -// with nowhere to forward to would accept an ateom's spans and drop them, which -// is worse than ateom finding no socket and falling back to a direct export. +// with nowhere to forward to would accept a worker's spans and drop them, which +// is worse than the worker finding no socket and falling back to a direct export. func NewServer(ctx context.Context, sockPath string) (*Server, error) { if sockPath == "" { return nil, nil @@ -397,7 +397,7 @@ func NewServer(ctx context.Context, sockPath string) (*Server, error) { // Resolved before the socket exists: a header set atelet cannot parse would // otherwise become a per-export failure against a collector that rejects the // unauthenticated calls, which is harder to read than refusing to start the - // relay. ateom then finds no socket and exports directly. + // relay. worker then finds no socket and exports directly. traceHeaders, err := upstreamHeaders(tracesHeadersEnv) if err != nil { return nil, err @@ -475,7 +475,7 @@ func (s *Server) Serve(ctx context.Context) error { return fmt.Errorf("while opening the OTLP relay socket %q: %w", s.sockPath, err) } // net.Listen applies the umask, which on atelet would typically leave the - // socket group/other-unwritable and unreachable from an ateom running as a + // socket group/other-unwritable and unreachable from a worker running as a // different uid. Widen it explicitly. if err := os.Chmod(s.sockPath, socketMode); err != nil { _ = lis.Close() diff --git a/internal/otlprelay/relay_test.go b/internal/otlprelay/relay_test.go index ce4170b10c..8fd5b9151b 100644 --- a/internal/otlprelay/relay_test.go +++ b/internal/otlprelay/relay_test.go @@ -189,8 +189,8 @@ func serviceResource(name string) *resourcepb.Resource { } // TestRelayForwardsTracesVerbatim is the property the whole design rests on: -// what an ateom exports is what the collector sees, including the resource -// attributes that attribute the spans to that ateom rather than to atelet. +// what a worker exports is what the collector sees, including the resource +// attributes that attribute the spans to that worker rather than to atelet. func TestRelayForwardsTracesVerbatim(t *testing.T) { sink, collector := startFakeCollector(t) sock := startRelay(t, collector) @@ -206,7 +206,7 @@ func TestRelayForwardsTracesVerbatim(t *testing.T) { req := &coltracepb.ExportTraceServiceRequest{ ResourceSpans: []*tracepb.ResourceSpans{{ - Resource: serviceResource("ateom-microvm"), + Resource: serviceResource("ateworker-microvm"), ScopeSpans: []*tracepb.ScopeSpans{{ Spans: []*tracepb.Span{{ Name: "RunWorkload", @@ -248,9 +248,9 @@ func TestRelayForwardsMetrics(t *testing.T) { req := &colmetricspb.ExportMetricsServiceRequest{ ResourceMetrics: []*metricspb.ResourceMetrics{{ - Resource: serviceResource("ateom-microvm"), + Resource: serviceResource("ateworker-microvm"), ScopeMetrics: []*metricspb.ScopeMetrics{{ - Metrics: []*metricspb.Metric{{Name: "ateom.workload.runs"}}, + Metrics: []*metricspb.Metric{{Name: "worker.workload.runs"}}, }}, }}, } @@ -274,11 +274,11 @@ func TestRelayForwardsMetrics(t *testing.T) { } } -// usageLogs is an ateom log batch. +// usageLogs is a worker log batch. func usageLogs() *collogspb.ExportLogsServiceRequest { return &collogspb.ExportLogsServiceRequest{ ResourceLogs: []*logspb.ResourceLogs{{ - Resource: serviceResource("ateom-gvisor"), + Resource: serviceResource("ateworker-gvisor"), ScopeLogs: []*logspb.ScopeLogs{{ LogRecords: []*logspb.LogRecord{{EventName: "ate.actor.usage_sampled"}}, }}, @@ -319,7 +319,7 @@ func TestRelayForwardsLogsVerbatim(t *testing.T) { // TestStopRemovesSocket matters for the restart path: a leftover socket makes // the next atelet's Listen fail with EADDRINUSE, and in the meantime makes -// every ateom on the node believe a relay is there. +// every worker on the node believe a relay is there. func TestStopRemovesSocket(t *testing.T) { _, collector := startFakeCollector(t) t.Setenv(endpointEnv, collector) @@ -425,8 +425,8 @@ func TestSourceGateLogsEachRejectionOnce(t *testing.T) { if err := gate.check(context.Background(), serviceResource("atelet")); err == nil { t.Fatal("gate.check accepted atelet") } - if err := gate.check(context.Background(), serviceResource("ateom-gvisor")); err != nil { - t.Fatalf("gate.check rejected an ateom: %v", err) + if err := gate.check(context.Background(), serviceResource("ateworker-gvisor")); err != nil { + t.Fatalf("gate.check rejected a worker: %v", err) } if got := strings.Count(buf.String(), "rejected telemetry"); got != 2 { @@ -438,10 +438,10 @@ func TestSourceGateLogsEachRejectionOnce(t *testing.T) { } } -// TestRelayRefusesNonAteomSource is the scoping contract. The empty +// TestRelayRefusesNonWorkerSource is the scoping contract. The empty // service.name case is the one worth keeping: that is the shape telemetry takes // when identity has not been injected, which is the actor situation in #761. -func TestRelayRefusesNonAteomSource(t *testing.T) { +func TestRelayRefusesNonWorkerSource(t *testing.T) { sink, collector := startFakeCollector(t) sock := startRelay(t, collector) @@ -504,7 +504,7 @@ func TestRelayRefusesMixedBatch(t *testing.T) { _, err = coltracepb.NewTraceServiceClient(conn).Export(context.Background(), &coltracepb.ExportTraceServiceRequest{ ResourceSpans: []*tracepb.ResourceSpans{ - {Resource: serviceResource("ateom-gvisor")}, + {Resource: serviceResource("ateworker-gvisor")}, {Resource: serviceResource("actor")}, }, }) @@ -514,7 +514,7 @@ func TestRelayRefusesMixedBatch(t *testing.T) { _, err = collogspb.NewLogsServiceClient(conn).Export(context.Background(), &collogspb.ExportLogsServiceRequest{ ResourceLogs: []*logspb.ResourceLogs{ - {Resource: serviceResource("ateom-gvisor")}, + {Resource: serviceResource("ateworker-gvisor")}, {Resource: serviceResource("actor")}, }, }) @@ -529,9 +529,9 @@ func TestRelayRefusesMixedBatch(t *testing.T) { } } -// TestRelayAcceptsEveryAteomService guards against the allowlist drifting from +// TestRelayAcceptsEveryWorkerService guards against the allowlist drifting from // the binaries in a way that silently drops all of one runtime's telemetry. -func TestRelayAcceptsEveryAteomService(t *testing.T) { +func TestRelayAcceptsEveryWorkerService(t *testing.T) { sink, collector := startFakeCollector(t) sock := startRelay(t, collector) @@ -541,7 +541,7 @@ func TestRelayAcceptsEveryAteomService(t *testing.T) { } defer conn.Close() - for service := range ateomServices { + for service := range workerServices { if _, err := coltracepb.NewTraceServiceClient(conn).Export(context.Background(), &coltracepb.ExportTraceServiceRequest{ ResourceSpans: []*tracepb.ResourceSpans{{Resource: serviceResource(service)}}, }); err != nil { @@ -568,33 +568,33 @@ func TestRelayAcceptsEveryAteomService(t *testing.T) { } } -// TestAteomServicesMatchTheAteomBinaries keeps the allowlist honest. A typo in -// it would otherwise be invisible: every real ateom export would be refused +// TestWorkerServicesMatchTheWorkerBinaries keeps the allowlist honest. A typo in +// it would otherwise be invisible: every real worker export would be refused // while every test here still passed, because they would share the typo. -func TestAteomServicesMatchTheAteomBinaries(t *testing.T) { - // Matches `const serviceName = "..."` in each ateom main package. +func TestWorkerServicesMatchTheWorkerBinaries(t *testing.T) { + // Matches `const serviceName = "..."` in each worker main package. decl := regexp.MustCompile(`(?m)^\s*const\s+serviceName\s*=\s*"([^"]+)"`) found := map[string]bool{} - for _, main := range []string{"../../cmd/ateom-gvisor/main.go", "../../cmd/ateom-microvm/main.go"} { + for _, main := range []string{"../../cmd/ateworker-gvisor/main.go", "../../cmd/ateworker-microvm/main.go"} { src, err := os.ReadFile(main) if err != nil { t.Fatalf("reading %s: %v", main, err) } m := decl.FindSubmatch(src) if m == nil { - t.Fatalf("no `const serviceName = \"...\"` found in %s; if it moved, this test and ateomServices both need updating", main) + t.Fatalf("no `const serviceName = \"...\"` found in %s; if it moved, this test and workerServices both need updating", main) } name := string(m[1]) found[name] = true - if !ateomServices[name] { - t.Errorf("%s reports service.name %q, which ateomServices does not allow; the relay would refuse all of its telemetry", main, name) + if !workerServices[name] { + t.Errorf("%s reports service.name %q, which workerServices does not allow; the relay would refuse all of its telemetry", main, name) } } - for name := range ateomServices { + for name := range workerServices { if !found[name] { - t.Errorf("ateomServices allows %q, but no ateom binary declares it", name) + t.Errorf("workerServices allows %q, but no worker binary declares it", name) } } } @@ -621,7 +621,7 @@ func TestNewServerRejectsRelativeSocketPath(t *testing.T) { relay.Stop() } if err == nil { - t.Fatal("NewServer with a relative socket path returned no error; it would listen somewhere no ateom can name") + t.Fatal("NewServer with a relative socket path returned no error; it would listen somewhere no worker can name") } if !strings.Contains(err.Error(), "absolute") { t.Errorf("NewServer error = %v, want it to say the path must be absolute", err) @@ -638,7 +638,7 @@ func TestServeLeavesAPopulatedDirectoryAlone(t *testing.T) { if err := os.MkdirAll(dir, 0o755); err != nil { t.Fatalf("MkdirAll: %v", err) } - occupant := filepath.Join(dir, "ateom.sock") + occupant := filepath.Join(dir, "worker.sock") if err := os.WriteFile(occupant, nil, 0o600); err != nil { t.Fatalf("planting a neighbouring socket: %v", err) } @@ -832,7 +832,7 @@ func exportAll(t *testing.T, sink *fakeCollector, sock, service string, ctx cont } // The upstream leg is atelet's connection, so its credentials are atelet's. An -// ateom that sets a header of its own must not get to choose what atelet +// worker that sets a header of its own must not get to choose what atelet // presents to the collector. func TestExportDropsClientMetadata(t *testing.T) { sink, collector := startFakeCollector(t) @@ -842,7 +842,7 @@ func TestExportDropsClientMetadata(t *testing.T) { "authorization", "Bearer client-token", "custom-header", "custom-value", ) - traceMD, metricMD, logMD := exportAll(t, sink, sock, "ateom-gvisor", ctx) + traceMD, metricMD, logMD := exportAll(t, sink, sock, "ateworker-gvisor", ctx) for _, tc := range []struct { signal string @@ -869,7 +869,7 @@ func TestExportAttachesAteletHeaders(t *testing.T) { // The client sends its own, which must lose to atelet's rather than // appending a second value the collector might pick either way. ctx := metadata.AppendToOutgoingContext(context.Background(), "authorization", "Bearer client-token") - traceMD, metricMD, logMD := exportAll(t, sink, sock, "ateom-gvisor", ctx) + traceMD, metricMD, logMD := exportAll(t, sink, sock, "ateworker-gvisor", ctx) if got := traceMD.Get("authorization"); len(got) != 1 || got[0] != "Bearer atelet-token" { t.Errorf("trace export authorization = %v, want exactly [Bearer atelet-token]", got) diff --git a/internal/plugins/gcp-secret-manager/internal/provider/projectauthz_test.go b/internal/plugins/gcp-secret-manager/internal/provider/projectauthz_test.go index 589d8a62cf..68b79ba615 100644 --- a/internal/plugins/gcp-secret-manager/internal/provider/projectauthz_test.go +++ b/internal/plugins/gcp-secret-manager/internal/provider/projectauthz_test.go @@ -123,7 +123,7 @@ func TestActorAtespace(t *testing.T) { {id: "spiffe://substrate-actor.local/actor/team-a/agent-1", want: "team-a"}, {id: "", wantErr: true}, {id: "spiffe://cluster.local/actor/team-a/agent-1", wantErr: true}, - {id: "spiffe://substrate-actor.local/ateom-for-actor/team-a/agent-1", wantErr: true}, + {id: "spiffe://substrate-actor.local/worker-for-actor/team-a/agent-1", wantErr: true}, {id: "spiffe://substrate-actor.local/actor/team-a", wantErr: true}, {id: "spiffe://substrate-actor.local/actor/team-a/agent-1/extra", wantErr: true}, {id: "spiffe://substrate-actor.local/actor/Team_A/agent-1", wantErr: true}, diff --git a/internal/preview/preview.go b/internal/preview/preview.go new file mode 100644 index 0000000000..968315596c --- /dev/null +++ b/internal/preview/preview.go @@ -0,0 +1,184 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package preview manages preview feature gates enabled via the --preview flag. +// +// The gates are process-wide. A binary calls Init exactly once, before +// anything calls IsEnabled or AsMap. +package preview + +import ( + "fmt" + "log/slog" + "slices" + "strings" + "sync" +) + +// Gate identifies a gate-controlled preview feature. Preview gates are +// disabled by default, except: +// - when a gated feature becomes "stable" (aka GA) +// - when a gated feature is removed +// +// In both cases, the gate is retained for a while (for compatibility with +// existing uses of the --preview flag), but the feature is either always +// enabled or totally disabled, respectively. Such gates are "vestigial". +type Gate string + +// Gates should always be named in UpperCamelCase. Please keep this list +// sorted alphabetically by value. +const ( + // GateExternalVolumes controls whether external volumes are supported. + GateExternalVolumes Gate = "ExternalVolumes" +) + +// All is the pseudo-gate name that enables every known gate. +const All = "ALL" + +// realKnownGates lists all declared preview gates. Preview gates are disabled by +// default and enabled via the --preview flag. +var realKnownGates = []Gate{ + GateExternalVolumes, +} + +var globalKnownGates []Gate // all known gates +var globalStateMap map[string]bool // gate-name -> enabled? +var globalDisabledMap map[string]bool // gate-name -> false (all disabled) +var globalInitialized bool // true if Init has been called, false otherwise +var globalInitLock sync.Mutex + +// init sets the globalStateMap to the default state of all known gates disabled. It +// is called automatically on package initialization, but can be overridden by +// calling Init. +func init() { + globalInitLock.Lock() + defer globalInitLock.Unlock() + if globalStateMap != nil { + return + } + + slices.Sort(realKnownGates) + if m, err := newGateMap(realKnownGates); err != nil { + panic(fmt.Sprintf("preview.init failed: %v", err)) + } else { + globalKnownGates = realKnownGates + globalStateMap = m + globalDisabledMap = newDisabledMap(realKnownGates) + } +} + +// Init sets the process-wide preview gates from the --preview flag values. +// The special value All enables all known gates. With no values, all gates are +// disabled. It panics if called again after the first successful call. +func Init(values ...string) error { + globalInitLock.Lock() + defer globalInitLock.Unlock() + if globalInitialized { + panic("preview.Init called more than once") + } + logFunc := func(state string) { + slog.Info("initializing preview gates", slog.String("state", state)) + } + return doInit(globalKnownGates, logFunc, values...) +} + +// This allows tests to use fake gates. +func doInit(knownGates []Gate, logFunc func(state string), values ...string) error { + if m, err := newGateMap(knownGates, values...); err != nil { + return err + } else { + globalKnownGates = knownGates + globalStateMap = m + globalDisabledMap = newDisabledMap(knownGates) + globalInitialized = true + + if logFunc != nil { + logStr := func() string { + en := make([]string, 0, len(knownGates)) + for _, g := range knownGates { + k := string(g) + v := "off" + if m[k] { + v = "on" + } + en = append(en, fmt.Sprintf("%s=%s", k, v)) + } + return fmt.Sprintf("[%s]", strings.Join(en, ", ")) + }() + logFunc(logStr) + } + } + return nil +} + +func newGateMap(knownGates []Gate, values ...string) (map[string]bool, error) { + tmpMap := make(map[string]bool, len(knownGates)) + all := false + // add all the specified gates + for _, v := range values { + if v == All { + all = true + } else if slices.Contains(knownGates, Gate(v)) { + tmpMap[string(v)] = true + } else { + return nil, fmt.Errorf("unknown preview gate %q: supported gates: %q", v, append([]Gate{All}, knownGates...)) + } + } + // add the rest + for _, g := range knownGates { + s := string(g) + tmpMap[s] = tmpMap[s] || all + } + return tmpMap, nil +} + +func newDisabledMap(knownGates []Gate) map[string]bool { + m := make(map[string]bool, len(knownGates)) + for _, g := range knownGates { + m[string(g)] = false + } + return m +} + +// Verify checks that the given values are all valid preview gates. +func Verify(values ...string) error { + return doVerify(globalKnownGates, values...) +} + +func doVerify(knownGates []Gate, values ...string) error { + _, err := newGateMap(knownGates, values...) + return err +} + +// IsEnabled reports whether the given preview gate is enabled. Unknown gates +// are always treated as disabled. +func IsEnabled(gate Gate) bool { + return globalStateMap[string(gate)] +} + +// AsMap returns the map of known gate names to their enabled state. +// The returned map MUST NOT be modified. +// +// TODO: Fix this to use an interface when DV supports it. +func AsMap() map[string]bool { + return globalStateMap +} + +// DisabledMap returns the map of known gate names, all disabled. +// The returned map MUST NOT be modified. +// +// TODO: Fix this to use an interface when DV supports it. +func DisabledMap() map[string]bool { + return globalDisabledMap +} diff --git a/internal/preview/preview_test.go b/internal/preview/preview_test.go new file mode 100644 index 0000000000..d903c48625 --- /dev/null +++ b/internal/preview/preview_test.go @@ -0,0 +1,189 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package preview + +import ( + "regexp" + "strings" + "testing" + + "github.com/google/go-cmp/cmp" +) + +// resetGlobalsForTest clears the global state. +func resetGlobalsForTest() { + globalKnownGates = nil + globalStateMap = nil + globalInitialized = false +} + +func wantPanic(t *testing.T, substr string, fn func()) { + t.Helper() + defer func() { + t.Helper() + r := recover() + if r == nil { + t.Fatalf("got no panic, want one containing %q", substr) + } + if msg, _ := r.(string); !strings.Contains(msg, substr) { + t.Fatalf("panic = %v, want one containing %q", r, substr) + } + }() + fn() +} + +func TestInit(t *testing.T) { + tests := []struct { + name string + values []string + wantErr bool + want map[string]bool + }{ + {name: "no values", want: map[string]bool{"Fake": false, "Phony": false}}, + {name: "ALL", values: []string{"ALL"}, want: map[string]bool{"Fake": true, "Phony": true}}, + {name: "Fake", values: []string{"Fake"}, want: map[string]bool{"Fake": true, "Phony": false}}, + {name: "Phony", values: []string{"Phony"}, want: map[string]bool{"Fake": false, "Phony": true}}, + {name: "Fake and Phony", values: []string{"Phony", "Fake"}, want: map[string]bool{"Fake": true, "Phony": true}}, + {name: "repeated ALL", values: []string{"ALL", "ALL"}, want: map[string]bool{"Fake": true, "Phony": true}}, + {name: "unknown", values: []string{"Bogus"}, wantErr: true}, + {name: "empty", values: []string{""}, wantErr: true}, + {name: "star is not a wildcard", values: []string{"*"}, wantErr: true}, + {name: "lowercase all", values: []string{"all"}, wantErr: true}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + resetGlobalsForTest() + err := doInit([]Gate{"Fake", "Phony"}, nil, tc.values...) + if (err != nil) != tc.wantErr { + t.Fatalf("Init(%q) error = %v, wantErr %v", tc.values, err, tc.wantErr) + } + if tc.wantErr { + if globalStateMap != nil { + t.Errorf("Init(%q) failed but set the gates", tc.values) + } + return + } + if diff := cmp.Diff(tc.want, AsMap()); diff != "" { + t.Errorf("AsMap() mismatch (-want +got):\n%s", diff) + } + }) + } +} + +var pascalCase = regexp.MustCompile(`^[A-Z][a-z0-9]+(?:[A-Z][a-z0-9]*)*$`) + +func TestRealGatesArePascalCase(t *testing.T) { + // Sanity-check the pattern itself. + for _, s := range []string{"ExternalVolumes", "Foo", "FooV2", "FooBar2Baz"} { + if !pascalCase.MatchString(s) { + t.Errorf("pattern rejected valid name %q", s) + } + } + for _, s := range []string{"", "externalVolumes", "External_Volumes", "External-Volumes", "EXTERNAL", All, "F", "Foo Bar"} { + if pascalCase.MatchString(s) { + t.Errorf("pattern accepted invalid name %q", s) + } + } + + if len(realKnownGates) == 0 { + t.Fatal("no real gates defined") + } + for _, g := range realKnownGates { + if !pascalCase.MatchString(string(g)) { + t.Errorf("gate %q is not PascalCase", g) + } + } +} + +func TestInitTwicePanics(t *testing.T) { + resetGlobalsForTest() + if err := Init(); err != nil { + t.Fatalf("Init(): %v", err) + } + wantPanic(t, "more than once", func() { _ = Init("ALL") }) +} + +func TestIsEnabledKnownGate(t *testing.T) { + InitForTestFake(t, []Gate{"Fake"}, "ALL") + if !IsEnabled(Gate("Fake")) { + t.Errorf("IsEnabled(unknown) = false, want true") + } +} + +func TestIsEnabledUnknownGate(t *testing.T) { + InitForTestFake(t, []Gate{"Fake"}, "ALL") + if IsEnabled(Gate("Bogus")) { + t.Errorf("IsEnabled(unknown) = true, want false") + } +} + +func TestVerify(t *testing.T) { + fakeGates := []Gate{"Fake", "Other"} + + for _, tc := range []struct { + values []string + wantErr bool + }{ + {values: nil}, + {values: []string{"ALL"}}, + {values: []string{"Fake"}}, + {values: []string{"Fake", "Other"}}, + {values: []string{"Bogus"}, wantErr: true}, + {values: []string{"ALL", "Bogus"}, wantErr: true}, + } { + if err := doVerify(fakeGates, tc.values...); (err != nil) != tc.wantErr { + t.Errorf("Verify(%q) = %v, want error %v", tc.values, err, tc.wantErr) + } + } +} + +func TestSetForTest(t *testing.T) { + resetGlobalsForTest() + t.Run("enabled", func(t *testing.T) { + InitForTestFake(t, []Gate{"Fake"}, "ALL") + if !IsEnabled("Fake") { + t.Errorf("IsEnabled(%q) = false, want true", "Fake") + } + t.Run("nested disabled", func(t *testing.T) { + InitForTest(t) + if IsEnabled("Fake") { + t.Errorf("IsEnabled(%q) = true, want false", "Fake") + } + }) + if !IsEnabled("Fake") { + t.Errorf("IsEnabled(%q) after nested test = false, want true", "Fake") + } + }) + if globalStateMap != nil { + t.Errorf("InitForTest did not restore the uninitialized state") + } + // Init still works once InitForTest has been undone. + if err := Init(); err != nil { + t.Errorf("Init() after InitForTest: %v", err) + } +} + +func TestSetForTestFakeRestoresKnownGates(t *testing.T) { + resetGlobalsForTest() + t.Run("fake", func(t *testing.T) { + InitForTestFake(t, []Gate{"Fake"}, "Fake") + if err := Verify("Fake"); err != nil { + t.Errorf("Verify(Fake) with the fake installed = %v", err) + } + }) + if err := Verify("Fake"); err == nil { + t.Errorf("Verify(Fake) after the fake's test ended = nil, want an error: %v", globalStateMap) + } +} diff --git a/internal/preview/request.go b/internal/preview/request.go new file mode 100644 index 0000000000..46283121e8 --- /dev/null +++ b/internal/preview/request.go @@ -0,0 +1,108 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package preview + +import ( + "context" + "strconv" + + "google.golang.org/grpc" + "google.golang.org/grpc/metadata" +) + +// Header is the gRPC metadata key with which an API client says it intends +// to use preview features. It is a coarse guard against using them by +// accident, not an access control: any client may send it, and it applies to +// every enabled gate at once. Its value is a boolean, e.g. "true". +const Header = "Ate-Preview-Allowed" + +type restrictedKey struct{} + +// ContextWithOptOut marks ctx as being opted-out from preview features. +func ContextWithOptOut(ctx context.Context) context.Context { + return context.WithValue(ctx, restrictedKey{}, true) +} + +// ContextWithOptIn marks ctx as being opted-in to preview features, overriding +// any earlier ContextWithOptOut. Code which acts for the system rather than +// for a user, such as workflows, uses it so that work which an API request +// started is not limited by that request's opt-out. +func ContextWithOptIn(ctx context.Context) context.Context { + return context.WithValue(ctx, restrictedKey{}, false) +} + +// IsOptedOut reports whether the context has opted-out of using preview +// features. By default, contexts are allowed to use preview, but some +// call-paths (e.g. public API) may restrict that, so we provide this mechanism +// to explicitly opt out. +func IsOptedOut(ctx context.Context) bool { + restricted, _ := ctx.Value(restrictedKey{}).(bool) + return restricted +} + +// IsEnabledFor reports whether gate is enabled in this process and ctx has +// not opted-out of preview features. +func IsEnabledFor(ctx context.Context, gate Gate) bool { + return IsEnabled(gate) && !IsOptedOut(ctx) +} + +// requestOptedIn reports whether the incoming request in ctx sent Header +// with a true value. +func requestOptedIn(ctx context.Context) bool { + md, _ := metadata.FromIncomingContext(ctx) + for _, v := range md.Get(Header) { + if b, err := strconv.ParseBool(v); err == nil && b { + return true + } + } + return false +} + +func optOutIfNotOptIn(ctx context.Context) context.Context { + if requestOptedIn(ctx) { + return ctx + } + return ContextWithOptOut(ctx) +} + +// UnaryServerInterceptor restricts each request that did not send Header. +func UnaryServerInterceptor(ctx context.Context, req any, _ *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) { + return handler(optOutIfNotOptIn(ctx), req) +} + +// StreamServerInterceptor restricts each stream that did not send Header. +func StreamServerInterceptor(srv any, ss grpc.ServerStream, _ *grpc.StreamServerInfo, handler grpc.StreamHandler) error { + return handler(srv, &restrictedStream{ServerStream: ss, ctx: optOutIfNotOptIn(ss.Context())}) +} + +type restrictedStream struct { + grpc.ServerStream + ctx context.Context +} + +func (s *restrictedStream) Context() context.Context { return s.ctx } + +// DialOptionsWithOptIn returns gRPC dial options which send Header on every call made +// over the connection. +func DialOptionsWithOptIn() []grpc.DialOption { + return []grpc.DialOption{ + grpc.WithChainUnaryInterceptor(func(ctx context.Context, method string, req, reply any, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error { + return invoker(metadata.AppendToOutgoingContext(ctx, Header, "true"), method, req, reply, cc, opts...) + }), + grpc.WithChainStreamInterceptor(func(ctx context.Context, desc *grpc.StreamDesc, cc *grpc.ClientConn, method string, streamer grpc.Streamer, opts ...grpc.CallOption) (grpc.ClientStream, error) { + return streamer(metadata.AppendToOutgoingContext(ctx, Header, "true"), desc, cc, method, opts...) + }), + } +} diff --git a/internal/preview/request_test.go b/internal/preview/request_test.go new file mode 100644 index 0000000000..3c0363d5e1 --- /dev/null +++ b/internal/preview/request_test.go @@ -0,0 +1,186 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package preview + +import ( + "context" + "net" + "testing" + + "google.golang.org/grpc" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/metadata" + "google.golang.org/grpc/status" + "google.golang.org/grpc/test/bufconn" + "google.golang.org/protobuf/types/known/emptypb" +) + +func TestUnaryServerInterceptor(t *testing.T) { + for _, tc := range []struct { + name string + values []string + want bool + }{ + {name: "no metadata", want: false}, + {name: "true", values: []string{"true"}, want: true}, + {name: "TRUE", values: []string{"TRUE"}, want: true}, + {name: "false", values: []string{"false"}, want: false}, + {name: "garbage", values: []string{"yes please"}, want: false}, + {name: "empty", values: []string{""}, want: false}, + {name: "any true", values: []string{"false", "true"}, want: true}, + } { + t.Run(tc.name, func(t *testing.T) { + ctx := context.Background() + if tc.values != nil { + md := metadata.MD{} + md.Append(Header, tc.values...) + ctx = metadata.NewIncomingContext(ctx, md) + } + var got bool + _, err := UnaryServerInterceptor(ctx, nil, &grpc.UnaryServerInfo{}, func(ctx context.Context, _ any) (any, error) { + got = !IsOptedOut(ctx) + return nil, nil + }) + if err != nil { + t.Fatal(err) + } + if got != tc.want { + t.Errorf("opted in (!IsOptedOut()) = %v, want %v", got, tc.want) + } + }) + } +} + +type fakeStream struct { + grpc.ServerStream + ctx context.Context +} + +func (s fakeStream) Context() context.Context { return s.ctx } + +func TestStreamServerInterceptor(t *testing.T) { + for _, allowed := range []bool{false, true} { + ctx := context.Background() + if allowed { + ctx = metadata.NewIncomingContext(ctx, metadata.Pairs(Header, "true")) + } + var got bool + err := StreamServerInterceptor(nil, fakeStream{ctx: ctx}, &grpc.StreamServerInfo{}, func(_ any, ss grpc.ServerStream) error { + got = !IsOptedOut(ss.Context()) + return nil + }) + if err != nil { + t.Fatal(err) + } + if got != allowed { + t.Errorf("header sent = %v: opted in (!IsOptedOut()) = %v", allowed, got) + } + } +} + +func TestContextWithOptInOut(t *testing.T) { + ctx := context.Background() + if IsOptedOut(ctx) { + t.Errorf("IsOptedOut(unmarked) = true, want false") + } + ctx = ContextWithOptOut(ctx) + if !IsOptedOut(ctx) { + t.Errorf("IsOptedOut(opted out) = false, want true") + } + ctx = ContextWithOptIn(ctx) + if IsOptedOut(ctx) { + t.Errorf("IsOptedOut(opted out, then in) = true, want false") + } + ctx = ContextWithOptOut(ctx) + if !IsOptedOut(ctx) { + t.Errorf("IsOptedOut(opted out, in, then out) = false, want true") + } +} + +func TestIsEnabledFor(t *testing.T) { + InitForTestFake(t, []Gate{"A", "B"}, "A") + for _, tc := range []struct { + name string + ctx context.Context + gate Gate + want bool + }{ + {name: "enabled, unmarked", ctx: context.Background(), gate: "A", want: true}, + {name: "enabled, opted out", ctx: ContextWithOptOut(context.Background()), gate: "A", want: false}, + {name: "enabled, opted in", ctx: ContextWithOptIn(ContextWithOptOut(context.Background())), gate: "A", want: true}, + {name: "disabled, opted in", ctx: ContextWithOptIn(context.Background()), gate: "B", want: false}, + } { + t.Run(tc.name, func(t *testing.T) { + if got := IsEnabledFor(tc.ctx, tc.gate); got != tc.want { + t.Errorf("IsEnabledFor(%s) = %v, want %v", tc.gate, got, tc.want) + } + }) + } +} + +func TestDialOptions(t *testing.T) { + lis := bufconn.Listen(1 << 16) + allowed := make(chan bool, 1) + srv := grpc.NewServer( + grpc.ChainStreamInterceptor(StreamServerInterceptor), + grpc.UnknownServiceHandler(func(_ any, ss grpc.ServerStream) error { + allowed <- !IsOptedOut(ss.Context()) + return status.Error(codes.Unimplemented, "test") + }), + ) + go func() { _ = srv.Serve(lis) }() + t.Cleanup(srv.Stop) + + dial := func(t *testing.T, extra ...grpc.DialOption) *grpc.ClientConn { + opts := append([]grpc.DialOption{ + grpc.WithTransportCredentials(insecure.NewCredentials()), + grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) { return lis.Dial() }), + }, extra...) + cc, err := grpc.NewClient("passthrough://bufnet", opts...) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = cc.Close() }) + return cc + } + + for _, tc := range []struct { + name string + opts []grpc.DialOption + want bool + }{ + {name: "without", want: false}, + {name: "with", opts: DialOptionsWithOptIn(), want: true}, + } { + t.Run(tc.name, func(t *testing.T) { + cc := dial(t, tc.opts...) + + _ = cc.Invoke(t.Context(), "/test.Svc/Unary", &emptypb.Empty{}, &emptypb.Empty{}) + if got := <-allowed; got != tc.want { + t.Errorf("unary: opted in (!IsOptedOut()) on server = %v, want %v", got, tc.want) + } + + stream, err := cc.NewStream(t.Context(), &grpc.StreamDesc{ServerStreams: true}, "/test.Svc/Stream") + if err != nil { + t.Fatal(err) + } + _ = stream.RecvMsg(&emptypb.Empty{}) + if got := <-allowed; got != tc.want { + t.Errorf("stream: opted in (!IsOptedOut()) on server = %v, want %v", got, tc.want) + } + }) + } +} diff --git a/internal/preview/testing.go b/internal/preview/testing.go new file mode 100644 index 0000000000..7a059fcd85 --- /dev/null +++ b/internal/preview/testing.go @@ -0,0 +1,66 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package preview + +// The methods we need from testing.TB, so we do not have to import testing +// into non-test code. +type testingTB interface { + Helper() + Setenv(key, value string) + Cleanup(func()) + Fatalf(format string, args ...any) + Log(args ...any) +} + +// InitForTest sets the process-wide preview gates for the duration of tb, as +// Init would with values, and restores the previous state when tb ends. It may +// be called whether or not Init has been. Tests that use it must not run in +// parallel. +func InitForTest(tb testingTB, values ...string) { + tb.Helper() + InitForTestFake(tb, realKnownGates, values...) +} + +// InitForTestFake is InitForTest with fake gate names. Tests of the preview +// gate system itself should use this to avoid depending on the real gate +// names. Tests that use it must not run in parallel. +func InitForTestFake(tb testingTB, fakeKnownGates []Gate, values ...string) { + tb.Helper() + + // Setenv fails the test if it, or an ancestor, is parallel. + tb.Setenv("ATE_PREVIEW_SET_FOR_TEST", "1") + + globalInitLock.Lock() + defer globalInitLock.Unlock() + + prevInitialized := globalInitialized + prevKnownGates := globalKnownGates + prevStateMap := globalStateMap + prevDisabledMap := globalDisabledMap + + logFunc := func(state string) { + tb.Log("initialized preview gates for test:", state) + } + if err := doInit(fakeKnownGates, logFunc, values...); err != nil { + tb.Fatalf("preview.InitForTestFake: %v", err) + } + + tb.Cleanup(func() { + globalInitialized = prevInitialized + globalKnownGates = prevKnownGates + globalStateMap = prevStateMap + globalDisabledMap = prevDisabledMap + }) +} diff --git a/internal/proto/ateletpb/atelet.pb.go b/internal/proto/ateletpb/atelet.pb.go index 4ebb7b9b23..b05826b2a6 100644 --- a/internal/proto/ateletpb/atelet.pb.go +++ b/internal/proto/ateletpb/atelet.pb.go @@ -1309,8 +1309,8 @@ func (x *MintActorCertificateResponse) GetActorCertificates() [][]byte { type TerminateRequest struct { state protoimpl.MessageState `protogen:"open.v1"` - // The UID of the worker pod whose ateom hosts the actor. If unset, atelet - // only cleans up the actor's node resources; it does not reach an ateom. + // The UID of the worker pod whose the worker hosts the actor. If unset, atelet + // only cleans up the actor's node resources; it does not reach a worker. // // +k8s:optional // +k8s:format=k8s-uuid @@ -1458,7 +1458,7 @@ func (*TerminateResponse) Descriptor() ([]byte, []int) { type RunRequest struct { state protoimpl.MessageState `protogen:"open.v1"` - // The UID of the worker pod whose ateom hosts the actor. + // The UID of the worker pod whose the worker hosts the actor. // // +k8s:required // +k8s:format=k8s-uuid @@ -3135,10 +3135,10 @@ type Snapshot struct { // for the actor, or for its newest under another snapshot_uuid: the // caller's view is stale. // - // TODO: Temporarily, the control plane does not set this and atelet does - // not check it. + // TODO: Temporarily, atelet does not compare it with the snapshots it holds. // - // +k8s:optional + // +k8s:required + // +k8s:minimum=1 Generation int32 `protobuf:"varint,2,opt,name=generation,proto3" json:"generation,omitempty"` // On Checkpoint: the fidelity to capture. // On Restore: the fidelity to restore the actor with. @@ -3428,7 +3428,7 @@ func (x *UploadConfig) GetMode() UploadMode { // +k8s:customValidation # immediate upload requires snapshot.object type CheckpointRequest struct { state protoimpl.MessageState `protogen:"open.v1"` - // The UID of the worker pod whose ateom hosts the actor. + // The UID of the worker pod whose the worker hosts the actor. // // +k8s:required // +k8s:format=k8s-uuid @@ -3575,9 +3575,6 @@ type CheckpointResponse struct { // snapshot, unless UPLOAD_MODE_IMMEDIATE kept no local copy. The control // plane can drop its entries for any local-only generation not named here: // atelet has deleted it, and will never upload it. - // - // TODO: Temporarily, atelet leaves this unset. An unset state says nothing - // about which snapshots atelet holds. State *ActorSnapshotState `protobuf:"bytes,1,opt,name=state,proto3" json:"state,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -3622,7 +3619,7 @@ func (x *CheckpointResponse) GetState() *ActorSnapshotState { type RestoreRequest struct { state protoimpl.MessageState `protogen:"open.v1"` - // The UID of the worker pod whose ateom hosts the actor. + // The UID of the worker pod whose the worker hosts the actor. // // +k8s:required // +k8s:format=k8s-uuid @@ -3941,9 +3938,6 @@ type PromoteSnapshotResponse struct { state protoimpl.MessageState `protogen:"open.v1"` // The actor's state once promoted: latest_durable is its latest snapshot, // and latest_local remains set if atelet still holds the local copy. - // - // TODO: Temporarily, atelet leaves this unset. An unset state says nothing - // about which snapshots atelet holds. State *ActorSnapshotState `protobuf:"bytes,1,opt,name=state,proto3" json:"state,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -4242,13 +4236,13 @@ const file_atelet_proto_rawDesc = "" + "UploadMode\x12\x1b\n" + "\x17UPLOAD_MODE_UNSPECIFIED\x10\x00\x12\x18\n" + "\x14UPLOAD_MODE_DEFERRED\x10\x01\x12\x19\n" + - "\x15UPLOAD_MODE_IMMEDIATE\x10\x022\x8d\x03\n" + - "\fAteomSupport\x12c\n" + + "\x15UPLOAD_MODE_IMMEDIATE\x10\x022\x8e\x03\n" + + "\rWorkerSupport\x12c\n" + "\x14MintActorCertificate\x12#.atelet.MintActorCertificateRequest\x1a$.atelet.MintActorCertificateResponse\"\x00\x12Q\n" + "\x0eRegisterWorker\x12\x1d.atelet.RegisterWorkerRequest\x1a\x1e.atelet.RegisterWorkerResponse\"\x00\x12`\n" + "\x13RequestActorSuspend\x12\".atelet.RequestActorSuspendRequest\x1a#.atelet.RequestActorSuspendResponse\"\x00\x12c\n" + - "\x14GetActorEgressPolicy\x12#.atelet.GetActorEgressPolicyRequest\x1a$.atelet.GetActorEgressPolicyResponse\"\x002\xde\x02\n" + - "\vAteomHerder\x120\n" + + "\x14GetActorEgressPolicy\x12#.atelet.GetActorEgressPolicyRequest\x1a$.atelet.GetActorEgressPolicyResponse\"\x002\xd9\x02\n" + + "\x06Atelet\x120\n" + "\x03Run\x12\x12.atelet.RunRequest\x1a\x13.atelet.RunResponse\"\x00\x12E\n" + "\n" + "Checkpoint\x12\x19.atelet.CheckpointRequest\x1a\x1a.atelet.CheckpointResponse\"\x00\x12<\n" + @@ -4399,24 +4393,24 @@ var file_atelet_proto_depIdxs = []int32{ 51, // 59: atelet.PromoteSnapshotResponse.state:type_name -> atelet.ActorSnapshotState 26, // 60: atelet.ArchAssets.FilesEntry.value:type_name -> atelet.AssetFile 27, // 61: atelet.SandboxAssets.AssetsEntry.value:type_name -> atelet.ArchAssets - 20, // 62: atelet.AteomSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest - 3, // 63: atelet.AteomSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest - 11, // 64: atelet.AteomSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest - 13, // 65: atelet.AteomSupport.GetActorEgressPolicy:input_type -> atelet.GetActorEgressPolicyRequest - 24, // 66: atelet.AteomHerder.Run:input_type -> atelet.RunRequest - 53, // 67: atelet.AteomHerder.Checkpoint:input_type -> atelet.CheckpointRequest - 55, // 68: atelet.AteomHerder.Restore:input_type -> atelet.RestoreRequest - 57, // 69: atelet.AteomHerder.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest - 22, // 70: atelet.AteomHerder.Terminate:input_type -> atelet.TerminateRequest - 21, // 71: atelet.AteomSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse - 10, // 72: atelet.AteomSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse - 12, // 73: atelet.AteomSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse - 14, // 74: atelet.AteomSupport.GetActorEgressPolicy:output_type -> atelet.GetActorEgressPolicyResponse - 47, // 75: atelet.AteomHerder.Run:output_type -> atelet.RunResponse - 54, // 76: atelet.AteomHerder.Checkpoint:output_type -> atelet.CheckpointResponse - 56, // 77: atelet.AteomHerder.Restore:output_type -> atelet.RestoreResponse - 58, // 78: atelet.AteomHerder.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse - 23, // 79: atelet.AteomHerder.Terminate:output_type -> atelet.TerminateResponse + 20, // 62: atelet.WorkerSupport.MintActorCertificate:input_type -> atelet.MintActorCertificateRequest + 3, // 63: atelet.WorkerSupport.RegisterWorker:input_type -> atelet.RegisterWorkerRequest + 11, // 64: atelet.WorkerSupport.RequestActorSuspend:input_type -> atelet.RequestActorSuspendRequest + 13, // 65: atelet.WorkerSupport.GetActorEgressPolicy:input_type -> atelet.GetActorEgressPolicyRequest + 24, // 66: atelet.Atelet.Run:input_type -> atelet.RunRequest + 53, // 67: atelet.Atelet.Checkpoint:input_type -> atelet.CheckpointRequest + 55, // 68: atelet.Atelet.Restore:input_type -> atelet.RestoreRequest + 57, // 69: atelet.Atelet.PromoteSnapshot:input_type -> atelet.PromoteSnapshotRequest + 22, // 70: atelet.Atelet.Terminate:input_type -> atelet.TerminateRequest + 21, // 71: atelet.WorkerSupport.MintActorCertificate:output_type -> atelet.MintActorCertificateResponse + 10, // 72: atelet.WorkerSupport.RegisterWorker:output_type -> atelet.RegisterWorkerResponse + 12, // 73: atelet.WorkerSupport.RequestActorSuspend:output_type -> atelet.RequestActorSuspendResponse + 14, // 74: atelet.WorkerSupport.GetActorEgressPolicy:output_type -> atelet.GetActorEgressPolicyResponse + 47, // 75: atelet.Atelet.Run:output_type -> atelet.RunResponse + 54, // 76: atelet.Atelet.Checkpoint:output_type -> atelet.CheckpointResponse + 56, // 77: atelet.Atelet.Restore:output_type -> atelet.RestoreResponse + 58, // 78: atelet.Atelet.PromoteSnapshot:output_type -> atelet.PromoteSnapshotResponse + 23, // 79: atelet.Atelet.Terminate:output_type -> atelet.TerminateResponse 71, // [71:80] is the sub-list for method output_type 62, // [62:71] is the sub-list for method input_type 62, // [62:62] is the sub-list for extension type_name diff --git a/internal/proto/ateletpb/atelet.proto b/internal/proto/ateletpb/atelet.proto index 06ce0fff84..7608234b8e 100644 --- a/internal/proto/ateletpb/atelet.proto +++ b/internal/proto/ateletpb/atelet.proto @@ -20,11 +20,11 @@ import "google/protobuf/timestamp.proto"; option go_package = "github.com/agent-substrate/substrate/internal/proto/ateletpb"; -// AteomSupport provides callouts for ateom. +// WorkerSupport provides callouts for worker implementations. // -// Called by individual ateom pods over a local connection, authenticated with -// ateom's k8s pod identity mTLS certificate. -service AteomSupport { +// Called by individual worker pods over a local connection, authenticated with +// the worker's k8s pod identity mTLS certificate. +service WorkerSupport { // Request an atunnel certificate for the given actor. // // TODO(identity): Rename to MintAtunnelCertificate, as distinct from @@ -43,7 +43,7 @@ service AteomSupport { // passed through as it was given, so a worker can tell a decision from a // failure to ask. Callers must not hold a lock the resulting Checkpoint // would need: the control plane answers this by driving a suspend back - // through atelet into the calling ateom. + // through atelet into the calling worker. rpc RequestActorSuspend(RequestActorSuspendRequest) returns (RequestActorSuspendResponse) {} // GetActorEgressPolicy returns the egress policy of the actor this @@ -307,25 +307,25 @@ message MintActorCertificateResponse { repeated bytes actor_certificates = 1; } -// AteomHerder allows ate-apiserver to issue control calls to the atelet. +// Atelet allows ate-apiserver to issue control calls to the atelet. // // Called by ate-api-server over cluster networking. ate-api-server // authenticates with its k8s pod identity mTLS certificate. -service AteomHerder { - // Run tells atelet to create a new containerized workload from scratch on an - // ateom. +service Atelet { + // Run tells atelet to create a new containerized workload from scratch on a + // worker. rpc Run(RunRequest) returns (RunResponse) {} // Checkpoint saves the workload's state as a new snapshot of the actor and - // returns the ateom to a blank state. upload_config says when the snapshot - // becomes durable. Once ateom has captured the workload, and before the new + // returns the worker to a blank state. upload_config says when the snapshot + // becomes durable. Once the worker has captured the workload, and before the new // snapshot is uploaded or stored locally, atelet deletes the actor's older // local snapshots, except one still being uploaded, which it deletes when // that upload ends. A Checkpoint that then fails to upload or store the new // snapshot leaves the actor with no local snapshot. rpc Checkpoint(CheckpointRequest) returns (CheckpointResponse) {} - // Restore restores a workload from a snapshot onto an ateom. + // Restore restores a workload from a snapshot onto a worker. rpc Restore(RestoreRequest) returns (RestoreResponse) {} // PromoteSnapshot makes `snapshot` durable in object storage while keeping @@ -348,8 +348,8 @@ service AteomHerder { } message TerminateRequest { - // The UID of the worker pod whose ateom hosts the actor. If unset, atelet - // only cleans up the actor's node resources; it does not reach an ateom. + // The UID of the worker pod whose the worker hosts the actor. If unset, atelet + // only cleans up the actor's node resources; it does not reach a worker. // // +k8s:optional // +k8s:format=k8s-uuid @@ -388,7 +388,7 @@ message TerminateResponse { } message RunRequest { - // The UID of the worker pod whose ateom hosts the actor. + // The UID of the worker pod whose the worker hosts the actor. // // +k8s:required // +k8s:format=k8s-uuid @@ -880,10 +880,10 @@ message Snapshot { // for the actor, or for its newest under another snapshot_uuid: the // caller's view is stale. // - // TODO: Temporarily, the control plane does not set this and atelet does - // not check it. + // TODO: Temporarily, atelet does not compare it with the snapshots it holds. // - // +k8s:optional + // +k8s:required + // +k8s:minimum=1 int32 generation = 2; // On Checkpoint: the fidelity to capture. @@ -987,7 +987,7 @@ enum UploadMode { // +k8s:customValidation # immediate upload requires snapshot.object message CheckpointRequest { - // The UID of the worker pod whose ateom hosts the actor. + // The UID of the worker pod whose the worker hosts the actor. // // +k8s:required // +k8s:format=k8s-uuid @@ -1046,14 +1046,11 @@ message CheckpointResponse { // snapshot, unless UPLOAD_MODE_IMMEDIATE kept no local copy. The control // plane can drop its entries for any local-only generation not named here: // atelet has deleted it, and will never upload it. - // - // TODO: Temporarily, atelet leaves this unset. An unset state says nothing - // about which snapshots atelet holds. ActorSnapshotState state = 1; } message RestoreRequest { - // The UID of the worker pod whose ateom hosts the actor. + // The UID of the worker pod whose the worker hosts the actor. // // +k8s:required // +k8s:format=k8s-uuid @@ -1163,8 +1160,5 @@ message PromoteSnapshotRequest { message PromoteSnapshotResponse { // The actor's state once promoted: latest_durable is its latest snapshot, // and latest_local remains set if atelet still holds the local copy. - // - // TODO: Temporarily, atelet leaves this unset. An unset state says nothing - // about which snapshots atelet holds. ActorSnapshotState state = 1; } diff --git a/internal/proto/ateletpb/atelet_grpc.pb.go b/internal/proto/ateletpb/atelet_grpc.pb.go index 38896e9e80..6a9edd52c7 100644 --- a/internal/proto/ateletpb/atelet_grpc.pb.go +++ b/internal/proto/ateletpb/atelet_grpc.pb.go @@ -33,21 +33,21 @@ import ( const _ = grpc.SupportPackageIsVersion9 const ( - AteomSupport_MintActorCertificate_FullMethodName = "/atelet.AteomSupport/MintActorCertificate" - AteomSupport_RegisterWorker_FullMethodName = "/atelet.AteomSupport/RegisterWorker" - AteomSupport_RequestActorSuspend_FullMethodName = "/atelet.AteomSupport/RequestActorSuspend" - AteomSupport_GetActorEgressPolicy_FullMethodName = "/atelet.AteomSupport/GetActorEgressPolicy" + WorkerSupport_MintActorCertificate_FullMethodName = "/atelet.WorkerSupport/MintActorCertificate" + WorkerSupport_RegisterWorker_FullMethodName = "/atelet.WorkerSupport/RegisterWorker" + WorkerSupport_RequestActorSuspend_FullMethodName = "/atelet.WorkerSupport/RequestActorSuspend" + WorkerSupport_GetActorEgressPolicy_FullMethodName = "/atelet.WorkerSupport/GetActorEgressPolicy" ) -// AteomSupportClient is the client API for AteomSupport service. +// WorkerSupportClient is the client API for WorkerSupport service. // // For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream. // -// AteomSupport provides callouts for ateom. +// WorkerSupport provides callouts for worker implementations. // -// Called by individual ateom pods over a local connection, authenticated with -// ateom's k8s pod identity mTLS certificate. -type AteomSupportClient interface { +// Called by individual worker pods over a local connection, authenticated with +// the worker's k8s pod identity mTLS certificate. +type WorkerSupportClient interface { // Request an atunnel certificate for the given actor. // // TODO(identity): Rename to MintAtunnelCertificate, as distinct from @@ -64,7 +64,7 @@ type AteomSupportClient interface { // passed through as it was given, so a worker can tell a decision from a // failure to ask. Callers must not hold a lock the resulting Checkpoint // would need: the control plane answers this by driving a suspend back - // through atelet into the calling ateom. + // through atelet into the calling worker. RequestActorSuspend(ctx context.Context, in *RequestActorSuspendRequest, opts ...grpc.CallOption) (*RequestActorSuspendResponse, error) // GetActorEgressPolicy returns the egress policy of the actor this // worker hosts, which atelet reads from the control plane's @@ -73,63 +73,63 @@ type AteomSupportClient interface { GetActorEgressPolicy(ctx context.Context, in *GetActorEgressPolicyRequest, opts ...grpc.CallOption) (*GetActorEgressPolicyResponse, error) } -type ateomSupportClient struct { +type workerSupportClient struct { cc grpc.ClientConnInterface } -func NewAteomSupportClient(cc grpc.ClientConnInterface) AteomSupportClient { - return &ateomSupportClient{cc} +func NewWorkerSupportClient(cc grpc.ClientConnInterface) WorkerSupportClient { + return &workerSupportClient{cc} } -func (c *ateomSupportClient) MintActorCertificate(ctx context.Context, in *MintActorCertificateRequest, opts ...grpc.CallOption) (*MintActorCertificateResponse, error) { +func (c *workerSupportClient) MintActorCertificate(ctx context.Context, in *MintActorCertificateRequest, opts ...grpc.CallOption) (*MintActorCertificateResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(MintActorCertificateResponse) - err := c.cc.Invoke(ctx, AteomSupport_MintActorCertificate_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, WorkerSupport_MintActorCertificate_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomSupportClient) RegisterWorker(ctx context.Context, in *RegisterWorkerRequest, opts ...grpc.CallOption) (*RegisterWorkerResponse, error) { +func (c *workerSupportClient) RegisterWorker(ctx context.Context, in *RegisterWorkerRequest, opts ...grpc.CallOption) (*RegisterWorkerResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(RegisterWorkerResponse) - err := c.cc.Invoke(ctx, AteomSupport_RegisterWorker_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, WorkerSupport_RegisterWorker_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomSupportClient) RequestActorSuspend(ctx context.Context, in *RequestActorSuspendRequest, opts ...grpc.CallOption) (*RequestActorSuspendResponse, error) { +func (c *workerSupportClient) RequestActorSuspend(ctx context.Context, in *RequestActorSuspendRequest, opts ...grpc.CallOption) (*RequestActorSuspendResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(RequestActorSuspendResponse) - err := c.cc.Invoke(ctx, AteomSupport_RequestActorSuspend_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, WorkerSupport_RequestActorSuspend_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomSupportClient) GetActorEgressPolicy(ctx context.Context, in *GetActorEgressPolicyRequest, opts ...grpc.CallOption) (*GetActorEgressPolicyResponse, error) { +func (c *workerSupportClient) GetActorEgressPolicy(ctx context.Context, in *GetActorEgressPolicyRequest, opts ...grpc.CallOption) (*GetActorEgressPolicyResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(GetActorEgressPolicyResponse) - err := c.cc.Invoke(ctx, AteomSupport_GetActorEgressPolicy_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, WorkerSupport_GetActorEgressPolicy_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -// AteomSupportServer is the server API for AteomSupport service. -// All implementations must embed UnimplementedAteomSupportServer +// WorkerSupportServer is the server API for WorkerSupport service. +// All implementations must embed UnimplementedWorkerSupportServer // for forward compatibility. // -// AteomSupport provides callouts for ateom. +// WorkerSupport provides callouts for worker implementations. // -// Called by individual ateom pods over a local connection, authenticated with -// ateom's k8s pod identity mTLS certificate. -type AteomSupportServer interface { +// Called by individual worker pods over a local connection, authenticated with +// the worker's k8s pod identity mTLS certificate. +type WorkerSupportServer interface { // Request an atunnel certificate for the given actor. // // TODO(identity): Rename to MintAtunnelCertificate, as distinct from @@ -146,150 +146,150 @@ type AteomSupportServer interface { // passed through as it was given, so a worker can tell a decision from a // failure to ask. Callers must not hold a lock the resulting Checkpoint // would need: the control plane answers this by driving a suspend back - // through atelet into the calling ateom. + // through atelet into the calling worker. RequestActorSuspend(context.Context, *RequestActorSuspendRequest) (*RequestActorSuspendResponse, error) // GetActorEgressPolicy returns the egress policy of the actor this // worker hosts, which atelet reads from the control plane's // Control.GetActorEgressPolicy. An error is the control plane's, passed // through as it was given. GetActorEgressPolicy(context.Context, *GetActorEgressPolicyRequest) (*GetActorEgressPolicyResponse, error) - mustEmbedUnimplementedAteomSupportServer() + mustEmbedUnimplementedWorkerSupportServer() } -// UnimplementedAteomSupportServer must be embedded to have +// UnimplementedWorkerSupportServer must be embedded to have // forward compatible implementations. // // NOTE: this should be embedded by value instead of pointer to avoid a nil // pointer dereference when methods are called. -type UnimplementedAteomSupportServer struct{} +type UnimplementedWorkerSupportServer struct{} -func (UnimplementedAteomSupportServer) MintActorCertificate(context.Context, *MintActorCertificateRequest) (*MintActorCertificateResponse, error) { +func (UnimplementedWorkerSupportServer) MintActorCertificate(context.Context, *MintActorCertificateRequest) (*MintActorCertificateResponse, error) { return nil, status.Error(codes.Unimplemented, "method MintActorCertificate not implemented") } -func (UnimplementedAteomSupportServer) RegisterWorker(context.Context, *RegisterWorkerRequest) (*RegisterWorkerResponse, error) { +func (UnimplementedWorkerSupportServer) RegisterWorker(context.Context, *RegisterWorkerRequest) (*RegisterWorkerResponse, error) { return nil, status.Error(codes.Unimplemented, "method RegisterWorker not implemented") } -func (UnimplementedAteomSupportServer) RequestActorSuspend(context.Context, *RequestActorSuspendRequest) (*RequestActorSuspendResponse, error) { +func (UnimplementedWorkerSupportServer) RequestActorSuspend(context.Context, *RequestActorSuspendRequest) (*RequestActorSuspendResponse, error) { return nil, status.Error(codes.Unimplemented, "method RequestActorSuspend not implemented") } -func (UnimplementedAteomSupportServer) GetActorEgressPolicy(context.Context, *GetActorEgressPolicyRequest) (*GetActorEgressPolicyResponse, error) { +func (UnimplementedWorkerSupportServer) GetActorEgressPolicy(context.Context, *GetActorEgressPolicyRequest) (*GetActorEgressPolicyResponse, error) { return nil, status.Error(codes.Unimplemented, "method GetActorEgressPolicy not implemented") } -func (UnimplementedAteomSupportServer) mustEmbedUnimplementedAteomSupportServer() {} -func (UnimplementedAteomSupportServer) testEmbeddedByValue() {} +func (UnimplementedWorkerSupportServer) mustEmbedUnimplementedWorkerSupportServer() {} +func (UnimplementedWorkerSupportServer) testEmbeddedByValue() {} -// UnsafeAteomSupportServer may be embedded to opt out of forward compatibility for this service. -// Use of this interface is not recommended, as added methods to AteomSupportServer will +// UnsafeWorkerSupportServer may be embedded to opt out of forward compatibility for this service. +// Use of this interface is not recommended, as added methods to WorkerSupportServer will // result in compilation errors. -type UnsafeAteomSupportServer interface { - mustEmbedUnimplementedAteomSupportServer() +type UnsafeWorkerSupportServer interface { + mustEmbedUnimplementedWorkerSupportServer() } -func RegisterAteomSupportServer(s grpc.ServiceRegistrar, srv AteomSupportServer) { - // If the following call panics, it indicates UnimplementedAteomSupportServer was +func RegisterWorkerSupportServer(s grpc.ServiceRegistrar, srv WorkerSupportServer) { + // If the following call panics, it indicates UnimplementedWorkerSupportServer was // embedded by pointer and is nil. This will cause panics if an // unimplemented method is ever invoked, so we test this at initialization // time to prevent it from happening at runtime later due to I/O. if t, ok := srv.(interface{ testEmbeddedByValue() }); ok { t.testEmbeddedByValue() } - s.RegisterService(&AteomSupport_ServiceDesc, srv) + s.RegisterService(&WorkerSupport_ServiceDesc, srv) } -func _AteomSupport_MintActorCertificate_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _WorkerSupport_MintActorCertificate_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(MintActorCertificateRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomSupportServer).MintActorCertificate(ctx, in) + return srv.(WorkerSupportServer).MintActorCertificate(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomSupport_MintActorCertificate_FullMethodName, + FullMethod: WorkerSupport_MintActorCertificate_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomSupportServer).MintActorCertificate(ctx, req.(*MintActorCertificateRequest)) + return srv.(WorkerSupportServer).MintActorCertificate(ctx, req.(*MintActorCertificateRequest)) } return interceptor(ctx, in, info, handler) } -func _AteomSupport_RegisterWorker_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _WorkerSupport_RegisterWorker_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(RegisterWorkerRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomSupportServer).RegisterWorker(ctx, in) + return srv.(WorkerSupportServer).RegisterWorker(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomSupport_RegisterWorker_FullMethodName, + FullMethod: WorkerSupport_RegisterWorker_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomSupportServer).RegisterWorker(ctx, req.(*RegisterWorkerRequest)) + return srv.(WorkerSupportServer).RegisterWorker(ctx, req.(*RegisterWorkerRequest)) } return interceptor(ctx, in, info, handler) } -func _AteomSupport_RequestActorSuspend_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _WorkerSupport_RequestActorSuspend_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(RequestActorSuspendRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomSupportServer).RequestActorSuspend(ctx, in) + return srv.(WorkerSupportServer).RequestActorSuspend(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomSupport_RequestActorSuspend_FullMethodName, + FullMethod: WorkerSupport_RequestActorSuspend_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomSupportServer).RequestActorSuspend(ctx, req.(*RequestActorSuspendRequest)) + return srv.(WorkerSupportServer).RequestActorSuspend(ctx, req.(*RequestActorSuspendRequest)) } return interceptor(ctx, in, info, handler) } -func _AteomSupport_GetActorEgressPolicy_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _WorkerSupport_GetActorEgressPolicy_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(GetActorEgressPolicyRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomSupportServer).GetActorEgressPolicy(ctx, in) + return srv.(WorkerSupportServer).GetActorEgressPolicy(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomSupport_GetActorEgressPolicy_FullMethodName, + FullMethod: WorkerSupport_GetActorEgressPolicy_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomSupportServer).GetActorEgressPolicy(ctx, req.(*GetActorEgressPolicyRequest)) + return srv.(WorkerSupportServer).GetActorEgressPolicy(ctx, req.(*GetActorEgressPolicyRequest)) } return interceptor(ctx, in, info, handler) } -// AteomSupport_ServiceDesc is the grpc.ServiceDesc for AteomSupport service. +// WorkerSupport_ServiceDesc is the grpc.ServiceDesc for WorkerSupport service. // It's only intended for direct use with grpc.RegisterService, // and not to be introspected or modified (even as a copy) -var AteomSupport_ServiceDesc = grpc.ServiceDesc{ - ServiceName: "atelet.AteomSupport", - HandlerType: (*AteomSupportServer)(nil), +var WorkerSupport_ServiceDesc = grpc.ServiceDesc{ + ServiceName: "atelet.WorkerSupport", + HandlerType: (*WorkerSupportServer)(nil), Methods: []grpc.MethodDesc{ { MethodName: "MintActorCertificate", - Handler: _AteomSupport_MintActorCertificate_Handler, + Handler: _WorkerSupport_MintActorCertificate_Handler, }, { MethodName: "RegisterWorker", - Handler: _AteomSupport_RegisterWorker_Handler, + Handler: _WorkerSupport_RegisterWorker_Handler, }, { MethodName: "RequestActorSuspend", - Handler: _AteomSupport_RequestActorSuspend_Handler, + Handler: _WorkerSupport_RequestActorSuspend_Handler, }, { MethodName: "GetActorEgressPolicy", - Handler: _AteomSupport_GetActorEgressPolicy_Handler, + Handler: _WorkerSupport_GetActorEgressPolicy_Handler, }, }, Streams: []grpc.StreamDesc{}, @@ -297,34 +297,34 @@ var AteomSupport_ServiceDesc = grpc.ServiceDesc{ } const ( - AteomHerder_Run_FullMethodName = "/atelet.AteomHerder/Run" - AteomHerder_Checkpoint_FullMethodName = "/atelet.AteomHerder/Checkpoint" - AteomHerder_Restore_FullMethodName = "/atelet.AteomHerder/Restore" - AteomHerder_PromoteSnapshot_FullMethodName = "/atelet.AteomHerder/PromoteSnapshot" - AteomHerder_Terminate_FullMethodName = "/atelet.AteomHerder/Terminate" + Atelet_Run_FullMethodName = "/atelet.Atelet/Run" + Atelet_Checkpoint_FullMethodName = "/atelet.Atelet/Checkpoint" + Atelet_Restore_FullMethodName = "/atelet.Atelet/Restore" + Atelet_PromoteSnapshot_FullMethodName = "/atelet.Atelet/PromoteSnapshot" + Atelet_Terminate_FullMethodName = "/atelet.Atelet/Terminate" ) -// AteomHerderClient is the client API for AteomHerder service. +// AteletClient is the client API for Atelet service. // // For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream. // -// AteomHerder allows ate-apiserver to issue control calls to the atelet. +// Atelet allows ate-apiserver to issue control calls to the atelet. // // Called by ate-api-server over cluster networking. ate-api-server // authenticates with its k8s pod identity mTLS certificate. -type AteomHerderClient interface { - // Run tells atelet to create a new containerized workload from scratch on an - // ateom. +type AteletClient interface { + // Run tells atelet to create a new containerized workload from scratch on a + // worker. Run(ctx context.Context, in *RunRequest, opts ...grpc.CallOption) (*RunResponse, error) // Checkpoint saves the workload's state as a new snapshot of the actor and - // returns the ateom to a blank state. upload_config says when the snapshot - // becomes durable. Once ateom has captured the workload, and before the new + // returns the worker to a blank state. upload_config says when the snapshot + // becomes durable. Once the worker has captured the workload, and before the new // snapshot is uploaded or stored locally, atelet deletes the actor's older // local snapshots, except one still being uploaded, which it deletes when // that upload ends. A Checkpoint that then fails to upload or store the new // snapshot leaves the actor with no local snapshot. Checkpoint(ctx context.Context, in *CheckpointRequest, opts ...grpc.CallOption) (*CheckpointResponse, error) - // Restore restores a workload from a snapshot onto an ateom. + // Restore restores a workload from a snapshot onto a worker. Restore(ctx context.Context, in *RestoreRequest, opts ...grpc.CallOption) (*RestoreResponse, error) // PromoteSnapshot makes `snapshot` durable in object storage while keeping // its local copy on this node. Used when a snapshot needs to be in object @@ -344,85 +344,85 @@ type AteomHerderClient interface { Terminate(ctx context.Context, in *TerminateRequest, opts ...grpc.CallOption) (*TerminateResponse, error) } -type ateomHerderClient struct { +type ateletClient struct { cc grpc.ClientConnInterface } -func NewAteomHerderClient(cc grpc.ClientConnInterface) AteomHerderClient { - return &ateomHerderClient{cc} +func NewAteletClient(cc grpc.ClientConnInterface) AteletClient { + return &ateletClient{cc} } -func (c *ateomHerderClient) Run(ctx context.Context, in *RunRequest, opts ...grpc.CallOption) (*RunResponse, error) { +func (c *ateletClient) Run(ctx context.Context, in *RunRequest, opts ...grpc.CallOption) (*RunResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(RunResponse) - err := c.cc.Invoke(ctx, AteomHerder_Run_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Atelet_Run_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomHerderClient) Checkpoint(ctx context.Context, in *CheckpointRequest, opts ...grpc.CallOption) (*CheckpointResponse, error) { +func (c *ateletClient) Checkpoint(ctx context.Context, in *CheckpointRequest, opts ...grpc.CallOption) (*CheckpointResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(CheckpointResponse) - err := c.cc.Invoke(ctx, AteomHerder_Checkpoint_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Atelet_Checkpoint_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomHerderClient) Restore(ctx context.Context, in *RestoreRequest, opts ...grpc.CallOption) (*RestoreResponse, error) { +func (c *ateletClient) Restore(ctx context.Context, in *RestoreRequest, opts ...grpc.CallOption) (*RestoreResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(RestoreResponse) - err := c.cc.Invoke(ctx, AteomHerder_Restore_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Atelet_Restore_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomHerderClient) PromoteSnapshot(ctx context.Context, in *PromoteSnapshotRequest, opts ...grpc.CallOption) (*PromoteSnapshotResponse, error) { +func (c *ateletClient) PromoteSnapshot(ctx context.Context, in *PromoteSnapshotRequest, opts ...grpc.CallOption) (*PromoteSnapshotResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(PromoteSnapshotResponse) - err := c.cc.Invoke(ctx, AteomHerder_PromoteSnapshot_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Atelet_PromoteSnapshot_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomHerderClient) Terminate(ctx context.Context, in *TerminateRequest, opts ...grpc.CallOption) (*TerminateResponse, error) { +func (c *ateletClient) Terminate(ctx context.Context, in *TerminateRequest, opts ...grpc.CallOption) (*TerminateResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(TerminateResponse) - err := c.cc.Invoke(ctx, AteomHerder_Terminate_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Atelet_Terminate_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -// AteomHerderServer is the server API for AteomHerder service. -// All implementations must embed UnimplementedAteomHerderServer +// AteletServer is the server API for Atelet service. +// All implementations must embed UnimplementedAteletServer // for forward compatibility. // -// AteomHerder allows ate-apiserver to issue control calls to the atelet. +// Atelet allows ate-apiserver to issue control calls to the atelet. // // Called by ate-api-server over cluster networking. ate-api-server // authenticates with its k8s pod identity mTLS certificate. -type AteomHerderServer interface { - // Run tells atelet to create a new containerized workload from scratch on an - // ateom. +type AteletServer interface { + // Run tells atelet to create a new containerized workload from scratch on a + // worker. Run(context.Context, *RunRequest) (*RunResponse, error) // Checkpoint saves the workload's state as a new snapshot of the actor and - // returns the ateom to a blank state. upload_config says when the snapshot - // becomes durable. Once ateom has captured the workload, and before the new + // returns the worker to a blank state. upload_config says when the snapshot + // becomes durable. Once the worker has captured the workload, and before the new // snapshot is uploaded or stored locally, atelet deletes the actor's older // local snapshots, except one still being uploaded, which it deletes when // that upload ends. A Checkpoint that then fails to upload or store the new // snapshot leaves the actor with no local snapshot. Checkpoint(context.Context, *CheckpointRequest) (*CheckpointResponse, error) - // Restore restores a workload from a snapshot onto an ateom. + // Restore restores a workload from a snapshot onto a worker. Restore(context.Context, *RestoreRequest) (*RestoreResponse, error) // PromoteSnapshot makes `snapshot` durable in object storage while keeping // its local copy on this node. Used when a snapshot needs to be in object @@ -440,168 +440,168 @@ type AteomHerderServer interface { // Terminate tells atelet to terminate/kill any running workload for an actor, // unmount its volumes, and clean up actor state on the node. Terminate(context.Context, *TerminateRequest) (*TerminateResponse, error) - mustEmbedUnimplementedAteomHerderServer() + mustEmbedUnimplementedAteletServer() } -// UnimplementedAteomHerderServer must be embedded to have +// UnimplementedAteletServer must be embedded to have // forward compatible implementations. // // NOTE: this should be embedded by value instead of pointer to avoid a nil // pointer dereference when methods are called. -type UnimplementedAteomHerderServer struct{} +type UnimplementedAteletServer struct{} -func (UnimplementedAteomHerderServer) Run(context.Context, *RunRequest) (*RunResponse, error) { +func (UnimplementedAteletServer) Run(context.Context, *RunRequest) (*RunResponse, error) { return nil, status.Error(codes.Unimplemented, "method Run not implemented") } -func (UnimplementedAteomHerderServer) Checkpoint(context.Context, *CheckpointRequest) (*CheckpointResponse, error) { +func (UnimplementedAteletServer) Checkpoint(context.Context, *CheckpointRequest) (*CheckpointResponse, error) { return nil, status.Error(codes.Unimplemented, "method Checkpoint not implemented") } -func (UnimplementedAteomHerderServer) Restore(context.Context, *RestoreRequest) (*RestoreResponse, error) { +func (UnimplementedAteletServer) Restore(context.Context, *RestoreRequest) (*RestoreResponse, error) { return nil, status.Error(codes.Unimplemented, "method Restore not implemented") } -func (UnimplementedAteomHerderServer) PromoteSnapshot(context.Context, *PromoteSnapshotRequest) (*PromoteSnapshotResponse, error) { +func (UnimplementedAteletServer) PromoteSnapshot(context.Context, *PromoteSnapshotRequest) (*PromoteSnapshotResponse, error) { return nil, status.Error(codes.Unimplemented, "method PromoteSnapshot not implemented") } -func (UnimplementedAteomHerderServer) Terminate(context.Context, *TerminateRequest) (*TerminateResponse, error) { +func (UnimplementedAteletServer) Terminate(context.Context, *TerminateRequest) (*TerminateResponse, error) { return nil, status.Error(codes.Unimplemented, "method Terminate not implemented") } -func (UnimplementedAteomHerderServer) mustEmbedUnimplementedAteomHerderServer() {} -func (UnimplementedAteomHerderServer) testEmbeddedByValue() {} +func (UnimplementedAteletServer) mustEmbedUnimplementedAteletServer() {} +func (UnimplementedAteletServer) testEmbeddedByValue() {} -// UnsafeAteomHerderServer may be embedded to opt out of forward compatibility for this service. -// Use of this interface is not recommended, as added methods to AteomHerderServer will +// UnsafeAteletServer may be embedded to opt out of forward compatibility for this service. +// Use of this interface is not recommended, as added methods to AteletServer will // result in compilation errors. -type UnsafeAteomHerderServer interface { - mustEmbedUnimplementedAteomHerderServer() +type UnsafeAteletServer interface { + mustEmbedUnimplementedAteletServer() } -func RegisterAteomHerderServer(s grpc.ServiceRegistrar, srv AteomHerderServer) { - // If the following call panics, it indicates UnimplementedAteomHerderServer was +func RegisterAteletServer(s grpc.ServiceRegistrar, srv AteletServer) { + // If the following call panics, it indicates UnimplementedAteletServer was // embedded by pointer and is nil. This will cause panics if an // unimplemented method is ever invoked, so we test this at initialization // time to prevent it from happening at runtime later due to I/O. if t, ok := srv.(interface{ testEmbeddedByValue() }); ok { t.testEmbeddedByValue() } - s.RegisterService(&AteomHerder_ServiceDesc, srv) + s.RegisterService(&Atelet_ServiceDesc, srv) } -func _AteomHerder_Run_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Atelet_Run_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(RunRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomHerderServer).Run(ctx, in) + return srv.(AteletServer).Run(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomHerder_Run_FullMethodName, + FullMethod: Atelet_Run_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomHerderServer).Run(ctx, req.(*RunRequest)) + return srv.(AteletServer).Run(ctx, req.(*RunRequest)) } return interceptor(ctx, in, info, handler) } -func _AteomHerder_Checkpoint_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Atelet_Checkpoint_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(CheckpointRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomHerderServer).Checkpoint(ctx, in) + return srv.(AteletServer).Checkpoint(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomHerder_Checkpoint_FullMethodName, + FullMethod: Atelet_Checkpoint_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomHerderServer).Checkpoint(ctx, req.(*CheckpointRequest)) + return srv.(AteletServer).Checkpoint(ctx, req.(*CheckpointRequest)) } return interceptor(ctx, in, info, handler) } -func _AteomHerder_Restore_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Atelet_Restore_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(RestoreRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomHerderServer).Restore(ctx, in) + return srv.(AteletServer).Restore(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomHerder_Restore_FullMethodName, + FullMethod: Atelet_Restore_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomHerderServer).Restore(ctx, req.(*RestoreRequest)) + return srv.(AteletServer).Restore(ctx, req.(*RestoreRequest)) } return interceptor(ctx, in, info, handler) } -func _AteomHerder_PromoteSnapshot_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Atelet_PromoteSnapshot_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(PromoteSnapshotRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomHerderServer).PromoteSnapshot(ctx, in) + return srv.(AteletServer).PromoteSnapshot(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomHerder_PromoteSnapshot_FullMethodName, + FullMethod: Atelet_PromoteSnapshot_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomHerderServer).PromoteSnapshot(ctx, req.(*PromoteSnapshotRequest)) + return srv.(AteletServer).PromoteSnapshot(ctx, req.(*PromoteSnapshotRequest)) } return interceptor(ctx, in, info, handler) } -func _AteomHerder_Terminate_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Atelet_Terminate_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(TerminateRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomHerderServer).Terminate(ctx, in) + return srv.(AteletServer).Terminate(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: AteomHerder_Terminate_FullMethodName, + FullMethod: Atelet_Terminate_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomHerderServer).Terminate(ctx, req.(*TerminateRequest)) + return srv.(AteletServer).Terminate(ctx, req.(*TerminateRequest)) } return interceptor(ctx, in, info, handler) } -// AteomHerder_ServiceDesc is the grpc.ServiceDesc for AteomHerder service. +// Atelet_ServiceDesc is the grpc.ServiceDesc for Atelet service. // It's only intended for direct use with grpc.RegisterService, // and not to be introspected or modified (even as a copy) -var AteomHerder_ServiceDesc = grpc.ServiceDesc{ - ServiceName: "atelet.AteomHerder", - HandlerType: (*AteomHerderServer)(nil), +var Atelet_ServiceDesc = grpc.ServiceDesc{ + ServiceName: "atelet.Atelet", + HandlerType: (*AteletServer)(nil), Methods: []grpc.MethodDesc{ { MethodName: "Run", - Handler: _AteomHerder_Run_Handler, + Handler: _Atelet_Run_Handler, }, { MethodName: "Checkpoint", - Handler: _AteomHerder_Checkpoint_Handler, + Handler: _Atelet_Checkpoint_Handler, }, { MethodName: "Restore", - Handler: _AteomHerder_Restore_Handler, + Handler: _Atelet_Restore_Handler, }, { MethodName: "PromoteSnapshot", - Handler: _AteomHerder_PromoteSnapshot_Handler, + Handler: _Atelet_PromoteSnapshot_Handler, }, { MethodName: "Terminate", - Handler: _AteomHerder_Terminate_Handler, + Handler: _Atelet_Terminate_Handler, }, }, Streams: []grpc.StreamDesc{}, diff --git a/internal/proto/ateompb/ateom.pb.go b/internal/proto/ateworkerpb/ateworker.pb.go similarity index 79% rename from internal/proto/ateompb/ateom.pb.go rename to internal/proto/ateworkerpb/ateworker.pb.go index 03eb7137d9..ee23673a67 100644 --- a/internal/proto/ateompb/ateom.pb.go +++ b/internal/proto/ateworkerpb/ateworker.pb.go @@ -16,9 +16,9 @@ // versions: // protoc-gen-go v1.36.11-devel // protoc v4.25.3 -// source: ateom.proto +// source: ateworker.proto -package ateompb +package ateworkerpb import ( protoreflect "google.golang.org/protobuf/reflect/protoreflect" @@ -81,11 +81,11 @@ func (x SnapshotFidelity) String() string { } func (SnapshotFidelity) Descriptor() protoreflect.EnumDescriptor { - return file_ateom_proto_enumTypes[0].Descriptor() + return file_ateworker_proto_enumTypes[0].Descriptor() } func (SnapshotFidelity) Type() protoreflect.EnumType { - return &file_ateom_proto_enumTypes[0] + return &file_ateworker_proto_enumTypes[0] } func (x SnapshotFidelity) Number() protoreflect.EnumNumber { @@ -94,7 +94,7 @@ func (x SnapshotFidelity) Number() protoreflect.EnumNumber { // Deprecated: Use SnapshotFidelity.Descriptor instead. func (SnapshotFidelity) EnumDescriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{0} + return file_ateworker_proto_rawDescGZIP(), []int{0} } // StatsSource is how a sample's measurements were obtained. The two are not @@ -141,11 +141,11 @@ func (x StatsSource) String() string { } func (StatsSource) Descriptor() protoreflect.EnumDescriptor { - return file_ateom_proto_enumTypes[1].Descriptor() + return file_ateworker_proto_enumTypes[1].Descriptor() } func (StatsSource) Type() protoreflect.EnumType { - return &file_ateom_proto_enumTypes[1] + return &file_ateworker_proto_enumTypes[1] } func (x StatsSource) Number() protoreflect.EnumNumber { @@ -154,11 +154,11 @@ func (x StatsSource) Number() protoreflect.EnumNumber { // Deprecated: Use StatsSource.Descriptor instead. func (StatsSource) EnumDescriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{1} + return file_ateworker_proto_rawDescGZIP(), []int{1} } // ActorDirs is the set of per-actor directories atelet prepares on the volume -// it shares with ateom. ateom takes them from the RPC instead of deriving +// it shares with the worker. The worker takes them from the RPC instead of deriving // them from the actor UID. // Every directory is under root_dir. type ActorDirs struct { @@ -183,7 +183,7 @@ type ActorDirs struct { func (x *ActorDirs) Reset() { *x = ActorDirs{} - mi := &file_ateom_proto_msgTypes[0] + mi := &file_ateworker_proto_msgTypes[0] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -195,7 +195,7 @@ func (x *ActorDirs) String() string { func (*ActorDirs) ProtoMessage() {} func (x *ActorDirs) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[0] + mi := &file_ateworker_proto_msgTypes[0] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -208,7 +208,7 @@ func (x *ActorDirs) ProtoReflect() protoreflect.Message { // Deprecated: Use ActorDirs.ProtoReflect.Descriptor instead. func (*ActorDirs) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{0} + return file_ateworker_proto_rawDescGZIP(), []int{0} } func (x *ActorDirs) GetRootDir() string { @@ -276,7 +276,7 @@ type TerminateWorkloadRequest struct { func (x *TerminateWorkloadRequest) Reset() { *x = TerminateWorkloadRequest{} - mi := &file_ateom_proto_msgTypes[1] + mi := &file_ateworker_proto_msgTypes[1] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -288,7 +288,7 @@ func (x *TerminateWorkloadRequest) String() string { func (*TerminateWorkloadRequest) ProtoMessage() {} func (x *TerminateWorkloadRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[1] + mi := &file_ateworker_proto_msgTypes[1] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -301,7 +301,7 @@ func (x *TerminateWorkloadRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use TerminateWorkloadRequest.ProtoReflect.Descriptor instead. func (*TerminateWorkloadRequest) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{1} + return file_ateworker_proto_rawDescGZIP(), []int{1} } func (x *TerminateWorkloadRequest) GetAtespace() string { @@ -368,7 +368,7 @@ type TerminateWorkloadResponse struct { func (x *TerminateWorkloadResponse) Reset() { *x = TerminateWorkloadResponse{} - mi := &file_ateom_proto_msgTypes[2] + mi := &file_ateworker_proto_msgTypes[2] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -380,7 +380,7 @@ func (x *TerminateWorkloadResponse) String() string { func (*TerminateWorkloadResponse) ProtoMessage() {} func (x *TerminateWorkloadResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[2] + mi := &file_ateworker_proto_msgTypes[2] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -393,7 +393,7 @@ func (x *TerminateWorkloadResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use TerminateWorkloadResponse.ProtoReflect.Descriptor instead. func (*TerminateWorkloadResponse) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{2} + return file_ateworker_proto_rawDescGZIP(), []int{2} } type RunWorkloadRequest struct { @@ -412,10 +412,10 @@ type RunWorkloadRequest struct { RuntimeAssetPaths map[string]string `protobuf:"bytes,8,rep,name=runtime_asset_paths,json=runtimeAssetPaths,proto3" json:"runtime_asset_paths,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // When absent the actor has no egress: its TCP is captured and refused. EgressGateway *EgressGateway `protobuf:"bytes,10,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"` - // The actor's declared size, from the ActorTemplate's resource limits. ateom + // The actor's declared size, from the ActorTemplate's resource limits. The worker // sizes the sandbox to these (cgroup caps via the OCI spec, and for the // micro-VM the VM's vCPU count and memory). Zero means "unset": keep the - // runtime default (unlimited for gVisor, ateom's own default for the micro-VM). + // runtime default (unlimited for gVisor, the worker's own default for the micro-VM). CpuMilli int64 `protobuf:"varint,11,opt,name=cpu_milli,json=cpuMilli,proto3" json:"cpu_milli,omitempty"` // CPU limit in millicores (1000 = one core). MemoryBytes int64 `protobuf:"varint,12,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes. ActorDirs *ActorDirs `protobuf:"bytes,13,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"` @@ -425,7 +425,7 @@ type RunWorkloadRequest struct { func (x *RunWorkloadRequest) Reset() { *x = RunWorkloadRequest{} - mi := &file_ateom_proto_msgTypes[3] + mi := &file_ateworker_proto_msgTypes[3] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -437,7 +437,7 @@ func (x *RunWorkloadRequest) String() string { func (*RunWorkloadRequest) ProtoMessage() {} func (x *RunWorkloadRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[3] + mi := &file_ateworker_proto_msgTypes[3] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -450,7 +450,7 @@ func (x *RunWorkloadRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RunWorkloadRequest.ProtoReflect.Descriptor instead. func (*RunWorkloadRequest) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{3} + return file_ateworker_proto_rawDescGZIP(), []int{3} } func (x *RunWorkloadRequest) GetAtespace() string { @@ -549,7 +549,7 @@ type EgressGateway struct { func (x *EgressGateway) Reset() { *x = EgressGateway{} - mi := &file_ateom_proto_msgTypes[4] + mi := &file_ateworker_proto_msgTypes[4] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -561,7 +561,7 @@ func (x *EgressGateway) String() string { func (*EgressGateway) ProtoMessage() {} func (x *EgressGateway) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[4] + mi := &file_ateworker_proto_msgTypes[4] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -574,7 +574,7 @@ func (x *EgressGateway) ProtoReflect() protoreflect.Message { // Deprecated: Use EgressGateway.ProtoReflect.Descriptor instead. func (*EgressGateway) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{4} + return file_ateworker_proto_rawDescGZIP(), []int{4} } func (x *EgressGateway) GetAddress() string { @@ -594,7 +594,7 @@ type WorkloadSpec struct { func (x *WorkloadSpec) Reset() { *x = WorkloadSpec{} - mi := &file_ateom_proto_msgTypes[5] + mi := &file_ateworker_proto_msgTypes[5] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -606,7 +606,7 @@ func (x *WorkloadSpec) String() string { func (*WorkloadSpec) ProtoMessage() {} func (x *WorkloadSpec) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[5] + mi := &file_ateworker_proto_msgTypes[5] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -619,7 +619,7 @@ func (x *WorkloadSpec) ProtoReflect() protoreflect.Message { // Deprecated: Use WorkloadSpec.ProtoReflect.Descriptor instead. func (*WorkloadSpec) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{5} + return file_ateworker_proto_rawDescGZIP(), []int{5} } func (x *WorkloadSpec) GetContainers() []*Container { @@ -644,7 +644,7 @@ type Container struct { SystemInfoVolumeMounts []*SystemInfoVolumeMount `protobuf:"bytes,5,rep,name=system_info_volume_mounts,json=systemInfoVolumeMounts,proto3" json:"system_info_volume_mounts,omitempty"` // image_volume_mounts are the image volumes this container mounts, if any. ImageVolumeMounts []*ImageVolumeMount `protobuf:"bytes,6,rep,name=image_volume_mounts,json=imageVolumeMounts,proto3" json:"image_volume_mounts,omitempty"` - // container_spec is what ateom builds the container's OCI spec from, + // container_spec is what the worker builds the container's OCI spec from, // besides its volume mounts. Unset on RPCs that start no process. ContainerSpec *ContainerSpec `protobuf:"bytes,7,opt,name=container_spec,json=containerSpec,proto3" json:"container_spec,omitempty"` unknownFields protoimpl.UnknownFields @@ -653,7 +653,7 @@ type Container struct { func (x *Container) Reset() { *x = Container{} - mi := &file_ateom_proto_msgTypes[6] + mi := &file_ateworker_proto_msgTypes[6] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -665,7 +665,7 @@ func (x *Container) String() string { func (*Container) ProtoMessage() {} func (x *Container) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[6] + mi := &file_ateworker_proto_msgTypes[6] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -678,7 +678,7 @@ func (x *Container) ProtoReflect() protoreflect.Message { // Deprecated: Use Container.ProtoReflect.Descriptor instead. func (*Container) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{6} + return file_ateworker_proto_rawDescGZIP(), []int{6} } func (x *Container) GetName() string { @@ -749,7 +749,7 @@ type ContainerSpec struct { func (x *ContainerSpec) Reset() { *x = ContainerSpec{} - mi := &file_ateom_proto_msgTypes[7] + mi := &file_ateworker_proto_msgTypes[7] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -761,7 +761,7 @@ func (x *ContainerSpec) String() string { func (*ContainerSpec) ProtoMessage() {} func (x *ContainerSpec) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[7] + mi := &file_ateworker_proto_msgTypes[7] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -774,7 +774,7 @@ func (x *ContainerSpec) ProtoReflect() protoreflect.Message { // Deprecated: Use ContainerSpec.ProtoReflect.Descriptor instead. func (*ContainerSpec) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{7} + return file_ateworker_proto_rawDescGZIP(), []int{7} } func (x *ContainerSpec) GetArgs() []string { @@ -818,7 +818,7 @@ type ResourceLimits struct { func (x *ResourceLimits) Reset() { *x = ResourceLimits{} - mi := &file_ateom_proto_msgTypes[8] + mi := &file_ateworker_proto_msgTypes[8] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -830,7 +830,7 @@ func (x *ResourceLimits) String() string { func (*ResourceLimits) ProtoMessage() {} func (x *ResourceLimits) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[8] + mi := &file_ateworker_proto_msgTypes[8] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -843,7 +843,7 @@ func (x *ResourceLimits) ProtoReflect() protoreflect.Message { // Deprecated: Use ResourceLimits.ProtoReflect.Descriptor instead. func (*ResourceLimits) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{8} + return file_ateworker_proto_rawDescGZIP(), []int{8} } func (x *ResourceLimits) GetMemoryBytes() int64 { @@ -871,7 +871,7 @@ type VolumeMount struct { func (x *VolumeMount) Reset() { *x = VolumeMount{} - mi := &file_ateom_proto_msgTypes[9] + mi := &file_ateworker_proto_msgTypes[9] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -883,7 +883,7 @@ func (x *VolumeMount) String() string { func (*VolumeMount) ProtoMessage() {} func (x *VolumeMount) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[9] + mi := &file_ateworker_proto_msgTypes[9] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -896,7 +896,7 @@ func (x *VolumeMount) ProtoReflect() protoreflect.Message { // Deprecated: Use VolumeMount.ProtoReflect.Descriptor instead. func (*VolumeMount) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{9} + return file_ateworker_proto_rawDescGZIP(), []int{9} } func (x *VolumeMount) GetVolumeName() string { @@ -927,7 +927,7 @@ type DurableDirVolumeMount struct { func (x *DurableDirVolumeMount) Reset() { *x = DurableDirVolumeMount{} - mi := &file_ateom_proto_msgTypes[10] + mi := &file_ateworker_proto_msgTypes[10] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -939,7 +939,7 @@ func (x *DurableDirVolumeMount) String() string { func (*DurableDirVolumeMount) ProtoMessage() {} func (x *DurableDirVolumeMount) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[10] + mi := &file_ateworker_proto_msgTypes[10] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -952,7 +952,7 @@ func (x *DurableDirVolumeMount) ProtoReflect() protoreflect.Message { // Deprecated: Use DurableDirVolumeMount.ProtoReflect.Descriptor instead. func (*DurableDirVolumeMount) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{10} + return file_ateworker_proto_rawDescGZIP(), []int{10} } func (x *DurableDirVolumeMount) GetVolumeName() string { @@ -985,7 +985,7 @@ type SystemInfoVolumeMount struct { func (x *SystemInfoVolumeMount) Reset() { *x = SystemInfoVolumeMount{} - mi := &file_ateom_proto_msgTypes[11] + mi := &file_ateworker_proto_msgTypes[11] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -997,7 +997,7 @@ func (x *SystemInfoVolumeMount) String() string { func (*SystemInfoVolumeMount) ProtoMessage() {} func (x *SystemInfoVolumeMount) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[11] + mi := &file_ateworker_proto_msgTypes[11] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1010,7 +1010,7 @@ func (x *SystemInfoVolumeMount) ProtoReflect() protoreflect.Message { // Deprecated: Use SystemInfoVolumeMount.ProtoReflect.Descriptor instead. func (*SystemInfoVolumeMount) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{11} + return file_ateworker_proto_rawDescGZIP(), []int{11} } func (x *SystemInfoVolumeMount) GetVolumeName() string { @@ -1027,7 +1027,7 @@ func (x *SystemInfoVolumeMount) GetMountPath() string { return "" } -// ImageVolumeMount is one image volume mounted into a container. ateom uses +// ImageVolumeMount is one image volume mounted into a container. The worker uses // these to construct the container's volume mounts — each names the volume // and its destination path. type ImageVolumeMount struct { @@ -1042,7 +1042,7 @@ type ImageVolumeMount struct { func (x *ImageVolumeMount) Reset() { *x = ImageVolumeMount{} - mi := &file_ateom_proto_msgTypes[12] + mi := &file_ateworker_proto_msgTypes[12] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1054,7 +1054,7 @@ func (x *ImageVolumeMount) String() string { func (*ImageVolumeMount) ProtoMessage() {} func (x *ImageVolumeMount) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[12] + mi := &file_ateworker_proto_msgTypes[12] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1067,7 +1067,7 @@ func (x *ImageVolumeMount) ProtoReflect() protoreflect.Message { // Deprecated: Use ImageVolumeMount.ProtoReflect.Descriptor instead. func (*ImageVolumeMount) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{12} + return file_ateworker_proto_rawDescGZIP(), []int{12} } func (x *ImageVolumeMount) GetVolumeName() string { @@ -1097,7 +1097,7 @@ type WakeupProbe struct { func (x *WakeupProbe) Reset() { *x = WakeupProbe{} - mi := &file_ateom_proto_msgTypes[13] + mi := &file_ateworker_proto_msgTypes[13] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1109,7 +1109,7 @@ func (x *WakeupProbe) String() string { func (*WakeupProbe) ProtoMessage() {} func (x *WakeupProbe) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[13] + mi := &file_ateworker_proto_msgTypes[13] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1122,7 +1122,7 @@ func (x *WakeupProbe) ProtoReflect() protoreflect.Message { // Deprecated: Use WakeupProbe.ProtoReflect.Descriptor instead. func (*WakeupProbe) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{13} + return file_ateworker_proto_rawDescGZIP(), []int{13} } func (x *WakeupProbe) GetHttpGet() *HttpGetAction { @@ -1152,7 +1152,7 @@ type HttpGetAction struct { func (x *HttpGetAction) Reset() { *x = HttpGetAction{} - mi := &file_ateom_proto_msgTypes[14] + mi := &file_ateworker_proto_msgTypes[14] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1164,7 +1164,7 @@ func (x *HttpGetAction) String() string { func (*HttpGetAction) ProtoMessage() {} func (x *HttpGetAction) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[14] + mi := &file_ateworker_proto_msgTypes[14] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1177,7 +1177,7 @@ func (x *HttpGetAction) ProtoReflect() protoreflect.Message { // Deprecated: Use HttpGetAction.ProtoReflect.Descriptor instead. func (*HttpGetAction) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{14} + return file_ateworker_proto_rawDescGZIP(), []int{14} } func (x *HttpGetAction) GetPath() string { @@ -1202,7 +1202,7 @@ type RunWorkloadResponse struct { func (x *RunWorkloadResponse) Reset() { *x = RunWorkloadResponse{} - mi := &file_ateom_proto_msgTypes[15] + mi := &file_ateworker_proto_msgTypes[15] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1214,7 +1214,7 @@ func (x *RunWorkloadResponse) String() string { func (*RunWorkloadResponse) ProtoMessage() {} func (x *RunWorkloadResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[15] + mi := &file_ateworker_proto_msgTypes[15] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1227,7 +1227,7 @@ func (x *RunWorkloadResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RunWorkloadResponse.ProtoReflect.Descriptor instead. func (*RunWorkloadResponse) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{15} + return file_ateworker_proto_rawDescGZIP(), []int{15} } type CheckpointWorkloadRequest struct { @@ -1254,7 +1254,7 @@ type CheckpointWorkloadRequest struct { // atelet fetched it to (see RunWorkloadRequest). Empty for gVisor. RuntimeAssetPaths map[string]string `protobuf:"bytes,9,rep,name=runtime_asset_paths,json=runtimeAssetPaths,proto3" json:"runtime_asset_paths,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // Fidelity the checkpoint captures. - Fidelity SnapshotFidelity `protobuf:"varint,10,opt,name=fidelity,proto3,enum=ateom.SnapshotFidelity" json:"fidelity,omitempty"` + Fidelity SnapshotFidelity `protobuf:"varint,10,opt,name=fidelity,proto3,enum=ateworker.SnapshotFidelity" json:"fidelity,omitempty"` ActorDirs *ActorDirs `protobuf:"bytes,11,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -1262,7 +1262,7 @@ type CheckpointWorkloadRequest struct { func (x *CheckpointWorkloadRequest) Reset() { *x = CheckpointWorkloadRequest{} - mi := &file_ateom_proto_msgTypes[16] + mi := &file_ateworker_proto_msgTypes[16] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1274,7 +1274,7 @@ func (x *CheckpointWorkloadRequest) String() string { func (*CheckpointWorkloadRequest) ProtoMessage() {} func (x *CheckpointWorkloadRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[16] + mi := &file_ateworker_proto_msgTypes[16] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1287,7 +1287,7 @@ func (x *CheckpointWorkloadRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use CheckpointWorkloadRequest.ProtoReflect.Descriptor instead. func (*CheckpointWorkloadRequest) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{16} + return file_ateworker_proto_rawDescGZIP(), []int{16} } func (x *CheckpointWorkloadRequest) GetAtespace() string { @@ -1369,7 +1369,7 @@ func (x *CheckpointWorkloadRequest) GetActorDirs() *ActorDirs { type CheckpointWorkloadResponse struct { state protoimpl.MessageState `protogen:"open.v1"` - // snapshot_files lists the files ateom wrote into the checkpoint directory + // snapshot_files lists the files the worker wrote into the checkpoint directory // (relative names) for atelet to ship to object storage. Each runtime reports // its own set (gVisor's image files, cloud-hypervisor's snapshot set, ...). SnapshotFiles []string `protobuf:"bytes,1,rep,name=snapshot_files,json=snapshotFiles,proto3" json:"snapshot_files,omitempty"` @@ -1382,7 +1382,7 @@ type CheckpointWorkloadResponse struct { func (x *CheckpointWorkloadResponse) Reset() { *x = CheckpointWorkloadResponse{} - mi := &file_ateom_proto_msgTypes[17] + mi := &file_ateworker_proto_msgTypes[17] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1394,7 +1394,7 @@ func (x *CheckpointWorkloadResponse) String() string { func (*CheckpointWorkloadResponse) ProtoMessage() {} func (x *CheckpointWorkloadResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[17] + mi := &file_ateworker_proto_msgTypes[17] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1407,7 +1407,7 @@ func (x *CheckpointWorkloadResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use CheckpointWorkloadResponse.ProtoReflect.Descriptor instead. func (*CheckpointWorkloadResponse) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{17} + return file_ateworker_proto_rawDescGZIP(), []int{17} } func (x *CheckpointWorkloadResponse) GetSnapshotFiles() []string { @@ -1440,7 +1440,7 @@ type RestoreWorkloadRequest struct { // atelet fetched it to (see RunWorkloadRequest). Empty for gVisor. RuntimeAssetPaths map[string]string `protobuf:"bytes,9,rep,name=runtime_asset_paths,json=runtimeAssetPaths,proto3" json:"runtime_asset_paths,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` // Fidelity to restore from the snapshot. - Fidelity SnapshotFidelity `protobuf:"varint,10,opt,name=fidelity,proto3,enum=ateom.SnapshotFidelity" json:"fidelity,omitempty"` + Fidelity SnapshotFidelity `protobuf:"varint,10,opt,name=fidelity,proto3,enum=ateworker.SnapshotFidelity" json:"fidelity,omitempty"` // When absent the actor has no egress: its TCP is captured and refused. EgressGateway *EgressGateway `protobuf:"bytes,12,opt,name=egress_gateway,json=egressGateway,proto3,oneof" json:"egress_gateway,omitempty"` // The actor's declared size, from the ActorTemplate's resource limits. Used to @@ -1451,7 +1451,7 @@ type RestoreWorkloadRequest struct { MemoryBytes int64 `protobuf:"varint,15,opt,name=memory_bytes,json=memoryBytes,proto3" json:"memory_bytes,omitempty"` // Memory limit in bytes. ActorDirs *ActorDirs `protobuf:"bytes,16,opt,name=actor_dirs,json=actorDirs,proto3" json:"actor_dirs,omitempty"` // actor_dirs.restore_dir is a preserved local snapshot, not a scratch copy: - // ateom must not delete or modify files in it. + // the worker must not delete or modify files in it. PreserveRestoreDir bool `protobuf:"varint,17,opt,name=preserve_restore_dir,json=preserveRestoreDir,proto3" json:"preserve_restore_dir,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -1459,7 +1459,7 @@ type RestoreWorkloadRequest struct { func (x *RestoreWorkloadRequest) Reset() { *x = RestoreWorkloadRequest{} - mi := &file_ateom_proto_msgTypes[18] + mi := &file_ateworker_proto_msgTypes[18] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1471,7 +1471,7 @@ func (x *RestoreWorkloadRequest) String() string { func (*RestoreWorkloadRequest) ProtoMessage() {} func (x *RestoreWorkloadRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[18] + mi := &file_ateworker_proto_msgTypes[18] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1484,7 +1484,7 @@ func (x *RestoreWorkloadRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use RestoreWorkloadRequest.ProtoReflect.Descriptor instead. func (*RestoreWorkloadRequest) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{18} + return file_ateworker_proto_rawDescGZIP(), []int{18} } func (x *RestoreWorkloadRequest) GetAtespace() string { @@ -1600,7 +1600,7 @@ type RestoreWorkloadResponse struct { func (x *RestoreWorkloadResponse) Reset() { *x = RestoreWorkloadResponse{} - mi := &file_ateom_proto_msgTypes[19] + mi := &file_ateworker_proto_msgTypes[19] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1612,7 +1612,7 @@ func (x *RestoreWorkloadResponse) String() string { func (*RestoreWorkloadResponse) ProtoMessage() {} func (x *RestoreWorkloadResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[19] + mi := &file_ateworker_proto_msgTypes[19] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1625,13 +1625,13 @@ func (x *RestoreWorkloadResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use RestoreWorkloadResponse.ProtoReflect.Descriptor instead. func (*RestoreWorkloadResponse) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{19} + return file_ateworker_proto_rawDescGZIP(), []int{19} } type GetWorkloadStatsRequest struct { state protoimpl.MessageState `protogen:"open.v1"` // The actor the caller believes is executing here. A worker can be recycled - // between the caller's view of the world and this call, so ateom answers + // between the caller's view of the world and this call, so the worker answers // NOT_FOUND on a mismatch rather than reporting a different actor's numbers // under the requested actor's identity. ActorUid string `protobuf:"bytes,1,opt,name=actor_uid,json=actorUid,proto3" json:"actor_uid,omitempty"` @@ -1641,7 +1641,7 @@ type GetWorkloadStatsRequest struct { func (x *GetWorkloadStatsRequest) Reset() { *x = GetWorkloadStatsRequest{} - mi := &file_ateom_proto_msgTypes[20] + mi := &file_ateworker_proto_msgTypes[20] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1653,7 +1653,7 @@ func (x *GetWorkloadStatsRequest) String() string { func (*GetWorkloadStatsRequest) ProtoMessage() {} func (x *GetWorkloadStatsRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[20] + mi := &file_ateworker_proto_msgTypes[20] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1666,7 +1666,7 @@ func (x *GetWorkloadStatsRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetWorkloadStatsRequest.ProtoReflect.Descriptor instead. func (*GetWorkloadStatsRequest) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{20} + return file_ateworker_proto_rawDescGZIP(), []int{20} } func (x *GetWorkloadStatsRequest) GetActorUid() string { @@ -1680,12 +1680,12 @@ func (x *GetWorkloadStatsRequest) GetActorUid() string { // shared by both stats reads. The unit of measurement is the SANDBOX, which // today equals the actor. Per-container attribution is not reported: the // micro-VM source could give it, since the guest keeps a cgroup per container -// and ateom sums them, but the gVisor source cannot split one at all without +// and the worker sums them, but the gVisor source cannot split one at all without // the sentry's own accounting, and a field only one runtime could ever fill // would be worse than none. type WorkloadStatsSample struct { state protoimpl.MessageState `protogen:"open.v1"` - // Identity of the measured actor, retained by ateom from the + // Identity of the measured actor, retained by the worker from the // RunWorkloadRequest / RestoreWorkloadRequest that started it. Echoed back so // the caller can attribute the sample without holding its own mapping from // worker to actor. @@ -1695,11 +1695,11 @@ type WorkloadStatsSample struct { ActorTemplateAtespace string `protobuf:"bytes,4,opt,name=actor_template_atespace,json=actorTemplateAtespace,proto3" json:"actor_template_atespace,omitempty"` ActorTemplateName string `protobuf:"bytes,5,opt,name=actor_template_name,json=actorTemplateName,proto3" json:"actor_template_name,omitempty"` // The sandbox runtime family that produced this sample, in the - // ate.dev/v1alpha1 SandboxClass form: "gvisor" (cmd/ateom-gvisor) or - // "microvm" (cmd/ateom-microvm). An ateom binary only ever reports its own. + // ate.dev/v1alpha1 SandboxClass form: "gvisor" (cmd/ateworker-gvisor) or + // "microvm" (cmd/ateworker-microvm). A worker binary only ever reports its own. SandboxClass string `protobuf:"bytes,6,opt,name=sandbox_class,json=sandboxClass,proto3" json:"sandbox_class,omitempty"` // How the measurements below were obtained. - Source StatsSource `protobuf:"varint,7,opt,name=source,proto3,enum=ateom.StatsSource" json:"source,omitempty"` + Source StatsSource `protobuf:"varint,7,opt,name=source,proto3,enum=ateworker.StatsSource" json:"source,omitempty"` // Measurements. All four are zero when source is STATS_SOURCE_UNSPECIFIED, // which means "not measured" rather than "measured as zero". // @@ -1731,7 +1731,7 @@ type WorkloadStatsSample struct { func (x *WorkloadStatsSample) Reset() { *x = WorkloadStatsSample{} - mi := &file_ateom_proto_msgTypes[21] + mi := &file_ateworker_proto_msgTypes[21] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1743,7 +1743,7 @@ func (x *WorkloadStatsSample) String() string { func (*WorkloadStatsSample) ProtoMessage() {} func (x *WorkloadStatsSample) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[21] + mi := &file_ateworker_proto_msgTypes[21] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1756,7 +1756,7 @@ func (x *WorkloadStatsSample) ProtoReflect() protoreflect.Message { // Deprecated: Use WorkloadStatsSample.ProtoReflect.Descriptor instead. func (*WorkloadStatsSample) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{21} + return file_ateworker_proto_rawDescGZIP(), []int{21} } func (x *WorkloadStatsSample) GetAtespace() string { @@ -1859,7 +1859,7 @@ type GetWorkloadStatsResponse struct { func (x *GetWorkloadStatsResponse) Reset() { *x = GetWorkloadStatsResponse{} - mi := &file_ateom_proto_msgTypes[22] + mi := &file_ateworker_proto_msgTypes[22] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1871,7 +1871,7 @@ func (x *GetWorkloadStatsResponse) String() string { func (*GetWorkloadStatsResponse) ProtoMessage() {} func (x *GetWorkloadStatsResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[22] + mi := &file_ateworker_proto_msgTypes[22] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1884,7 +1884,7 @@ func (x *GetWorkloadStatsResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetWorkloadStatsResponse.ProtoReflect.Descriptor instead. func (*GetWorkloadStatsResponse) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{22} + return file_ateworker_proto_rawDescGZIP(), []int{22} } func (x *GetWorkloadStatsResponse) GetSample() *WorkloadStatsSample { @@ -1902,7 +1902,7 @@ type GetActiveWorkloadStatsRequest struct { func (x *GetActiveWorkloadStatsRequest) Reset() { *x = GetActiveWorkloadStatsRequest{} - mi := &file_ateom_proto_msgTypes[23] + mi := &file_ateworker_proto_msgTypes[23] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1914,7 +1914,7 @@ func (x *GetActiveWorkloadStatsRequest) String() string { func (*GetActiveWorkloadStatsRequest) ProtoMessage() {} func (x *GetActiveWorkloadStatsRequest) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[23] + mi := &file_ateworker_proto_msgTypes[23] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1927,17 +1927,17 @@ func (x *GetActiveWorkloadStatsRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use GetActiveWorkloadStatsRequest.ProtoReflect.Descriptor instead. func (*GetActiveWorkloadStatsRequest) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{23} + return file_ateworker_proto_rawDescGZIP(), []int{23} } type GetActiveWorkloadStatsResponse struct { state protoimpl.MessageState `protogen:"open.v1"` - // One entry per workload this ateom is executing; empty when it is + // One entry per workload this worker is executing; empty when it is // "available". Each sample is self-describing (see the attribution rule on // the rpc); an entry with source = STATS_SOURCE_UNSPECIFIED is a workload // with no numbers to give yet (boot, restore, teardown, or a lifecycle // transition underneath the read) -- skip its measurements and take the - // next sample. An ateom serves one actor at a time today, so the list holds + // next sample. A worker serves one actor at a time today, so the list holds // at most one entry until multi-actor workers land; consumers must not // assume that. Samples []*WorkloadStatsSample `protobuf:"bytes,1,rep,name=samples,proto3" json:"samples,omitempty"` @@ -1947,7 +1947,7 @@ type GetActiveWorkloadStatsResponse struct { func (x *GetActiveWorkloadStatsResponse) Reset() { *x = GetActiveWorkloadStatsResponse{} - mi := &file_ateom_proto_msgTypes[24] + mi := &file_ateworker_proto_msgTypes[24] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1959,7 +1959,7 @@ func (x *GetActiveWorkloadStatsResponse) String() string { func (*GetActiveWorkloadStatsResponse) ProtoMessage() {} func (x *GetActiveWorkloadStatsResponse) ProtoReflect() protoreflect.Message { - mi := &file_ateom_proto_msgTypes[24] + mi := &file_ateworker_proto_msgTypes[24] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1972,7 +1972,7 @@ func (x *GetActiveWorkloadStatsResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use GetActiveWorkloadStatsResponse.ProtoReflect.Descriptor instead. func (*GetActiveWorkloadStatsResponse) Descriptor() ([]byte, []int) { - return file_ateom_proto_rawDescGZIP(), []int{24} + return file_ateworker_proto_rawDescGZIP(), []int{24} } func (x *GetActiveWorkloadStatsResponse) GetSamples() []*WorkloadStatsSample { @@ -1982,11 +1982,11 @@ func (x *GetActiveWorkloadStatsResponse) GetSamples() []*WorkloadStatsSample { return nil } -var File_ateom_proto protoreflect.FileDescriptor +var File_ateworker_proto protoreflect.FileDescriptor -const file_ateom_proto_rawDesc = "" + +const file_ateworker_proto_rawDesc = "" + "\n" + - "\vateom.proto\x12\x05ateom\"\xb7\x02\n" + + "\x0fateworker.proto\x12\tateworker\"\xb7\x02\n" + "\tActorDirs\x12\x19\n" + "\broot_dir\x18\x01 \x01(\tR\arootDir\x12$\n" + "\x0eoci_bundle_dir\x18\x02 \x01(\tR\fociBundleDir\x12%\n" + @@ -1996,7 +1996,7 @@ const file_ateom_proto_rawDesc = "" + "\x1ddurable_dir_volume_mounts_dir\x18\x05 \x01(\tR\x19durableDirVolumeMountsDir\x12>\n" + "\x1csystem_info_volume_roots_dir\x18\x06 \x01(\tR\x18systemInfoVolumeRootsDir\x12\x1f\n" + "\vvolumes_dir\x18\a \x01(\tR\n" + - "volumesDir\"\xd3\x02\n" + + "volumesDir\"\xdb\x02\n" + "\x18TerminateWorkloadRequest\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + "\n" + @@ -2005,11 +2005,11 @@ const file_ateom_proto_rawDesc = "" + "\x17actor_template_atespace\x18\x04 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x12\x1d\n" + "\n" + - "runsc_path\x18\x06 \x01(\tR\trunscPath\x12'\n" + - "\x04spec\x18\a \x01(\v2\x13.ateom.WorkloadSpecR\x04spec\x12/\n" + + "runsc_path\x18\x06 \x01(\tR\trunscPath\x12+\n" + + "\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x123\n" + "\n" + - "actor_dirs\x18\b \x01(\v2\x10.ateom.ActorDirsR\tactorDirs\"\x1b\n" + - "\x19TerminateWorkloadResponse\"\x8a\x05\n" + + "actor_dirs\x18\b \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\"\x1b\n" + + "\x19TerminateWorkloadResponse\"\x9a\x05\n" + "\x12RunWorkloadRequest\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + "\n" + @@ -2018,38 +2018,38 @@ const file_ateom_proto_rawDesc = "" + "\x17actor_template_atespace\x18\x04 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x12\x1d\n" + "\n" + - "runsc_path\x18\x06 \x01(\tR\trunscPath\x12'\n" + - "\x04spec\x18\a \x01(\v2\x13.ateom.WorkloadSpecR\x04spec\x12`\n" + - "\x13runtime_asset_paths\x18\b \x03(\v20.ateom.RunWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x12@\n" + + "runsc_path\x18\x06 \x01(\tR\trunscPath\x12+\n" + + "\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x12d\n" + + "\x13runtime_asset_paths\x18\b \x03(\v24.ateworker.RunWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x12D\n" + "\x0eegress_gateway\x18\n" + - " \x01(\v2\x14.ateom.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + + " \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + "\tcpu_milli\x18\v \x01(\x03R\bcpuMilli\x12!\n" + - "\fmemory_bytes\x18\f \x01(\x03R\vmemoryBytes\x12/\n" + + "\fmemory_bytes\x18\f \x01(\x03R\vmemoryBytes\x123\n" + "\n" + - "actor_dirs\x18\r \x01(\v2\x10.ateom.ActorDirsR\tactorDirs\x1aD\n" + + "actor_dirs\x18\r \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x1aD\n" + "\x16RuntimeAssetPathsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01B\x11\n" + "\x0f_egress_gateway\")\n" + "\rEgressGateway\x12\x18\n" + - "\aaddress\x18\x01 \x01(\tR\aaddress\"@\n" + - "\fWorkloadSpec\x120\n" + + "\aaddress\x18\x01 \x01(\tR\aaddress\"D\n" + + "\fWorkloadSpec\x124\n" + "\n" + - "containers\x18\x01 \x03(\v2\x10.ateom.ContainerR\n" + - "containers\"\xce\x03\n" + + "containers\x18\x01 \x03(\v2\x14.ateworker.ContainerR\n" + + "containers\"\xe6\x03\n" + "\tContainer\x12\x12\n" + - "\x04name\x18\x01 \x01(\tR\x04name\x125\n" + - "\fwakeup_probe\x18\x02 \x01(\v2\x12.ateom.WakeupProbeR\vwakeupProbe\x12W\n" + - "\x19durable_dir_volume_mounts\x18\x03 \x03(\v2\x1c.ateom.DurableDirVolumeMountR\x16durableDirVolumeMounts\x12>\n" + - "\x11csi_volume_mounts\x18\x04 \x03(\v2\x12.ateom.VolumeMountR\x0fcsiVolumeMounts\x12W\n" + - "\x19system_info_volume_mounts\x18\x05 \x03(\v2\x1c.ateom.SystemInfoVolumeMountR\x16systemInfoVolumeMounts\x12G\n" + - "\x13image_volume_mounts\x18\x06 \x03(\v2\x17.ateom.ImageVolumeMountR\x11imageVolumeMounts\x12;\n" + - "\x0econtainer_spec\x18\a \x01(\v2\x14.ateom.ContainerSpecR\rcontainerSpec\"\x93\x01\n" + + "\x04name\x18\x01 \x01(\tR\x04name\x129\n" + + "\fwakeup_probe\x18\x02 \x01(\v2\x16.ateworker.WakeupProbeR\vwakeupProbe\x12[\n" + + "\x19durable_dir_volume_mounts\x18\x03 \x03(\v2 .ateworker.DurableDirVolumeMountR\x16durableDirVolumeMounts\x12B\n" + + "\x11csi_volume_mounts\x18\x04 \x03(\v2\x16.ateworker.VolumeMountR\x0fcsiVolumeMounts\x12[\n" + + "\x19system_info_volume_mounts\x18\x05 \x03(\v2 .ateworker.SystemInfoVolumeMountR\x16systemInfoVolumeMounts\x12K\n" + + "\x13image_volume_mounts\x18\x06 \x03(\v2\x1b.ateworker.ImageVolumeMountR\x11imageVolumeMounts\x12?\n" + + "\x0econtainer_spec\x18\a \x01(\v2\x18.ateworker.ContainerSpecR\rcontainerSpec\"\x97\x01\n" + "\rContainerSpec\x12\x12\n" + "\x04args\x18\x01 \x03(\tR\x04args\x12\x15\n" + "\x03env\x18\x02 \x03(\tB\x03\x80\x01\x01R\x03env\x12\"\n" + - "\fcapabilities\x18\x03 \x03(\tR\fcapabilities\x123\n" + - "\tresources\x18\x04 \x01(\v2\x15.ateom.ResourceLimitsR\tresources\"R\n" + + "\fcapabilities\x18\x03 \x03(\tR\fcapabilities\x127\n" + + "\tresources\x18\x04 \x01(\v2\x19.ateworker.ResourceLimitsR\tresources\"R\n" + "\x0eResourceLimits\x12!\n" + "\fmemory_bytes\x18\x01 \x01(\x03R\vmemoryBytes\x12\x1d\n" + "\n" + @@ -2073,14 +2073,14 @@ const file_ateom_proto_rawDesc = "" + "\vvolume_name\x18\x01 \x01(\tR\n" + "volumeName\x12\x1d\n" + "\n" + - "mount_path\x18\x02 \x01(\tR\tmountPath\"g\n" + - "\vWakeupProbe\x12/\n" + - "\bhttp_get\x18\x01 \x01(\v2\x14.ateom.HttpGetActionR\ahttpGet\x12'\n" + + "mount_path\x18\x02 \x01(\tR\tmountPath\"k\n" + + "\vWakeupProbe\x123\n" + + "\bhttp_get\x18\x01 \x01(\v2\x18.ateworker.HttpGetActionR\ahttpGet\x12'\n" + "\x0ftimeout_seconds\x18\x02 \x01(\x05R\x0etimeoutSeconds\"7\n" + "\rHttpGetAction\x12\x12\n" + "\x04path\x18\x01 \x01(\tR\x04path\x12\x12\n" + "\x04port\x18\x02 \x01(\x05R\x04port\"\x15\n" + - "\x13RunWorkloadResponse\"\xdb\x04\n" + + "\x13RunWorkloadResponse\"\xeb\x04\n" + "\x19CheckpointWorkloadRequest\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + "\n" + @@ -2089,20 +2089,20 @@ const file_ateom_proto_rawDesc = "" + "\x17actor_template_atespace\x18\x04 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x12\x1d\n" + "\n" + - "runsc_path\x18\x06 \x01(\tR\trunscPath\x12'\n" + - "\x04spec\x18\a \x01(\v2\x13.ateom.WorkloadSpecR\x04spec\x12!\n" + - "\fsnapshot_uri\x18\b \x01(\tR\vsnapshotUri\x12g\n" + - "\x13runtime_asset_paths\x18\t \x03(\v27.ateom.CheckpointWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x123\n" + + "runsc_path\x18\x06 \x01(\tR\trunscPath\x12+\n" + + "\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x12!\n" + + "\fsnapshot_uri\x18\b \x01(\tR\vsnapshotUri\x12k\n" + + "\x13runtime_asset_paths\x18\t \x03(\v2;.ateworker.CheckpointWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x127\n" + "\bfidelity\x18\n" + - " \x01(\x0e2\x17.ateom.SnapshotFidelityR\bfidelity\x12/\n" + + " \x01(\x0e2\x1b.ateworker.SnapshotFidelityR\bfidelity\x123\n" + "\n" + - "actor_dirs\x18\v \x01(\v2\x10.ateom.ActorDirsR\tactorDirs\x1aD\n" + + "actor_dirs\x18\v \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x1aD\n" + "\x16RuntimeAssetPathsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"s\n" + "\x1aCheckpointWorkloadResponse\x12%\n" + "\x0esnapshot_files\x18\x01 \x03(\tR\rsnapshotFiles\x12.\n" + - "\x13data_snapshot_files\x18\x02 \x03(\tR\x11dataSnapshotFiles\"\x9c\x06\n" + + "\x13data_snapshot_files\x18\x02 \x03(\tR\x11dataSnapshotFiles\"\xb0\x06\n" + "\x16RestoreWorkloadRequest\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + "\n" + @@ -2111,17 +2111,17 @@ const file_ateom_proto_rawDesc = "" + "\x17actor_template_atespace\x18\x04 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x12\x1d\n" + "\n" + - "runsc_path\x18\x06 \x01(\tR\trunscPath\x12'\n" + - "\x04spec\x18\a \x01(\v2\x13.ateom.WorkloadSpecR\x04spec\x12!\n" + - "\fsnapshot_uri\x18\b \x01(\tR\vsnapshotUri\x12d\n" + - "\x13runtime_asset_paths\x18\t \x03(\v24.ateom.RestoreWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x123\n" + + "runsc_path\x18\x06 \x01(\tR\trunscPath\x12+\n" + + "\x04spec\x18\a \x01(\v2\x17.ateworker.WorkloadSpecR\x04spec\x12!\n" + + "\fsnapshot_uri\x18\b \x01(\tR\vsnapshotUri\x12h\n" + + "\x13runtime_asset_paths\x18\t \x03(\v28.ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntryR\x11runtimeAssetPaths\x127\n" + "\bfidelity\x18\n" + - " \x01(\x0e2\x17.ateom.SnapshotFidelityR\bfidelity\x12@\n" + - "\x0eegress_gateway\x18\f \x01(\v2\x14.ateom.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + + " \x01(\x0e2\x1b.ateworker.SnapshotFidelityR\bfidelity\x12D\n" + + "\x0eegress_gateway\x18\f \x01(\v2\x18.ateworker.EgressGatewayH\x00R\regressGateway\x88\x01\x01\x12\x1b\n" + "\tcpu_milli\x18\x0e \x01(\x03R\bcpuMilli\x12!\n" + - "\fmemory_bytes\x18\x0f \x01(\x03R\vmemoryBytes\x12/\n" + + "\fmemory_bytes\x18\x0f \x01(\x03R\vmemoryBytes\x123\n" + "\n" + - "actor_dirs\x18\x10 \x01(\v2\x10.ateom.ActorDirsR\tactorDirs\x120\n" + + "actor_dirs\x18\x10 \x01(\v2\x14.ateworker.ActorDirsR\tactorDirs\x120\n" + "\x14preserve_restore_dir\x18\x11 \x01(\bR\x12preserveRestoreDir\x1aD\n" + "\x16RuntimeAssetPathsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + @@ -2129,7 +2129,7 @@ const file_ateom_proto_rawDesc = "" + "\x0f_egress_gateway\"\x19\n" + "\x17RestoreWorkloadResponse\"6\n" + "\x17GetWorkloadStatsRequest\x12\x1b\n" + - "\tactor_uid\x18\x01 \x01(\tR\bactorUid\"\xbe\x04\n" + + "\tactor_uid\x18\x01 \x01(\tR\bactorUid\"\xc2\x04\n" + "\x13WorkloadStatsSample\x12\x1a\n" + "\batespace\x18\x01 \x01(\tR\batespace\x12\x1d\n" + "\n" + @@ -2137,20 +2137,20 @@ const file_ateom_proto_rawDesc = "" + "\tactor_uid\x18\x03 \x01(\tR\bactorUid\x126\n" + "\x17actor_template_atespace\x18\x04 \x01(\tR\x15actorTemplateAtespace\x12.\n" + "\x13actor_template_name\x18\x05 \x01(\tR\x11actorTemplateName\x12#\n" + - "\rsandbox_class\x18\x06 \x01(\tR\fsandboxClass\x12*\n" + - "\x06source\x18\a \x01(\x0e2\x12.ateom.StatsSourceR\x06source\x120\n" + + "\rsandbox_class\x18\x06 \x01(\tR\fsandboxClass\x12.\n" + + "\x06source\x18\a \x01(\x0e2\x16.ateworker.StatsSourceR\x06source\x120\n" + "\x14memory_current_bytes\x18\b \x01(\x04R\x12memoryCurrentBytes\x12*\n" + "\x11memory_peak_bytes\x18\t \x01(\x04R\x0fmemoryPeakBytes\x127\n" + "\x18memory_working_set_bytes\x18\n" + " \x01(\x04R\x15memoryWorkingSetBytes\x12$\n" + "\x0ecpu_usage_usec\x18\v \x01(\x04R\fcpuUsageUsec\x121\n" + "\x15observed_at_unix_nano\x18\f \x01(\x03R\x12observedAtUnixNano\x12&\n" + - "\x0fepoch_unix_nano\x18\r \x01(\x03R\repochUnixNano\"N\n" + - "\x18GetWorkloadStatsResponse\x122\n" + - "\x06sample\x18\x01 \x01(\v2\x1a.ateom.WorkloadStatsSampleR\x06sample\"\x1f\n" + - "\x1dGetActiveWorkloadStatsRequest\"V\n" + - "\x1eGetActiveWorkloadStatsResponse\x124\n" + - "\asamples\x18\x01 \x03(\v2\x1a.ateom.WorkloadStatsSampleR\asamples*\x90\x01\n" + + "\x0fepoch_unix_nano\x18\r \x01(\x03R\repochUnixNano\"R\n" + + "\x18GetWorkloadStatsResponse\x126\n" + + "\x06sample\x18\x01 \x01(\v2\x1e.ateworker.WorkloadStatsSampleR\x06sample\"\x1f\n" + + "\x1dGetActiveWorkloadStatsRequest\"Z\n" + + "\x1eGetActiveWorkloadStatsResponse\x128\n" + + "\asamples\x18\x01 \x03(\v2\x1e.ateworker.WorkloadStatsSampleR\asamples*\x90\x01\n" + "\x10SnapshotFidelity\x12!\n" + "\x1dSNAPSHOT_FIDELITY_UNSPECIFIED\x10\x00\x12\x1d\n" + "\x19SNAPSHOT_FIDELITY_VOLUMES\x10\x01\x12\x1c\n" + @@ -2159,101 +2159,101 @@ const file_ateom_proto_rawDesc = "" + "\vStatsSource\x12\x1c\n" + "\x18STATS_SOURCE_UNSPECIFIED\x10\x00\x12\x17\n" + "\x13STATS_SOURCE_CGROUP\x10\x01\x12\x1c\n" + - "\x18STATS_SOURCE_GUEST_AGENT\x10\x022\x9a\x04\n" + - "\x05Ateom\x12F\n" + - "\vRunWorkload\x12\x19.ateom.RunWorkloadRequest\x1a\x1a.ateom.RunWorkloadResponse\"\x00\x12[\n" + - "\x12CheckpointWorkload\x12 .ateom.CheckpointWorkloadRequest\x1a!.ateom.CheckpointWorkloadResponse\"\x00\x12R\n" + - "\x0fRestoreWorkload\x12\x1d.ateom.RestoreWorkloadRequest\x1a\x1e.ateom.RestoreWorkloadResponse\"\x00\x12U\n" + - "\x10GetWorkloadStats\x12\x1e.ateom.GetWorkloadStatsRequest\x1a\x1f.ateom.GetWorkloadStatsResponse\"\x00\x12g\n" + - "\x16GetActiveWorkloadStats\x12$.ateom.GetActiveWorkloadStatsRequest\x1a%.ateom.GetActiveWorkloadStatsResponse\"\x00\x12X\n" + - "\x11TerminateWorkload\x12\x1f.ateom.TerminateWorkloadRequest\x1a .ateom.TerminateWorkloadResponse\"\x00B=Z;github.com/agent-substrate/substrate/internal/proto/ateompbb\x06proto3" + "\x18STATS_SOURCE_GUEST_AGENT\x10\x022\xcb\x04\n" + + "\x06Worker\x12N\n" + + "\vRunWorkload\x12\x1d.ateworker.RunWorkloadRequest\x1a\x1e.ateworker.RunWorkloadResponse\"\x00\x12c\n" + + "\x12CheckpointWorkload\x12$.ateworker.CheckpointWorkloadRequest\x1a%.ateworker.CheckpointWorkloadResponse\"\x00\x12Z\n" + + "\x0fRestoreWorkload\x12!.ateworker.RestoreWorkloadRequest\x1a\".ateworker.RestoreWorkloadResponse\"\x00\x12]\n" + + "\x10GetWorkloadStats\x12\".ateworker.GetWorkloadStatsRequest\x1a#.ateworker.GetWorkloadStatsResponse\"\x00\x12o\n" + + "\x16GetActiveWorkloadStats\x12(.ateworker.GetActiveWorkloadStatsRequest\x1a).ateworker.GetActiveWorkloadStatsResponse\"\x00\x12`\n" + + "\x11TerminateWorkload\x12#.ateworker.TerminateWorkloadRequest\x1a$.ateworker.TerminateWorkloadResponse\"\x00BAZ?github.com/agent-substrate/substrate/internal/proto/ateworkerpbb\x06proto3" var ( - file_ateom_proto_rawDescOnce sync.Once - file_ateom_proto_rawDescData []byte + file_ateworker_proto_rawDescOnce sync.Once + file_ateworker_proto_rawDescData []byte ) -func file_ateom_proto_rawDescGZIP() []byte { - file_ateom_proto_rawDescOnce.Do(func() { - file_ateom_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_ateom_proto_rawDesc), len(file_ateom_proto_rawDesc))) +func file_ateworker_proto_rawDescGZIP() []byte { + file_ateworker_proto_rawDescOnce.Do(func() { + file_ateworker_proto_rawDescData = protoimpl.X.CompressGZIP(unsafe.Slice(unsafe.StringData(file_ateworker_proto_rawDesc), len(file_ateworker_proto_rawDesc))) }) - return file_ateom_proto_rawDescData -} - -var file_ateom_proto_enumTypes = make([]protoimpl.EnumInfo, 2) -var file_ateom_proto_msgTypes = make([]protoimpl.MessageInfo, 28) -var file_ateom_proto_goTypes = []any{ - (SnapshotFidelity)(0), // 0: ateom.SnapshotFidelity - (StatsSource)(0), // 1: ateom.StatsSource - (*ActorDirs)(nil), // 2: ateom.ActorDirs - (*TerminateWorkloadRequest)(nil), // 3: ateom.TerminateWorkloadRequest - (*TerminateWorkloadResponse)(nil), // 4: ateom.TerminateWorkloadResponse - (*RunWorkloadRequest)(nil), // 5: ateom.RunWorkloadRequest - (*EgressGateway)(nil), // 6: ateom.EgressGateway - (*WorkloadSpec)(nil), // 7: ateom.WorkloadSpec - (*Container)(nil), // 8: ateom.Container - (*ContainerSpec)(nil), // 9: ateom.ContainerSpec - (*ResourceLimits)(nil), // 10: ateom.ResourceLimits - (*VolumeMount)(nil), // 11: ateom.VolumeMount - (*DurableDirVolumeMount)(nil), // 12: ateom.DurableDirVolumeMount - (*SystemInfoVolumeMount)(nil), // 13: ateom.SystemInfoVolumeMount - (*ImageVolumeMount)(nil), // 14: ateom.ImageVolumeMount - (*WakeupProbe)(nil), // 15: ateom.WakeupProbe - (*HttpGetAction)(nil), // 16: ateom.HttpGetAction - (*RunWorkloadResponse)(nil), // 17: ateom.RunWorkloadResponse - (*CheckpointWorkloadRequest)(nil), // 18: ateom.CheckpointWorkloadRequest - (*CheckpointWorkloadResponse)(nil), // 19: ateom.CheckpointWorkloadResponse - (*RestoreWorkloadRequest)(nil), // 20: ateom.RestoreWorkloadRequest - (*RestoreWorkloadResponse)(nil), // 21: ateom.RestoreWorkloadResponse - (*GetWorkloadStatsRequest)(nil), // 22: ateom.GetWorkloadStatsRequest - (*WorkloadStatsSample)(nil), // 23: ateom.WorkloadStatsSample - (*GetWorkloadStatsResponse)(nil), // 24: ateom.GetWorkloadStatsResponse - (*GetActiveWorkloadStatsRequest)(nil), // 25: ateom.GetActiveWorkloadStatsRequest - (*GetActiveWorkloadStatsResponse)(nil), // 26: ateom.GetActiveWorkloadStatsResponse - nil, // 27: ateom.RunWorkloadRequest.RuntimeAssetPathsEntry - nil, // 28: ateom.CheckpointWorkloadRequest.RuntimeAssetPathsEntry - nil, // 29: ateom.RestoreWorkloadRequest.RuntimeAssetPathsEntry -} -var file_ateom_proto_depIdxs = []int32{ - 7, // 0: ateom.TerminateWorkloadRequest.spec:type_name -> ateom.WorkloadSpec - 2, // 1: ateom.TerminateWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs - 7, // 2: ateom.RunWorkloadRequest.spec:type_name -> ateom.WorkloadSpec - 27, // 3: ateom.RunWorkloadRequest.runtime_asset_paths:type_name -> ateom.RunWorkloadRequest.RuntimeAssetPathsEntry - 6, // 4: ateom.RunWorkloadRequest.egress_gateway:type_name -> ateom.EgressGateway - 2, // 5: ateom.RunWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs - 8, // 6: ateom.WorkloadSpec.containers:type_name -> ateom.Container - 15, // 7: ateom.Container.wakeup_probe:type_name -> ateom.WakeupProbe - 12, // 8: ateom.Container.durable_dir_volume_mounts:type_name -> ateom.DurableDirVolumeMount - 11, // 9: ateom.Container.csi_volume_mounts:type_name -> ateom.VolumeMount - 13, // 10: ateom.Container.system_info_volume_mounts:type_name -> ateom.SystemInfoVolumeMount - 14, // 11: ateom.Container.image_volume_mounts:type_name -> ateom.ImageVolumeMount - 9, // 12: ateom.Container.container_spec:type_name -> ateom.ContainerSpec - 10, // 13: ateom.ContainerSpec.resources:type_name -> ateom.ResourceLimits - 16, // 14: ateom.WakeupProbe.http_get:type_name -> ateom.HttpGetAction - 7, // 15: ateom.CheckpointWorkloadRequest.spec:type_name -> ateom.WorkloadSpec - 28, // 16: ateom.CheckpointWorkloadRequest.runtime_asset_paths:type_name -> ateom.CheckpointWorkloadRequest.RuntimeAssetPathsEntry - 0, // 17: ateom.CheckpointWorkloadRequest.fidelity:type_name -> ateom.SnapshotFidelity - 2, // 18: ateom.CheckpointWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs - 7, // 19: ateom.RestoreWorkloadRequest.spec:type_name -> ateom.WorkloadSpec - 29, // 20: ateom.RestoreWorkloadRequest.runtime_asset_paths:type_name -> ateom.RestoreWorkloadRequest.RuntimeAssetPathsEntry - 0, // 21: ateom.RestoreWorkloadRequest.fidelity:type_name -> ateom.SnapshotFidelity - 6, // 22: ateom.RestoreWorkloadRequest.egress_gateway:type_name -> ateom.EgressGateway - 2, // 23: ateom.RestoreWorkloadRequest.actor_dirs:type_name -> ateom.ActorDirs - 1, // 24: ateom.WorkloadStatsSample.source:type_name -> ateom.StatsSource - 23, // 25: ateom.GetWorkloadStatsResponse.sample:type_name -> ateom.WorkloadStatsSample - 23, // 26: ateom.GetActiveWorkloadStatsResponse.samples:type_name -> ateom.WorkloadStatsSample - 5, // 27: ateom.Ateom.RunWorkload:input_type -> ateom.RunWorkloadRequest - 18, // 28: ateom.Ateom.CheckpointWorkload:input_type -> ateom.CheckpointWorkloadRequest - 20, // 29: ateom.Ateom.RestoreWorkload:input_type -> ateom.RestoreWorkloadRequest - 22, // 30: ateom.Ateom.GetWorkloadStats:input_type -> ateom.GetWorkloadStatsRequest - 25, // 31: ateom.Ateom.GetActiveWorkloadStats:input_type -> ateom.GetActiveWorkloadStatsRequest - 3, // 32: ateom.Ateom.TerminateWorkload:input_type -> ateom.TerminateWorkloadRequest - 17, // 33: ateom.Ateom.RunWorkload:output_type -> ateom.RunWorkloadResponse - 19, // 34: ateom.Ateom.CheckpointWorkload:output_type -> ateom.CheckpointWorkloadResponse - 21, // 35: ateom.Ateom.RestoreWorkload:output_type -> ateom.RestoreWorkloadResponse - 24, // 36: ateom.Ateom.GetWorkloadStats:output_type -> ateom.GetWorkloadStatsResponse - 26, // 37: ateom.Ateom.GetActiveWorkloadStats:output_type -> ateom.GetActiveWorkloadStatsResponse - 4, // 38: ateom.Ateom.TerminateWorkload:output_type -> ateom.TerminateWorkloadResponse + return file_ateworker_proto_rawDescData +} + +var file_ateworker_proto_enumTypes = make([]protoimpl.EnumInfo, 2) +var file_ateworker_proto_msgTypes = make([]protoimpl.MessageInfo, 28) +var file_ateworker_proto_goTypes = []any{ + (SnapshotFidelity)(0), // 0: ateworker.SnapshotFidelity + (StatsSource)(0), // 1: ateworker.StatsSource + (*ActorDirs)(nil), // 2: ateworker.ActorDirs + (*TerminateWorkloadRequest)(nil), // 3: ateworker.TerminateWorkloadRequest + (*TerminateWorkloadResponse)(nil), // 4: ateworker.TerminateWorkloadResponse + (*RunWorkloadRequest)(nil), // 5: ateworker.RunWorkloadRequest + (*EgressGateway)(nil), // 6: ateworker.EgressGateway + (*WorkloadSpec)(nil), // 7: ateworker.WorkloadSpec + (*Container)(nil), // 8: ateworker.Container + (*ContainerSpec)(nil), // 9: ateworker.ContainerSpec + (*ResourceLimits)(nil), // 10: ateworker.ResourceLimits + (*VolumeMount)(nil), // 11: ateworker.VolumeMount + (*DurableDirVolumeMount)(nil), // 12: ateworker.DurableDirVolumeMount + (*SystemInfoVolumeMount)(nil), // 13: ateworker.SystemInfoVolumeMount + (*ImageVolumeMount)(nil), // 14: ateworker.ImageVolumeMount + (*WakeupProbe)(nil), // 15: ateworker.WakeupProbe + (*HttpGetAction)(nil), // 16: ateworker.HttpGetAction + (*RunWorkloadResponse)(nil), // 17: ateworker.RunWorkloadResponse + (*CheckpointWorkloadRequest)(nil), // 18: ateworker.CheckpointWorkloadRequest + (*CheckpointWorkloadResponse)(nil), // 19: ateworker.CheckpointWorkloadResponse + (*RestoreWorkloadRequest)(nil), // 20: ateworker.RestoreWorkloadRequest + (*RestoreWorkloadResponse)(nil), // 21: ateworker.RestoreWorkloadResponse + (*GetWorkloadStatsRequest)(nil), // 22: ateworker.GetWorkloadStatsRequest + (*WorkloadStatsSample)(nil), // 23: ateworker.WorkloadStatsSample + (*GetWorkloadStatsResponse)(nil), // 24: ateworker.GetWorkloadStatsResponse + (*GetActiveWorkloadStatsRequest)(nil), // 25: ateworker.GetActiveWorkloadStatsRequest + (*GetActiveWorkloadStatsResponse)(nil), // 26: ateworker.GetActiveWorkloadStatsResponse + nil, // 27: ateworker.RunWorkloadRequest.RuntimeAssetPathsEntry + nil, // 28: ateworker.CheckpointWorkloadRequest.RuntimeAssetPathsEntry + nil, // 29: ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntry +} +var file_ateworker_proto_depIdxs = []int32{ + 7, // 0: ateworker.TerminateWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec + 2, // 1: ateworker.TerminateWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs + 7, // 2: ateworker.RunWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec + 27, // 3: ateworker.RunWorkloadRequest.runtime_asset_paths:type_name -> ateworker.RunWorkloadRequest.RuntimeAssetPathsEntry + 6, // 4: ateworker.RunWorkloadRequest.egress_gateway:type_name -> ateworker.EgressGateway + 2, // 5: ateworker.RunWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs + 8, // 6: ateworker.WorkloadSpec.containers:type_name -> ateworker.Container + 15, // 7: ateworker.Container.wakeup_probe:type_name -> ateworker.WakeupProbe + 12, // 8: ateworker.Container.durable_dir_volume_mounts:type_name -> ateworker.DurableDirVolumeMount + 11, // 9: ateworker.Container.csi_volume_mounts:type_name -> ateworker.VolumeMount + 13, // 10: ateworker.Container.system_info_volume_mounts:type_name -> ateworker.SystemInfoVolumeMount + 14, // 11: ateworker.Container.image_volume_mounts:type_name -> ateworker.ImageVolumeMount + 9, // 12: ateworker.Container.container_spec:type_name -> ateworker.ContainerSpec + 10, // 13: ateworker.ContainerSpec.resources:type_name -> ateworker.ResourceLimits + 16, // 14: ateworker.WakeupProbe.http_get:type_name -> ateworker.HttpGetAction + 7, // 15: ateworker.CheckpointWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec + 28, // 16: ateworker.CheckpointWorkloadRequest.runtime_asset_paths:type_name -> ateworker.CheckpointWorkloadRequest.RuntimeAssetPathsEntry + 0, // 17: ateworker.CheckpointWorkloadRequest.fidelity:type_name -> ateworker.SnapshotFidelity + 2, // 18: ateworker.CheckpointWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs + 7, // 19: ateworker.RestoreWorkloadRequest.spec:type_name -> ateworker.WorkloadSpec + 29, // 20: ateworker.RestoreWorkloadRequest.runtime_asset_paths:type_name -> ateworker.RestoreWorkloadRequest.RuntimeAssetPathsEntry + 0, // 21: ateworker.RestoreWorkloadRequest.fidelity:type_name -> ateworker.SnapshotFidelity + 6, // 22: ateworker.RestoreWorkloadRequest.egress_gateway:type_name -> ateworker.EgressGateway + 2, // 23: ateworker.RestoreWorkloadRequest.actor_dirs:type_name -> ateworker.ActorDirs + 1, // 24: ateworker.WorkloadStatsSample.source:type_name -> ateworker.StatsSource + 23, // 25: ateworker.GetWorkloadStatsResponse.sample:type_name -> ateworker.WorkloadStatsSample + 23, // 26: ateworker.GetActiveWorkloadStatsResponse.samples:type_name -> ateworker.WorkloadStatsSample + 5, // 27: ateworker.Worker.RunWorkload:input_type -> ateworker.RunWorkloadRequest + 18, // 28: ateworker.Worker.CheckpointWorkload:input_type -> ateworker.CheckpointWorkloadRequest + 20, // 29: ateworker.Worker.RestoreWorkload:input_type -> ateworker.RestoreWorkloadRequest + 22, // 30: ateworker.Worker.GetWorkloadStats:input_type -> ateworker.GetWorkloadStatsRequest + 25, // 31: ateworker.Worker.GetActiveWorkloadStats:input_type -> ateworker.GetActiveWorkloadStatsRequest + 3, // 32: ateworker.Worker.TerminateWorkload:input_type -> ateworker.TerminateWorkloadRequest + 17, // 33: ateworker.Worker.RunWorkload:output_type -> ateworker.RunWorkloadResponse + 19, // 34: ateworker.Worker.CheckpointWorkload:output_type -> ateworker.CheckpointWorkloadResponse + 21, // 35: ateworker.Worker.RestoreWorkload:output_type -> ateworker.RestoreWorkloadResponse + 24, // 36: ateworker.Worker.GetWorkloadStats:output_type -> ateworker.GetWorkloadStatsResponse + 26, // 37: ateworker.Worker.GetActiveWorkloadStats:output_type -> ateworker.GetActiveWorkloadStatsResponse + 4, // 38: ateworker.Worker.TerminateWorkload:output_type -> ateworker.TerminateWorkloadResponse 33, // [33:39] is the sub-list for method output_type 27, // [27:33] is the sub-list for method input_type 27, // [27:27] is the sub-list for extension type_name @@ -2261,29 +2261,29 @@ var file_ateom_proto_depIdxs = []int32{ 0, // [0:27] is the sub-list for field type_name } -func init() { file_ateom_proto_init() } -func file_ateom_proto_init() { - if File_ateom_proto != nil { +func init() { file_ateworker_proto_init() } +func file_ateworker_proto_init() { + if File_ateworker_proto != nil { return } - file_ateom_proto_msgTypes[3].OneofWrappers = []any{} - file_ateom_proto_msgTypes[18].OneofWrappers = []any{} + file_ateworker_proto_msgTypes[3].OneofWrappers = []any{} + file_ateworker_proto_msgTypes[18].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), - RawDescriptor: unsafe.Slice(unsafe.StringData(file_ateom_proto_rawDesc), len(file_ateom_proto_rawDesc)), + RawDescriptor: unsafe.Slice(unsafe.StringData(file_ateworker_proto_rawDesc), len(file_ateworker_proto_rawDesc)), NumEnums: 2, NumMessages: 28, NumExtensions: 0, NumServices: 1, }, - GoTypes: file_ateom_proto_goTypes, - DependencyIndexes: file_ateom_proto_depIdxs, - EnumInfos: file_ateom_proto_enumTypes, - MessageInfos: file_ateom_proto_msgTypes, + GoTypes: file_ateworker_proto_goTypes, + DependencyIndexes: file_ateworker_proto_depIdxs, + EnumInfos: file_ateworker_proto_enumTypes, + MessageInfos: file_ateworker_proto_msgTypes, }.Build() - File_ateom_proto = out.File - file_ateom_proto_goTypes = nil - file_ateom_proto_depIdxs = nil + File_ateworker_proto = out.File + file_ateworker_proto_goTypes = nil + file_ateworker_proto_depIdxs = nil } diff --git a/internal/proto/ateompb/ateom.proto b/internal/proto/ateworkerpb/ateworker.proto similarity index 88% rename from internal/proto/ateompb/ateom.proto rename to internal/proto/ateworkerpb/ateworker.proto index 3a253cca13..79b8da40b8 100644 --- a/internal/proto/ateompb/ateom.proto +++ b/internal/proto/ateworkerpb/ateworker.proto @@ -14,45 +14,45 @@ syntax = "proto3"; -package ateom; +package ateworker; -option go_package = "github.com/agent-substrate/substrate/internal/proto/ateompb"; +option go_package = "github.com/agent-substrate/substrate/internal/proto/ateworkerpb"; -// Ateom is the interface to control a single gVisor (or, in the future microVM) +// Worker is the interface to control a single gVisor (or, in the future microVM) // guest inside a worker pod. // -// Each ateom server has two main states, "available" and "executing". +// Each worker has two main states, "available" and "executing". // -// When the ateom is "available", the substrate control plane is free to either +// When the worker is "available", the substrate control plane is free to either // boot a new workload (using RunWorkload), or restore an existing workload from -// a checkpoint (using RestoreWorkload). These calls move the ateom into +// a checkpoint (using RestoreWorkload). These calls move the worker into // "executing" state. // -// When the ateom is "executing", the substrate control plane can checkpoint the -// running workload (with CheckpointWorkload). This moves the ateom back to +// When the worker is "executing", the substrate control plane can checkpoint the +// running workload (with CheckpointWorkload). This moves the worker back to // "free" state. -service Ateom { - // RunWorkload tells ateom to begin running a new workload (one or more +service Worker { + // RunWorkload tells the worker to begin running a new workload (one or more // containers, potentially with shared filesystems). rpc RunWorkload(RunWorkloadRequest) returns (RunWorkloadResponse) {} - // CheckpointWorkload tells ateom to save the current state of the running + // CheckpointWorkload tells the worker to save the current state of the running // workload to object storage, and then completely reset itself to a blank // state (back to "available" state.) rpc CheckpointWorkload(CheckpointWorkloadRequest) returns (CheckpointWorkloadResponse) {} // RestoreWorkload restores a workload from checkpoint that was previously - // written by CheckpointWorkload. Ateom will handle downloading the correct + // written by CheckpointWorkload. The worker will handle downloading the correct // gVisor / runsc version to match the checkpoint. rpc RestoreWorkload(RestoreWorkloadRequest) returns (RestoreWorkloadResponse) {} // GetWorkloadStats returns a point-in-time resource-usage sample for the - // workload this ateom is currently executing. + // workload this worker is currently executing. // - // It is a pure read: unlike the three calls above it does not move the ateom + // It is a pure read: unlike the three calls above it does not move the worker // between "available" and "executing", so it is safe to call on a timer for // the whole lifetime of a workload. It also does not wait on them. Those - // three serialize on a mutex an ateom holds for a whole boot, restore, or + // three serialize on a mutex a worker holds for a whole boot, restore, or // checkpoint; this call reads its state from outside that mutex, so a poll // landing in the middle of one answers immediately instead of going quiet for // the duration -- which would silence the poller during exactly the phases @@ -61,31 +61,31 @@ service Ateom { // Two ways it declines to give a sample, and they ask different things of the // caller: // - // * NOT_FOUND -- this ateom is not executing the actor in the request. It + // * NOT_FOUND -- this worker is not executing the actor in the request. It // may be "available", or a recycled worker may have moved on to a // different actor. Retrying on the same timer will not change the answer: // the caller's worker-to-actor mapping is stale and wants re-resolving. // - // * FAILED_PRECONDITION -- this ateom is executing the requested actor but + // * FAILED_PRECONDITION -- this worker is executing the requested actor but // has no sample to give yet. It accepts an actor before the sandbox it // will measure exists, so a poll landing in the boot lands here. Read it // as "no numbers right now", not as "the actor is gone": it is transient, // and the caller should skip this sample and take the next one. // // The split is worth the two codes because the identity is retained from the - // moment the ateom accepts the actor, on both runtimes. That is what makes + // moment the worker accepts the actor, on both runtimes. That is what makes // "booting" distinguishable from "not here" at all, and it means a workload // that dies during boot is attributable rather than anonymous. rpc GetWorkloadStats(GetWorkloadStatsRequest) returns (GetWorkloadStatsResponse) {} - // GetActiveWorkloadStats reports every actor this ateom hosts, without + // GetActiveWorkloadStats reports every actor this worker hosts, without // asserting an identity. It is the discovery read for a - // scraper that enumerates ateoms and holds no worker-to-actor mapping; + // scraper that enumerates workers and holds no worker-to-actor mapping; // GetWorkloadStats above is the verified read for a caller that must be // answered about a specific actor. // - // It serves each actor's latest sample from the ateom's own sampling timer, - // so a poll reads no sandbox and has no error path: an "available" ateom + // It serves each actor's latest sample from the worker's own sampling timer, + // so a poll reads no sandbox and has no error path: an "available" worker // answers an empty samples list, and a workload with nothing measured yet // (a boot, a restore, or a guest the last sweep did not reach) appears as a // sample carrying its attribution with source = STATS_SOURCE_UNSPECIFIED -- @@ -101,12 +101,12 @@ service Ateom { rpc GetActiveWorkloadStats(GetActiveWorkloadStatsRequest) returns (GetActiveWorkloadStatsResponse) {} // TerminateWorkload stops and deletes container workloads and cleans up - // network and bundle overlays on ateom. + // network and bundle overlays on the worker. rpc TerminateWorkload(TerminateWorkloadRequest) returns (TerminateWorkloadResponse) {} } // ActorDirs is the set of per-actor directories atelet prepares on the volume -// it shares with ateom. ateom takes them from the RPC instead of deriving +// it shares with the worker. The worker takes them from the RPC instead of deriving // them from the actor UID. // Every directory is under root_dir. message ActorDirs { @@ -160,10 +160,10 @@ message RunWorkloadRequest { // When absent the actor has no egress: its TCP is captured and refused. optional EgressGateway egress_gateway = 10; - // The actor's declared size, from the ActorTemplate's resource limits. ateom + // The actor's declared size, from the ActorTemplate's resource limits. The worker // sizes the sandbox to these (cgroup caps via the OCI spec, and for the // micro-VM the VM's vCPU count and memory). Zero means "unset": keep the - // runtime default (unlimited for gVisor, ateom's own default for the micro-VM). + // runtime default (unlimited for gVisor, the worker's own default for the micro-VM). int64 cpu_milli = 11; // CPU limit in millicores (1000 = one core). int64 memory_bytes = 12; // Memory limit in bytes. ActorDirs actor_dirs = 13; @@ -200,7 +200,7 @@ message Container { // image_volume_mounts are the image volumes this container mounts, if any. repeated ImageVolumeMount image_volume_mounts = 6; - // container_spec is what ateom builds the container's OCI spec from, + // container_spec is what the worker builds the container's OCI spec from, // besides its volume mounts. Unset on RPCs that start no process. ContainerSpec container_spec = 7; } @@ -256,7 +256,7 @@ message SystemInfoVolumeMount { string mount_path = 2; } -// ImageVolumeMount is one image volume mounted into a container. ateom uses +// ImageVolumeMount is one image volume mounted into a container. The worker uses // these to construct the container's volume mounts — each names the volume // and its destination path. message ImageVolumeMount { @@ -335,7 +335,7 @@ message CheckpointWorkloadRequest { } message CheckpointWorkloadResponse { - // snapshot_files lists the files ateom wrote into the checkpoint directory + // snapshot_files lists the files the worker wrote into the checkpoint directory // (relative names) for atelet to ship to object storage. Each runtime reports // its own set (gVisor's image files, cloud-hypervisor's snapshot set, ...). repeated string snapshot_files = 1; @@ -379,7 +379,7 @@ message RestoreWorkloadRequest { ActorDirs actor_dirs = 16; // actor_dirs.restore_dir is a preserved local snapshot, not a scratch copy: - // ateom must not delete or modify files in it. + // the worker must not delete or modify files in it. bool preserve_restore_dir = 17; } @@ -388,7 +388,7 @@ message RestoreWorkloadResponse { message GetWorkloadStatsRequest { // The actor the caller believes is executing here. A worker can be recycled - // between the caller's view of the world and this call, so ateom answers + // between the caller's view of the world and this call, so the worker answers // NOT_FOUND on a mismatch rather than reporting a different actor's numbers // under the requested actor's identity. string actor_uid = 1; @@ -415,11 +415,11 @@ enum StatsSource { // shared by both stats reads. The unit of measurement is the SANDBOX, which // today equals the actor. Per-container attribution is not reported: the // micro-VM source could give it, since the guest keeps a cgroup per container -// and ateom sums them, but the gVisor source cannot split one at all without +// and the worker sums them, but the gVisor source cannot split one at all without // the sentry's own accounting, and a field only one runtime could ever fill // would be worse than none. message WorkloadStatsSample { - // Identity of the measured actor, retained by ateom from the + // Identity of the measured actor, retained by the worker from the // RunWorkloadRequest / RestoreWorkloadRequest that started it. Echoed back so // the caller can attribute the sample without holding its own mapping from // worker to actor. @@ -430,8 +430,8 @@ message WorkloadStatsSample { string actor_template_name = 5; // The sandbox runtime family that produced this sample, in the - // ate.dev/v1alpha1 SandboxClass form: "gvisor" (cmd/ateom-gvisor) or - // "microvm" (cmd/ateom-microvm). An ateom binary only ever reports its own. + // ate.dev/v1alpha1 SandboxClass form: "gvisor" (cmd/ateworker-gvisor) or + // "microvm" (cmd/ateworker-microvm). A worker binary only ever reports its own. string sandbox_class = 6; // How the measurements below were obtained. @@ -474,12 +474,12 @@ message GetActiveWorkloadStatsRequest { } message GetActiveWorkloadStatsResponse { - // One entry per workload this ateom is executing; empty when it is + // One entry per workload this worker is executing; empty when it is // "available". Each sample is self-describing (see the attribution rule on // the rpc); an entry with source = STATS_SOURCE_UNSPECIFIED is a workload // with no numbers to give yet (boot, restore, teardown, or a lifecycle // transition underneath the read) -- skip its measurements and take the - // next sample. An ateom serves one actor at a time today, so the list holds + // next sample. A worker serves one actor at a time today, so the list holds // at most one entry until multi-actor workers land; consumers must not // assume that. repeated WorkloadStatsSample samples = 1; diff --git a/internal/proto/ateompb/ateom_grpc.pb.go b/internal/proto/ateworkerpb/ateworker_grpc.pb.go similarity index 60% rename from internal/proto/ateompb/ateom_grpc.pb.go rename to internal/proto/ateworkerpb/ateworker_grpc.pb.go index 572fee2b79..2c4854e5a4 100644 --- a/internal/proto/ateompb/ateom_grpc.pb.go +++ b/internal/proto/ateworkerpb/ateworker_grpc.pb.go @@ -16,9 +16,9 @@ // versions: // - protoc-gen-go-grpc v1.6.2 // - protoc v4.25.3 -// source: ateom.proto +// source: ateworker.proto -package ateompb +package ateworkerpb import ( context "context" @@ -33,50 +33,50 @@ import ( const _ = grpc.SupportPackageIsVersion9 const ( - Ateom_RunWorkload_FullMethodName = "/ateom.Ateom/RunWorkload" - Ateom_CheckpointWorkload_FullMethodName = "/ateom.Ateom/CheckpointWorkload" - Ateom_RestoreWorkload_FullMethodName = "/ateom.Ateom/RestoreWorkload" - Ateom_GetWorkloadStats_FullMethodName = "/ateom.Ateom/GetWorkloadStats" - Ateom_GetActiveWorkloadStats_FullMethodName = "/ateom.Ateom/GetActiveWorkloadStats" - Ateom_TerminateWorkload_FullMethodName = "/ateom.Ateom/TerminateWorkload" + Worker_RunWorkload_FullMethodName = "/ateworker.Worker/RunWorkload" + Worker_CheckpointWorkload_FullMethodName = "/ateworker.Worker/CheckpointWorkload" + Worker_RestoreWorkload_FullMethodName = "/ateworker.Worker/RestoreWorkload" + Worker_GetWorkloadStats_FullMethodName = "/ateworker.Worker/GetWorkloadStats" + Worker_GetActiveWorkloadStats_FullMethodName = "/ateworker.Worker/GetActiveWorkloadStats" + Worker_TerminateWorkload_FullMethodName = "/ateworker.Worker/TerminateWorkload" ) -// AteomClient is the client API for Ateom service. +// WorkerClient is the client API for Worker service. // // For semantics around ctx use and closing/ending streaming RPCs, please refer to https://pkg.go.dev/google.golang.org/grpc/?tab=doc#ClientConn.NewStream. // -// Ateom is the interface to control a single gVisor (or, in the future microVM) +// Worker is the interface to control a single gVisor (or, in the future microVM) // guest inside a worker pod. // -// Each ateom server has two main states, "available" and "executing". +// Each worker has two main states, "available" and "executing". // -// When the ateom is "available", the substrate control plane is free to either +// When the worker is "available", the substrate control plane is free to either // boot a new workload (using RunWorkload), or restore an existing workload from -// a checkpoint (using RestoreWorkload). These calls move the ateom into +// a checkpoint (using RestoreWorkload). These calls move the worker into // "executing" state. // -// When the ateom is "executing", the substrate control plane can checkpoint the -// running workload (with CheckpointWorkload). This moves the ateom back to +// When the worker is "executing", the substrate control plane can checkpoint the +// running workload (with CheckpointWorkload). This moves the worker back to // "free" state. -type AteomClient interface { - // RunWorkload tells ateom to begin running a new workload (one or more +type WorkerClient interface { + // RunWorkload tells the worker to begin running a new workload (one or more // containers, potentially with shared filesystems). RunWorkload(ctx context.Context, in *RunWorkloadRequest, opts ...grpc.CallOption) (*RunWorkloadResponse, error) - // CheckpointWorkload tells ateom to save the current state of the running + // CheckpointWorkload tells the worker to save the current state of the running // workload to object storage, and then completely reset itself to a blank // state (back to "available" state.) CheckpointWorkload(ctx context.Context, in *CheckpointWorkloadRequest, opts ...grpc.CallOption) (*CheckpointWorkloadResponse, error) // RestoreWorkload restores a workload from checkpoint that was previously - // written by CheckpointWorkload. Ateom will handle downloading the correct + // written by CheckpointWorkload. The worker will handle downloading the correct // gVisor / runsc version to match the checkpoint. RestoreWorkload(ctx context.Context, in *RestoreWorkloadRequest, opts ...grpc.CallOption) (*RestoreWorkloadResponse, error) // GetWorkloadStats returns a point-in-time resource-usage sample for the - // workload this ateom is currently executing. + // workload this worker is currently executing. // - // It is a pure read: unlike the three calls above it does not move the ateom + // It is a pure read: unlike the three calls above it does not move the worker // between "available" and "executing", so it is safe to call on a timer for // the whole lifetime of a workload. It also does not wait on them. Those - // three serialize on a mutex an ateom holds for a whole boot, restore, or + // three serialize on a mutex a worker holds for a whole boot, restore, or // checkpoint; this call reads its state from outside that mutex, so a poll // landing in the middle of one answers immediately instead of going quiet for // the duration -- which would silence the poller during exactly the phases @@ -85,30 +85,30 @@ type AteomClient interface { // Two ways it declines to give a sample, and they ask different things of the // caller: // - // * NOT_FOUND -- this ateom is not executing the actor in the request. It + // * NOT_FOUND -- this worker is not executing the actor in the request. It // may be "available", or a recycled worker may have moved on to a // different actor. Retrying on the same timer will not change the answer: // the caller's worker-to-actor mapping is stale and wants re-resolving. // - // * FAILED_PRECONDITION -- this ateom is executing the requested actor but + // * FAILED_PRECONDITION -- this worker is executing the requested actor but // has no sample to give yet. It accepts an actor before the sandbox it // will measure exists, so a poll landing in the boot lands here. Read it // as "no numbers right now", not as "the actor is gone": it is transient, // and the caller should skip this sample and take the next one. // // The split is worth the two codes because the identity is retained from the - // moment the ateom accepts the actor, on both runtimes. That is what makes + // moment the worker accepts the actor, on both runtimes. That is what makes // "booting" distinguishable from "not here" at all, and it means a workload // that dies during boot is attributable rather than anonymous. GetWorkloadStats(ctx context.Context, in *GetWorkloadStatsRequest, opts ...grpc.CallOption) (*GetWorkloadStatsResponse, error) - // GetActiveWorkloadStats reports every actor this ateom hosts, without + // GetActiveWorkloadStats reports every actor this worker hosts, without // asserting an identity. It is the discovery read for a - // scraper that enumerates ateoms and holds no worker-to-actor mapping; + // scraper that enumerates workers and holds no worker-to-actor mapping; // GetWorkloadStats above is the verified read for a caller that must be // answered about a specific actor. // - // It serves each actor's latest sample from the ateom's own sampling timer, - // so a poll reads no sandbox and has no error path: an "available" ateom + // It serves each actor's latest sample from the worker's own sampling timer, + // so a poll reads no sandbox and has no error path: an "available" worker // answers an empty samples list, and a workload with nothing measured yet // (a boot, a restore, or a guest the last sweep did not reach) appears as a // sample carrying its attribution with source = STATS_SOURCE_UNSPECIFIED -- @@ -123,114 +123,114 @@ type AteomClient interface { // it is safe on a timer and does not touch the lifecycle mutex. GetActiveWorkloadStats(ctx context.Context, in *GetActiveWorkloadStatsRequest, opts ...grpc.CallOption) (*GetActiveWorkloadStatsResponse, error) // TerminateWorkload stops and deletes container workloads and cleans up - // network and bundle overlays on ateom. + // network and bundle overlays on the worker. TerminateWorkload(ctx context.Context, in *TerminateWorkloadRequest, opts ...grpc.CallOption) (*TerminateWorkloadResponse, error) } -type ateomClient struct { +type workerClient struct { cc grpc.ClientConnInterface } -func NewAteomClient(cc grpc.ClientConnInterface) AteomClient { - return &ateomClient{cc} +func NewWorkerClient(cc grpc.ClientConnInterface) WorkerClient { + return &workerClient{cc} } -func (c *ateomClient) RunWorkload(ctx context.Context, in *RunWorkloadRequest, opts ...grpc.CallOption) (*RunWorkloadResponse, error) { +func (c *workerClient) RunWorkload(ctx context.Context, in *RunWorkloadRequest, opts ...grpc.CallOption) (*RunWorkloadResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(RunWorkloadResponse) - err := c.cc.Invoke(ctx, Ateom_RunWorkload_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Worker_RunWorkload_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomClient) CheckpointWorkload(ctx context.Context, in *CheckpointWorkloadRequest, opts ...grpc.CallOption) (*CheckpointWorkloadResponse, error) { +func (c *workerClient) CheckpointWorkload(ctx context.Context, in *CheckpointWorkloadRequest, opts ...grpc.CallOption) (*CheckpointWorkloadResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(CheckpointWorkloadResponse) - err := c.cc.Invoke(ctx, Ateom_CheckpointWorkload_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Worker_CheckpointWorkload_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomClient) RestoreWorkload(ctx context.Context, in *RestoreWorkloadRequest, opts ...grpc.CallOption) (*RestoreWorkloadResponse, error) { +func (c *workerClient) RestoreWorkload(ctx context.Context, in *RestoreWorkloadRequest, opts ...grpc.CallOption) (*RestoreWorkloadResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(RestoreWorkloadResponse) - err := c.cc.Invoke(ctx, Ateom_RestoreWorkload_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Worker_RestoreWorkload_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomClient) GetWorkloadStats(ctx context.Context, in *GetWorkloadStatsRequest, opts ...grpc.CallOption) (*GetWorkloadStatsResponse, error) { +func (c *workerClient) GetWorkloadStats(ctx context.Context, in *GetWorkloadStatsRequest, opts ...grpc.CallOption) (*GetWorkloadStatsResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(GetWorkloadStatsResponse) - err := c.cc.Invoke(ctx, Ateom_GetWorkloadStats_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Worker_GetWorkloadStats_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomClient) GetActiveWorkloadStats(ctx context.Context, in *GetActiveWorkloadStatsRequest, opts ...grpc.CallOption) (*GetActiveWorkloadStatsResponse, error) { +func (c *workerClient) GetActiveWorkloadStats(ctx context.Context, in *GetActiveWorkloadStatsRequest, opts ...grpc.CallOption) (*GetActiveWorkloadStatsResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(GetActiveWorkloadStatsResponse) - err := c.cc.Invoke(ctx, Ateom_GetActiveWorkloadStats_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Worker_GetActiveWorkloadStats_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *ateomClient) TerminateWorkload(ctx context.Context, in *TerminateWorkloadRequest, opts ...grpc.CallOption) (*TerminateWorkloadResponse, error) { +func (c *workerClient) TerminateWorkload(ctx context.Context, in *TerminateWorkloadRequest, opts ...grpc.CallOption) (*TerminateWorkloadResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) out := new(TerminateWorkloadResponse) - err := c.cc.Invoke(ctx, Ateom_TerminateWorkload_FullMethodName, in, out, cOpts...) + err := c.cc.Invoke(ctx, Worker_TerminateWorkload_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -// AteomServer is the server API for Ateom service. -// All implementations must embed UnimplementedAteomServer +// WorkerServer is the server API for Worker service. +// All implementations must embed UnimplementedWorkerServer // for forward compatibility. // -// Ateom is the interface to control a single gVisor (or, in the future microVM) +// Worker is the interface to control a single gVisor (or, in the future microVM) // guest inside a worker pod. // -// Each ateom server has two main states, "available" and "executing". +// Each worker has two main states, "available" and "executing". // -// When the ateom is "available", the substrate control plane is free to either +// When the worker is "available", the substrate control plane is free to either // boot a new workload (using RunWorkload), or restore an existing workload from -// a checkpoint (using RestoreWorkload). These calls move the ateom into +// a checkpoint (using RestoreWorkload). These calls move the worker into // "executing" state. // -// When the ateom is "executing", the substrate control plane can checkpoint the -// running workload (with CheckpointWorkload). This moves the ateom back to +// When the worker is "executing", the substrate control plane can checkpoint the +// running workload (with CheckpointWorkload). This moves the worker back to // "free" state. -type AteomServer interface { - // RunWorkload tells ateom to begin running a new workload (one or more +type WorkerServer interface { + // RunWorkload tells the worker to begin running a new workload (one or more // containers, potentially with shared filesystems). RunWorkload(context.Context, *RunWorkloadRequest) (*RunWorkloadResponse, error) - // CheckpointWorkload tells ateom to save the current state of the running + // CheckpointWorkload tells the worker to save the current state of the running // workload to object storage, and then completely reset itself to a blank // state (back to "available" state.) CheckpointWorkload(context.Context, *CheckpointWorkloadRequest) (*CheckpointWorkloadResponse, error) // RestoreWorkload restores a workload from checkpoint that was previously - // written by CheckpointWorkload. Ateom will handle downloading the correct + // written by CheckpointWorkload. The worker will handle downloading the correct // gVisor / runsc version to match the checkpoint. RestoreWorkload(context.Context, *RestoreWorkloadRequest) (*RestoreWorkloadResponse, error) // GetWorkloadStats returns a point-in-time resource-usage sample for the - // workload this ateom is currently executing. + // workload this worker is currently executing. // - // It is a pure read: unlike the three calls above it does not move the ateom + // It is a pure read: unlike the three calls above it does not move the worker // between "available" and "executing", so it is safe to call on a timer for // the whole lifetime of a workload. It also does not wait on them. Those - // three serialize on a mutex an ateom holds for a whole boot, restore, or + // three serialize on a mutex a worker holds for a whole boot, restore, or // checkpoint; this call reads its state from outside that mutex, so a poll // landing in the middle of one answers immediately instead of going quiet for // the duration -- which would silence the poller during exactly the phases @@ -239,30 +239,30 @@ type AteomServer interface { // Two ways it declines to give a sample, and they ask different things of the // caller: // - // * NOT_FOUND -- this ateom is not executing the actor in the request. It + // * NOT_FOUND -- this worker is not executing the actor in the request. It // may be "available", or a recycled worker may have moved on to a // different actor. Retrying on the same timer will not change the answer: // the caller's worker-to-actor mapping is stale and wants re-resolving. // - // * FAILED_PRECONDITION -- this ateom is executing the requested actor but + // * FAILED_PRECONDITION -- this worker is executing the requested actor but // has no sample to give yet. It accepts an actor before the sandbox it // will measure exists, so a poll landing in the boot lands here. Read it // as "no numbers right now", not as "the actor is gone": it is transient, // and the caller should skip this sample and take the next one. // // The split is worth the two codes because the identity is retained from the - // moment the ateom accepts the actor, on both runtimes. That is what makes + // moment the worker accepts the actor, on both runtimes. That is what makes // "booting" distinguishable from "not here" at all, and it means a workload // that dies during boot is attributable rather than anonymous. GetWorkloadStats(context.Context, *GetWorkloadStatsRequest) (*GetWorkloadStatsResponse, error) - // GetActiveWorkloadStats reports every actor this ateom hosts, without + // GetActiveWorkloadStats reports every actor this worker hosts, without // asserting an identity. It is the discovery read for a - // scraper that enumerates ateoms and holds no worker-to-actor mapping; + // scraper that enumerates workers and holds no worker-to-actor mapping; // GetWorkloadStats above is the verified read for a caller that must be // answered about a specific actor. // - // It serves each actor's latest sample from the ateom's own sampling timer, - // so a poll reads no sandbox and has no error path: an "available" ateom + // It serves each actor's latest sample from the worker's own sampling timer, + // so a poll reads no sandbox and has no error path: an "available" worker // answers an empty samples list, and a workload with nothing measured yet // (a boot, a restore, or a guest the last sweep did not reach) appears as a // sample carrying its attribution with source = STATS_SOURCE_UNSPECIFIED -- @@ -277,197 +277,197 @@ type AteomServer interface { // it is safe on a timer and does not touch the lifecycle mutex. GetActiveWorkloadStats(context.Context, *GetActiveWorkloadStatsRequest) (*GetActiveWorkloadStatsResponse, error) // TerminateWorkload stops and deletes container workloads and cleans up - // network and bundle overlays on ateom. + // network and bundle overlays on the worker. TerminateWorkload(context.Context, *TerminateWorkloadRequest) (*TerminateWorkloadResponse, error) - mustEmbedUnimplementedAteomServer() + mustEmbedUnimplementedWorkerServer() } -// UnimplementedAteomServer must be embedded to have +// UnimplementedWorkerServer must be embedded to have // forward compatible implementations. // // NOTE: this should be embedded by value instead of pointer to avoid a nil // pointer dereference when methods are called. -type UnimplementedAteomServer struct{} +type UnimplementedWorkerServer struct{} -func (UnimplementedAteomServer) RunWorkload(context.Context, *RunWorkloadRequest) (*RunWorkloadResponse, error) { +func (UnimplementedWorkerServer) RunWorkload(context.Context, *RunWorkloadRequest) (*RunWorkloadResponse, error) { return nil, status.Error(codes.Unimplemented, "method RunWorkload not implemented") } -func (UnimplementedAteomServer) CheckpointWorkload(context.Context, *CheckpointWorkloadRequest) (*CheckpointWorkloadResponse, error) { +func (UnimplementedWorkerServer) CheckpointWorkload(context.Context, *CheckpointWorkloadRequest) (*CheckpointWorkloadResponse, error) { return nil, status.Error(codes.Unimplemented, "method CheckpointWorkload not implemented") } -func (UnimplementedAteomServer) RestoreWorkload(context.Context, *RestoreWorkloadRequest) (*RestoreWorkloadResponse, error) { +func (UnimplementedWorkerServer) RestoreWorkload(context.Context, *RestoreWorkloadRequest) (*RestoreWorkloadResponse, error) { return nil, status.Error(codes.Unimplemented, "method RestoreWorkload not implemented") } -func (UnimplementedAteomServer) GetWorkloadStats(context.Context, *GetWorkloadStatsRequest) (*GetWorkloadStatsResponse, error) { +func (UnimplementedWorkerServer) GetWorkloadStats(context.Context, *GetWorkloadStatsRequest) (*GetWorkloadStatsResponse, error) { return nil, status.Error(codes.Unimplemented, "method GetWorkloadStats not implemented") } -func (UnimplementedAteomServer) GetActiveWorkloadStats(context.Context, *GetActiveWorkloadStatsRequest) (*GetActiveWorkloadStatsResponse, error) { +func (UnimplementedWorkerServer) GetActiveWorkloadStats(context.Context, *GetActiveWorkloadStatsRequest) (*GetActiveWorkloadStatsResponse, error) { return nil, status.Error(codes.Unimplemented, "method GetActiveWorkloadStats not implemented") } -func (UnimplementedAteomServer) TerminateWorkload(context.Context, *TerminateWorkloadRequest) (*TerminateWorkloadResponse, error) { +func (UnimplementedWorkerServer) TerminateWorkload(context.Context, *TerminateWorkloadRequest) (*TerminateWorkloadResponse, error) { return nil, status.Error(codes.Unimplemented, "method TerminateWorkload not implemented") } -func (UnimplementedAteomServer) mustEmbedUnimplementedAteomServer() {} -func (UnimplementedAteomServer) testEmbeddedByValue() {} +func (UnimplementedWorkerServer) mustEmbedUnimplementedWorkerServer() {} +func (UnimplementedWorkerServer) testEmbeddedByValue() {} -// UnsafeAteomServer may be embedded to opt out of forward compatibility for this service. -// Use of this interface is not recommended, as added methods to AteomServer will +// UnsafeWorkerServer may be embedded to opt out of forward compatibility for this service. +// Use of this interface is not recommended, as added methods to WorkerServer will // result in compilation errors. -type UnsafeAteomServer interface { - mustEmbedUnimplementedAteomServer() +type UnsafeWorkerServer interface { + mustEmbedUnimplementedWorkerServer() } -func RegisterAteomServer(s grpc.ServiceRegistrar, srv AteomServer) { - // If the following call panics, it indicates UnimplementedAteomServer was +func RegisterWorkerServer(s grpc.ServiceRegistrar, srv WorkerServer) { + // If the following call panics, it indicates UnimplementedWorkerServer was // embedded by pointer and is nil. This will cause panics if an // unimplemented method is ever invoked, so we test this at initialization // time to prevent it from happening at runtime later due to I/O. if t, ok := srv.(interface{ testEmbeddedByValue() }); ok { t.testEmbeddedByValue() } - s.RegisterService(&Ateom_ServiceDesc, srv) + s.RegisterService(&Worker_ServiceDesc, srv) } -func _Ateom_RunWorkload_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Worker_RunWorkload_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(RunWorkloadRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomServer).RunWorkload(ctx, in) + return srv.(WorkerServer).RunWorkload(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: Ateom_RunWorkload_FullMethodName, + FullMethod: Worker_RunWorkload_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomServer).RunWorkload(ctx, req.(*RunWorkloadRequest)) + return srv.(WorkerServer).RunWorkload(ctx, req.(*RunWorkloadRequest)) } return interceptor(ctx, in, info, handler) } -func _Ateom_CheckpointWorkload_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Worker_CheckpointWorkload_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(CheckpointWorkloadRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomServer).CheckpointWorkload(ctx, in) + return srv.(WorkerServer).CheckpointWorkload(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: Ateom_CheckpointWorkload_FullMethodName, + FullMethod: Worker_CheckpointWorkload_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomServer).CheckpointWorkload(ctx, req.(*CheckpointWorkloadRequest)) + return srv.(WorkerServer).CheckpointWorkload(ctx, req.(*CheckpointWorkloadRequest)) } return interceptor(ctx, in, info, handler) } -func _Ateom_RestoreWorkload_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Worker_RestoreWorkload_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(RestoreWorkloadRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomServer).RestoreWorkload(ctx, in) + return srv.(WorkerServer).RestoreWorkload(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: Ateom_RestoreWorkload_FullMethodName, + FullMethod: Worker_RestoreWorkload_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomServer).RestoreWorkload(ctx, req.(*RestoreWorkloadRequest)) + return srv.(WorkerServer).RestoreWorkload(ctx, req.(*RestoreWorkloadRequest)) } return interceptor(ctx, in, info, handler) } -func _Ateom_GetWorkloadStats_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Worker_GetWorkloadStats_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(GetWorkloadStatsRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomServer).GetWorkloadStats(ctx, in) + return srv.(WorkerServer).GetWorkloadStats(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: Ateom_GetWorkloadStats_FullMethodName, + FullMethod: Worker_GetWorkloadStats_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomServer).GetWorkloadStats(ctx, req.(*GetWorkloadStatsRequest)) + return srv.(WorkerServer).GetWorkloadStats(ctx, req.(*GetWorkloadStatsRequest)) } return interceptor(ctx, in, info, handler) } -func _Ateom_GetActiveWorkloadStats_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Worker_GetActiveWorkloadStats_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(GetActiveWorkloadStatsRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomServer).GetActiveWorkloadStats(ctx, in) + return srv.(WorkerServer).GetActiveWorkloadStats(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: Ateom_GetActiveWorkloadStats_FullMethodName, + FullMethod: Worker_GetActiveWorkloadStats_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomServer).GetActiveWorkloadStats(ctx, req.(*GetActiveWorkloadStatsRequest)) + return srv.(WorkerServer).GetActiveWorkloadStats(ctx, req.(*GetActiveWorkloadStatsRequest)) } return interceptor(ctx, in, info, handler) } -func _Ateom_TerminateWorkload_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { +func _Worker_TerminateWorkload_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { in := new(TerminateWorkloadRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(AteomServer).TerminateWorkload(ctx, in) + return srv.(WorkerServer).TerminateWorkload(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: Ateom_TerminateWorkload_FullMethodName, + FullMethod: Worker_TerminateWorkload_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(AteomServer).TerminateWorkload(ctx, req.(*TerminateWorkloadRequest)) + return srv.(WorkerServer).TerminateWorkload(ctx, req.(*TerminateWorkloadRequest)) } return interceptor(ctx, in, info, handler) } -// Ateom_ServiceDesc is the grpc.ServiceDesc for Ateom service. +// Worker_ServiceDesc is the grpc.ServiceDesc for Worker service. // It's only intended for direct use with grpc.RegisterService, // and not to be introspected or modified (even as a copy) -var Ateom_ServiceDesc = grpc.ServiceDesc{ - ServiceName: "ateom.Ateom", - HandlerType: (*AteomServer)(nil), +var Worker_ServiceDesc = grpc.ServiceDesc{ + ServiceName: "ateworker.Worker", + HandlerType: (*WorkerServer)(nil), Methods: []grpc.MethodDesc{ { MethodName: "RunWorkload", - Handler: _Ateom_RunWorkload_Handler, + Handler: _Worker_RunWorkload_Handler, }, { MethodName: "CheckpointWorkload", - Handler: _Ateom_CheckpointWorkload_Handler, + Handler: _Worker_CheckpointWorkload_Handler, }, { MethodName: "RestoreWorkload", - Handler: _Ateom_RestoreWorkload_Handler, + Handler: _Worker_RestoreWorkload_Handler, }, { MethodName: "GetWorkloadStats", - Handler: _Ateom_GetWorkloadStats_Handler, + Handler: _Worker_GetWorkloadStats_Handler, }, { MethodName: "GetActiveWorkloadStats", - Handler: _Ateom_GetActiveWorkloadStats_Handler, + Handler: _Worker_GetActiveWorkloadStats_Handler, }, { MethodName: "TerminateWorkload", - Handler: _Ateom_TerminateWorkload_Handler, + Handler: _Worker_TerminateWorkload_Handler, }, }, Streams: []grpc.StreamDesc{}, - Metadata: "ateom.proto", + Metadata: "ateworker.proto", } diff --git a/internal/protoredact/protoredact_test.go b/internal/protoredact/protoredact_test.go index ada4fd8151..d3a6425e95 100644 --- a/internal/protoredact/protoredact_test.go +++ b/internal/protoredact/protoredact_test.go @@ -24,7 +24,7 @@ import ( "testing" "github.com/agent-substrate/substrate/internal/proto/ateletpb" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/proto/glutton" "github.com/agent-substrate/substrate/internal/proto/grpcechopb" "github.com/agent-substrate/substrate/internal/protoredact" @@ -379,7 +379,7 @@ func TestNeedsRedactionIsSafeUnderConcurrentFirstUse(t *testing.T) { var ourProtoFiles = []protoreflect.FileDescriptor{ ateapipb.File_ateapi_proto, ateletpb.File_atelet_proto, - ateompb.File_ateom_proto, + ateworkerpb.File_ateworker_proto, credproviderpb.File_credprovider_proto, glutton.File_glutton_proto, grpcechopb.File_grpcecho_proto, @@ -417,10 +417,10 @@ func TestDebugRedactFieldsArePinned(t *testing.T) { want := map[string]bool{ "ateapi.EnvVar.value": true, "ateapi.MintActorJwtResponse.actor_jwt": true, - "ateapi.MintAteomActorJwtResponse.actor_jwt": true, + "ateapi.MintWorkerActorJwtResponse.actor_jwt": true, "atelet.AssetFile.url": true, "atelet.EnvEntry.value": true, - "ateom.ContainerSpec.env": true, + "ateworker.ContainerSpec.env": true, "credprovider.FetchSecretResponse.opaque_bytes": true, "objectstore.v1.FetchAssetRequest.asset_uri": true, "objectstore.v1.FetchSnapshotRequest.actor_jwt": true, diff --git a/internal/resources/actorattribution.go b/internal/resources/actorattribution.go index cc2463f9b5..a1907690d5 100644 --- a/internal/resources/actorattribution.go +++ b/internal/resources/actorattribution.go @@ -19,7 +19,7 @@ import "github.com/agent-substrate/substrate/pkg/proto/ateapipb" // ActorAttribution is what telemetry about an actor is attributed to: an // ActorRef plus the two things a ref does not carry, the server-assigned uid and // the template the actor was built from. Shared by ateattr, actorlog, and -// ateom's usage sampling so those producers cannot drift apart. +// worker's usage sampling so those producers cannot drift apart. // // Unrelated to the credential sense of "actor identity" elsewhere in the repo // (ateapi's ActorIdentity service, substratex509) diff --git a/internal/resources/spiffe.go b/internal/resources/spiffe.go index 300cf37ab6..cfa73b96b6 100644 --- a/internal/resources/spiffe.go +++ b/internal/resources/spiffe.go @@ -65,36 +65,36 @@ func ActorRefFromActorSPIFFEURL(u *url.URL) (ActorRef, error) { return ActorRef{Atespace: atespace, Name: name}, nil } -// AteomForActorSPIFFEID returns -// "spiffe://substrate-actor.local/ateom-for-actor//", which +// WorkerForActorSPIFFEID returns +// "spiffe://substrate-actor.local/worker-for-actor//", which // ateapi mints into the actor certificate's URI SAN. -func AteomForActorSPIFFEID(r ActorRef) *url.URL { +func WorkerForActorSPIFFEID(r ActorRef) *url.URL { return &url.URL{ Scheme: "spiffe", Host: ActorSPIFFETrustDomain, // TODO(identity): Prefix with "atunnel" to prevent // confusion between atunnel and an actor pretending to be // an atunnel. - Path: path.Join("ateom-for-actor", r.Atespace, r.Name), + Path: path.Join("worker-for-actor", r.Atespace, r.Name), } } -// ActorRefFromAteomForActorSPIFFEID parses an ID built by AteomForActorSPIFFEID. -func ActorRefFromAteomForActorSPIFFEID(id string) (ActorRef, error) { +// ActorRefFromWorkerForActorSPIFFEID parses an ID built by WorkerForActorSPIFFEID. +func ActorRefFromWorkerForActorSPIFFEID(id string) (ActorRef, error) { u, err := url.Parse(id) if err != nil { - return ActorRef{}, fmt.Errorf("invalid ateom-for-actor SPIFFE ID %q: %w", id, err) + return ActorRef{}, fmt.Errorf("invalid worker-for-actor SPIFFE ID %q: %w", id, err) } - return ActorRefFromAteomForActorSPIFFEURL(u) + return ActorRefFromWorkerForActorSPIFFEURL(u) } -func ActorRefFromAteomForActorSPIFFEURL(u *url.URL) (ActorRef, error) { +func ActorRefFromWorkerForActorSPIFFEURL(u *url.URL) (ActorRef, error) { if u.Scheme != "spiffe" || u.Host != ActorSPIFFETrustDomain || u.User != nil || u.RawQuery != "" || u.Fragment != "" { - return ActorRef{}, fmt.Errorf("%q does not have format spiffe:///ateom-for-actor//", u.String()) + return ActorRef{}, fmt.Errorf("%q does not have format spiffe:///worker-for-actor//", u.String()) } segments := strings.Split(strings.TrimPrefix(u.Path, "/"), "/") - if len(segments) != 3 || segments[0] != "ateom-for-actor" { - return ActorRef{}, fmt.Errorf("%q does not have format spiffe:///ateom-for-actor//", u.String()) + if len(segments) != 3 || segments[0] != "worker-for-actor" { + return ActorRef{}, fmt.Errorf("%q does not have format spiffe:///worker-for-actor//", u.String()) } atespace, name := segments[1], segments[2] if !IsValidResourceName(atespace) { diff --git a/internal/resources/spiffe_test.go b/internal/resources/spiffe_test.go index 3bcfc27a46..cba8f812ee 100644 --- a/internal/resources/spiffe_test.go +++ b/internal/resources/spiffe_test.go @@ -39,7 +39,7 @@ func TestActorRefFromSPIFFEIDRejects(t *testing.T) { "spiffe://substrate-actor.local/atespace/team/actor/agent/", "spiffe://substrate-actor.local/atespace/te%2Fam/actor/agent", } { - if ref, err := ActorRefFromAteomForActorSPIFFEID(id); err == nil { + if ref, err := ActorRefFromWorkerForActorSPIFFEID(id); err == nil { t.Errorf("ActorRefFromSPIFFEID(%q) = %v, want error", id, ref) } } @@ -84,20 +84,20 @@ func TestActorRoundtrip(t *testing.T) { } } -func TestAteomForActorRoundtrip(t *testing.T) { +func TestWorkerForActorRoundtrip(t *testing.T) { testCases := []struct { uri string wantRef ActorRef }{ { - uri: "spiffe://substrate-actor.local/ateom-for-actor/foo/bar", + uri: "spiffe://substrate-actor.local/worker-for-actor/foo/bar", wantRef: ActorRef{ Atespace: "foo", Name: "bar", }, }, { - uri: "spiffe://substrate-actor.local/ateom-for-actor/fd6cab8c-17c8-4c9e-8893-28e28aff724b/045841a7-5dcb-47eb-a76f-6d8460bfe009", + uri: "spiffe://substrate-actor.local/worker-for-actor/fd6cab8c-17c8-4c9e-8893-28e28aff724b/045841a7-5dcb-47eb-a76f-6d8460bfe009", wantRef: ActorRef{ Atespace: "fd6cab8c-17c8-4c9e-8893-28e28aff724b", Name: "045841a7-5dcb-47eb-a76f-6d8460bfe009", @@ -106,7 +106,7 @@ func TestAteomForActorRoundtrip(t *testing.T) { } for _, tc := range testCases { t.Run(tc.uri, func(t *testing.T) { - ref, err := ActorRefFromAteomForActorSPIFFEID(tc.uri) + ref, err := ActorRefFromWorkerForActorSPIFFEID(tc.uri) if err != nil { t.Fatalf("Unexpected error parsing uri: %v", err) } @@ -115,7 +115,7 @@ func TestAteomForActorRoundtrip(t *testing.T) { t.Fatalf("Bad ref; diff (-got +want)\n%s", diff) } - gotURI := AteomForActorSPIFFEID(ref) + gotURI := WorkerForActorSPIFFEID(ref) if diff := cmp.Diff(gotURI.String(), tc.uri); diff != "" { t.Fatalf("URI didn't round-trip; diff (-got +want)\n%s", diff) } diff --git a/internal/resources/validate.go b/internal/resources/validate.go index ec9cb78d1e..03002aad97 100644 --- a/internal/resources/validate.go +++ b/internal/resources/validate.go @@ -28,7 +28,7 @@ import ( "strings" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/volumepath" "github.com/agent-substrate/substrate/pkg/sandboxclass" "github.com/distribution/reference" @@ -44,7 +44,7 @@ import ( // ToGRPCStatusError turns validation errors into the InvalidArgument error an // RPC handler responds with. Callers check len(errs) > 0 first. // -// TODO: Delete once atelet's AteomSupport server returns apierrors, and use +// TODO: Delete once atelet's WorkerSupport server returns apierrors, and use // ToAPIError instead. func ToGRPCStatusError(errs field.ErrorList) error { return status.Error(codes.InvalidArgument, errs.ToAggregate().Error()) @@ -96,7 +96,7 @@ func IsValidResourceName(name string) bool { } // ValidateWorkerPodUID rejects a worker pod UID that could escape the host -// path built from it: the ateom control socket (.../ateoms//ateom.sock). +// path built from it: the worker control socket (.../workers//worker.sock). // Kubernetes pod UIDs are UUIDs, which are valid DNS-1123 labels, so a label // check accepts every legitimate value while rejecting separators and "..". func ValidateWorkerPodUID(workerPodUID string) error { @@ -106,9 +106,9 @@ func ValidateWorkerPodUID(workerPodUID string) error { return nil } -// ValidateActorDirs checks that every directory atelet passes to ateom is +// ValidateActorDirs checks that every directory atelet passes to worker is // set, absolute and clean. -func ValidateActorDirs(actorDirs *ateompb.ActorDirs, fldPath *field.Path) field.ErrorList { +func ValidateActorDirs(actorDirs *ateworkerpb.ActorDirs, fldPath *field.Path) field.ErrorList { if actorDirs == nil { return field.ErrorList{field.Required(fldPath, "")} } @@ -131,8 +131,8 @@ func ValidateActorDirs(actorDirs *ateompb.ActorDirs, fldPath *field.Path) field. // fidelity the calling sandbox runtime cannot serve. Each runtime passes the // levels it implements; anything else, including an unset fidelity, is // refused before the request touches the guest. -func ValidateSnapshotFidelity(fidelity ateompb.SnapshotFidelity, supported []ateompb.SnapshotFidelity, fldPath *field.Path) field.ErrorList { - if fidelity == ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED { +func ValidateSnapshotFidelity(fidelity ateworkerpb.SnapshotFidelity, supported []ateworkerpb.SnapshotFidelity, fldPath *field.Path) field.ErrorList { + if fidelity == ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED { return field.ErrorList{field.Required(fldPath, "")} } names := make([]string, 0, len(supported)) @@ -165,7 +165,7 @@ func validateAbsDir(dir string, fldPath *field.Path) field.ErrorList { } // ValidateRuntimeAssetPath ensures p is a regular file under root, with no symlinks -// ateom runs these as root, so they must be assets atelet fetched into root +// worker runs these as root, so they must be assets atelet fetched into root func ValidateRuntimeAssetPath(root, p string, fldPath *field.Path) field.ErrorList { if p == "" { return field.ErrorList{field.Required(fldPath, "")} @@ -191,7 +191,7 @@ func ValidateRuntimeAssetPath(root, p string, fldPath *field.Path) field.ErrorLi // ValidateAssetHash ensures a sandbox asset's SHA-256 hash is exactly 64 hex // characters before it is used to build the on-disk asset path -// (static-files/asset-) and, on a cache hit, returned for ateom to +// (static-files/asset-) and, on a cache hit, returned for worker to // execute. Without this, a hash containing path separators or ".." could // point the cache-hit early return (and the download target) at an arbitrary // binary outside the static-files dir. diff --git a/internal/resources/validate_test.go b/internal/resources/validate_test.go index d923207ae6..0cf25d8261 100644 --- a/internal/resources/validate_test.go +++ b/internal/resources/validate_test.go @@ -23,7 +23,7 @@ import ( "testing" "github.com/agent-substrate/substrate/internal/apierror" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/pkg/proto/ateapipb" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" @@ -313,8 +313,8 @@ func TestValidateLimit(t *testing.T) { } } -func testActorDirs() *ateompb.ActorDirs { - return &ateompb.ActorDirs{ +func testActorDirs() *ateworkerpb.ActorDirs { + return &ateworkerpb.ActorDirs{ RootDir: "/node/actors/a", OciBundleDir: "/node/actors/a/bundles", CheckpointDir: "/node/actors/a/checkpoint-state", @@ -326,23 +326,23 @@ func testActorDirs() *ateompb.ActorDirs { } func TestValidateSnapshotFidelity(t *testing.T) { - volumes := ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES - rootfs := ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS - memory := ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY - all := []ateompb.SnapshotFidelity{volumes, rootfs, memory} - noRootfs := []ateompb.SnapshotFidelity{volumes, memory} + volumes := ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_VOLUMES + rootfs := ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS + memory := ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_MEMORY + all := []ateworkerpb.SnapshotFidelity{volumes, rootfs, memory} + noRootfs := []ateworkerpb.SnapshotFidelity{volumes, memory} for _, tc := range []struct { name string - fidelity ateompb.SnapshotFidelity - supported []ateompb.SnapshotFidelity + fidelity ateworkerpb.SnapshotFidelity + supported []ateworkerpb.SnapshotFidelity wantErr bool }{ {"volumes", volumes, all, false}, {"memory", memory, all, false}, {"rootfs where supported", rootfs, all, false}, {"rootfs where unsupported", rootfs, noRootfs, true}, - {"unspecified", ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, all, true}, - {"outside the enum", ateompb.SnapshotFidelity(99), all, true}, + {"unspecified", ateworkerpb.SnapshotFidelity_SNAPSHOT_FIDELITY_UNSPECIFIED, all, true}, + {"outside the enum", ateworkerpb.SnapshotFidelity(99), all, true}, } { t.Run(tc.name, func(t *testing.T) { errs := ValidateSnapshotFidelity(tc.fidelity, tc.supported, field.NewPath("fidelity")) @@ -361,22 +361,25 @@ func TestValidateSnapshotFidelity(t *testing.T) { func TestValidateActorDirs(t *testing.T) { for _, tc := range []struct { name string - mutate func(*ateompb.ActorDirs) *ateompb.ActorDirs + mutate func(*ateworkerpb.ActorDirs) *ateworkerpb.ActorDirs // wantField is the field the one expected error names; empty means valid. wantField string }{ - {"valid", func(actorDirs *ateompb.ActorDirs) *ateompb.ActorDirs { return actorDirs }, ""}, - {"nil", func(*ateompb.ActorDirs) *ateompb.ActorDirs { return nil }, "actor_dirs"}, - {"missing dir", func(actorDirs *ateompb.ActorDirs) *ateompb.ActorDirs { actorDirs.RestoreDir = ""; return actorDirs }, "actor_dirs.restore_dir"}, - {"relative dir", func(actorDirs *ateompb.ActorDirs) *ateompb.ActorDirs { + {"valid", func(actorDirs *ateworkerpb.ActorDirs) *ateworkerpb.ActorDirs { return actorDirs }, ""}, + {"nil", func(*ateworkerpb.ActorDirs) *ateworkerpb.ActorDirs { return nil }, "actor_dirs"}, + {"missing dir", func(actorDirs *ateworkerpb.ActorDirs) *ateworkerpb.ActorDirs { + actorDirs.RestoreDir = "" + return actorDirs + }, "actor_dirs.restore_dir"}, + {"relative dir", func(actorDirs *ateworkerpb.ActorDirs) *ateworkerpb.ActorDirs { actorDirs.OciBundleDir = "bundles" return actorDirs }, "actor_dirs.oci_bundle_dir"}, - {"unclean dir", func(actorDirs *ateompb.ActorDirs) *ateompb.ActorDirs { + {"unclean dir", func(actorDirs *ateworkerpb.ActorDirs) *ateworkerpb.ActorDirs { actorDirs.CheckpointDir = "/node/actors/a/../b/checkpoint-state" return actorDirs }, "actor_dirs.checkpoint_dir"}, - {"relative root", func(actorDirs *ateompb.ActorDirs) *ateompb.ActorDirs { + {"relative root", func(actorDirs *ateworkerpb.ActorDirs) *ateworkerpb.ActorDirs { actorDirs.RootDir = "node/actors/a" return actorDirs }, "actor_dirs.root_dir"}, diff --git a/internal/sandboxversions/sandboxversions.go b/internal/sandboxversions/sandboxversions.go index d9dcf6496f..3ae871c9f0 100644 --- a/internal/sandboxversions/sandboxversions.go +++ b/internal/sandboxversions/sandboxversions.go @@ -13,7 +13,7 @@ // limitations under the License. // Package sandboxversions reads and writes a worker's sandbox versions file, -// which atelet keeps in the worker's directory (nodepath.AteomPath) as +// which atelet keeps in the worker's directory (nodepath.WorkerPath) as // nodepath.SandboxVersionsFileName. package sandboxversions @@ -27,7 +27,7 @@ import ( // File is the content of a worker's sandbox versions file: the sandbox // runtimes the worker can run, as the compatibility attributes atelet computed -// for them. The ateom adds each runtime's attributes to the ones it probes +// for them. The worker adds each runtime's attributes to the ones it probes // itself and registers the result as RegisterWorkerRequest's default_runtime // and restorable_runtimes. type File struct { diff --git a/internal/serverboot/logging.go b/internal/serverboot/logging.go index 5f44f19dee..d50142d607 100644 --- a/internal/serverboot/logging.go +++ b/internal/serverboot/logging.go @@ -48,7 +48,7 @@ type LoggingOptions struct { // ResolveLogsExporter. Exporter Exporters // ExporterConn and RelayCapable are the logs counterpart of the same fields - // on TracingOptions: ateom passes its relay connection and marks itself + // on TracingOptions: worker passes its relay connection and marks itself // relay-capable so the resource carries ate.otlp.relay. ExporterConn *grpc.ClientConn RelayCapable bool diff --git a/internal/serverboot/logging_test.go b/internal/serverboot/logging_test.go index ed17344eab..03c4286e4b 100644 --- a/internal/serverboot/logging_test.go +++ b/internal/serverboot/logging_test.go @@ -107,7 +107,7 @@ func emittedResource(t *testing.T, relayCapable bool, conn *grpc.ClientConn) map t.Helper() proc := withCaptureProcessor(t) lp, err := newLoggerProvider(context.Background(), LoggingOptions{ - ServiceName: "ateom-gvisor", + ServiceName: "ateworker-gvisor", Exporter: Exporters{ExporterOTLP: true}, ExporterConn: conn, RelayCapable: relayCapable, diff --git a/internal/serverboot/serverboot.go b/internal/serverboot/serverboot.go index bdd0c61c24..cf52e4ef36 100644 --- a/internal/serverboot/serverboot.go +++ b/internal/serverboot/serverboot.go @@ -13,8 +13,8 @@ // limitations under the License. // Package serverboot collects the startup boilerplate shared by the -// long-running substrate server binaries (ateapi, atelet, ateom-gvisor, -// ateom-microvm): slog wiring, OTel tracer + meter providers, a Prometheus +// long-running substrate server binaries (ateapi, atelet, ateworker-gvisor, +// ateworker-microvm): slog wiring, OTel tracer + meter providers, a Prometheus // /metrics server, a /readyz and /healthz server for the kubelet probes, and a // couple of small helpers for startup fail-fast. package serverboot @@ -61,7 +61,7 @@ func InitLogger() { // InitLoggerWithWriter is InitLogger with an explicit destination. Use it to share // one synchronized writer between the runtime logger and a separate writer (e.g. -// ateom's actor-log forwarder) so their lines don't interleave. +// worker's actor-log forwarder) so their lines don't interleave. func InitLoggerWithWriter(w io.Writer) { slog.SetDefault(slog.New(contextlogging.NewHandler(slog.NewJSONHandler(w, &slog.HandlerOptions{Level: &logLevel})))) } @@ -116,7 +116,7 @@ func newResource(ctx context.Context, serviceName string, extraAttrs ...attribut // gets no attribute at all; true means it did, and conn says whether it got it. // // The distinction matters because a nil conn on a relay-capable component is -// exactly the degraded case worth alerting on: the ateom asked for the relay, +// exactly the degraded case worth alerting on: the worker asked for the relay, // could not dial it, and is now exporting over the worker pod's own network. // Collapsing that into the same "no attribute" bucket as atecontroller would // hide it. @@ -140,7 +140,7 @@ type TracingOptions struct { // default. Sampling TraceSampling // ExporterConn, when non-nil, is the connection the OTLP exporter sends - // over, instead of dialing OTEL_EXPORTER_OTLP_ENDPOINT itself. ateom passes + // over, instead of dialing OTEL_EXPORTER_OTLP_ENDPOINT itself. worker passes // the unix socket to atelet's relay (internal/otlprelay) so a worker pod // exports without a network path of its own; nil keeps the direct dial. // @@ -148,9 +148,9 @@ type TracingOptions struct { // connection it did not create. ExporterConn *grpc.ClientConn // RelayCapable marks a component that is meant to export through the relay, - // whether or not it managed to (see relayAttrs). Only the ateoms set it. It + // whether or not it managed to (see relayAttrs). Only the workers set it. It // is separate from ExporterConn because a nil conn on its own cannot tell - // "the ateom tried and fell back" from "this component never had a relay". + // "the worker tried and fell back" from "this component never had a relay". RelayCapable bool } @@ -305,9 +305,9 @@ func InitMetricsBridged(ctx context.Context, serviceName string, reg interface { } // InitMetricsPushOnlyVia is InitMetrics without the Prometheus reader, for a -// binary that runs no metrics HTTP server of its own (ateom): a pull reader +// binary that runs no metrics HTTP server of its own (worker): a pull reader // would collect into a registry nothing serves. conn is the metrics counterpart -// of TracingOptions.ExporterConn: ateom passes atelet's relay socket +// of TracingOptions.ExporterConn: worker passes atelet's relay socket // (internal/otlprelay) so the worker pod needs no network path of its own; a // nil conn keeps the direct dial to OTEL_EXPORTER_OTLP_ENDPOINT. // @@ -362,7 +362,7 @@ func newMeterProvider(ctx context.Context, serviceName string, push, relayCapabl // dropRPCServerAddress removes server.address from the otelgrpc instruments. // otelgrpc reads it from the remote address of the client connection: the -// socket path atelet dials holds the ateom pod UID, and a TCP address is the IP +// socket path atelet dials holds the worker pod UID, and a TCP address is the IP // of one server pod. Either way the key has one value per pod. server.port stays, // because it has one value per service. var dropRPCServerAddress = sdkmetric.NewView( diff --git a/internal/serverboot/serverboot_test.go b/internal/serverboot/serverboot_test.go index c17e796d23..48763c8700 100644 --- a/internal/serverboot/serverboot_test.go +++ b/internal/serverboot/serverboot_test.go @@ -118,7 +118,7 @@ func TestRelayAttrs(t *testing.T) { if tc.conn { conn = lazyConn(t) } - res, err := newResource(context.Background(), "ateom-gvisor", relayAttrs(tc.relayCapable, conn)...) + res, err := newResource(context.Background(), "ateworker-gvisor", relayAttrs(tc.relayCapable, conn)...) if err != nil { t.Fatalf("newResource: %v", err) } @@ -143,7 +143,7 @@ func TestRelayAttrs(t *testing.T) { func collectedResource(t *testing.T, relayCapable bool, conn *grpc.ClientConn) map[string]string { t.Helper() reader := sdkmetric.NewManualReader() - mp, err := newMeterProvider(context.Background(), "ateom-gvisor", true, relayCapable, conn, nil, reader) + mp, err := newMeterProvider(context.Background(), "ateworker-gvisor", true, relayCapable, conn, nil, reader) if err != nil { t.Fatalf("newMeterProvider: %v", err) } diff --git a/internal/sizing/sizing.go b/internal/sizing/sizing.go index e48a205f1d..8122981c19 100644 --- a/internal/sizing/sizing.go +++ b/internal/sizing/sizing.go @@ -13,10 +13,10 @@ // limitations under the License. // Package sizing right-sizes a sandbox to the actor's declared resource limits. -// The actor's limits arrive over the ateom RPCs (RunWorkload / RestoreWorkload) -// as a SandboxSize. ateom-gvisor calls ApplyToOCISpec to write them into the +// The actor's limits arrive over the worker RPCs (RunWorkload / RestoreWorkload) +// as a SandboxSize. ateworker-gvisor calls ApplyToOCISpec to write them into the // sandbox container's OCI spec, which runsc then applies to the host cgroup -// leaf. ateom-microvm does not call ApplyToOCISpec: it uses SandboxSize only to +// leaf. ateworker-microvm does not call ApplyToOCISpec: it uses SandboxSize only to // size the VM itself (VCPUs, and MemoryBytes via resolveGuestMemMiB). A // micro-VM container's own cgroup limit comes instead from that container's // declared `spec.containers[].resources`, written into its OCI spec by atelet. @@ -33,8 +33,7 @@ const ( ) // SandboxSize is the sandbox's target size, derived from the actor's declared -// resource limits. A zero field means "unset": the caller keeps its own default -// (kata.DefaultMemoryMiB / kata.DefaultVCPUs for the micro-VM, unlimited for gVisor). +// resource limits. A zero field means "unset". type SandboxSize struct { // MilliCPU is the CPU limit in millicores (1000 = one core), or 0 if unset. MilliCPU int64 @@ -43,8 +42,8 @@ type SandboxSize struct { } // FromLimits builds a SandboxSize from an actor's declared limits (millicores and -// bytes) as carried on the ateom RPCs. It is runtime-agnostic; both ateom-gvisor -// and ateom-microvm call it. Negative values are clamped to zero ("unset"). +// bytes) as carried on the worker RPCs. It is runtime-agnostic; both ateworker-gvisor +// and ateworker-microvm call it. Negative values are clamped to zero ("unset"). func FromLimits(milliCPU, memoryBytes int64) SandboxSize { if milliCPU < 0 { milliCPU = 0 @@ -72,7 +71,7 @@ func (s SandboxSize) VCPUs() int { // ApplyToOCISpec writes the pod's CPU/memory limits into the container OCI spec's // linux.resources so the sandbox cgroup is created with the right values. This is -// the gVisor sandbox-cgroup path: only ateom-gvisor calls it, so runsc applies the +// the gVisor sandbox-cgroup path: only ateworker-gvisor calls it, so runsc applies the // result to the host cgroup leaf. Fields that are unset in SandboxSize are left // untouched, preserving any existing values on the spec. func (s SandboxSize) ApplyToOCISpec(spec *specs.Spec) { diff --git a/internal/tarutil/fifo_linux.go b/internal/tarutil/fifo_linux.go index 79b1856810..481603e379 100644 --- a/internal/tarutil/fifo_linux.go +++ b/internal/tarutil/fifo_linux.go @@ -53,7 +53,7 @@ func createFifo(root *os.Root, name string, mode os.FileMode) error { // Device nodes matter here because overlayfs records a deleted lower-layer // file as a 0:0 character device ("whiteout") in the upper — dropping one at // extraction would resurrect the deleted file on restore. mknod requires -// privilege; extraction runs as root in ateom, and the tests gate on it. +// privilege; extraction runs as root in worker, and the tests gate on it. func createDevice(root *os.Root, name string, hdr *tar.Header, mode os.FileMode) error { dir, base := filepath.Split(name) if dir == "" { diff --git a/internal/tarutil/owner_linux.go b/internal/tarutil/owner_linux.go index 9d33a0785b..6c9ac4db8e 100644 --- a/internal/tarutil/owner_linux.go +++ b/internal/tarutil/owner_linux.go @@ -67,7 +67,7 @@ func setOwner(hdr *tar.Header, info os.FileInfo) { // lchownEntry restores an entry's ownership without following symlinks. // -// Changing a file's owner requires privilege. The production caller (ateom) is +// Changing a file's owner requires privilege. The production caller (worker) is // root in its worker pod, so ownership is restored faithfully and any EPERM // there signals a real problem worth failing on. An unprivileged process cannot // chown at all, so rather than making the package unusable outside a root diff --git a/internal/tarutil/tarutil.go b/internal/tarutil/tarutil.go index 0dee7b7acc..2940c5ac2f 100644 --- a/internal/tarutil/tarutil.go +++ b/internal/tarutil/tarutil.go @@ -21,7 +21,7 @@ // records). // // It exists for snapshotting durable-dir volumes and rootfs overlay uppers -// (see cmd/ateom-microvm): the contents are written by the sandboxed workload +// (see cmd/ateworker-microvm): the contents are written by the sandboxed workload // under arbitrary uids, shipped to object storage, and restored — possibly // onto another node — where the workload must see them unchanged. The upper // is why device nodes and overlay xattrs matter: the host kernel's overlayfs @@ -534,7 +534,7 @@ func restoreMeta(root *os.Root, name string, hdr *tar.Header) error { // restoreOverlayXattrs re-applies the xattrs recorded in the entry's PAX // records (writeTree's SCHILY.xattr.*) that keepXattr allows. Writing -// trusted.* requires CAP_SYS_ADMIN; extraction runs as root in ateom, and the +// trusted.* requires CAP_SYS_ADMIN; extraction runs as root in worker, and the // tests gate on it. // // The target is addressed THROUGH its parent directory opened via root (the @@ -600,7 +600,7 @@ func keepXattr(attr string) bool { // readOverlayXattrs returns the extended attributes of name under root that // keepXattr allows. Filesystems without xattr support report none rather than // failing: tarutil archives arbitrary workload trees. Reading trusted.* -// requires root, which archiving in ateom always has. +// requires root, which archiving in worker always has. // // name is addressed through its parent directory opened via root, like // restoreOverlayXattrs, and the L* calls do not follow a final symlink. diff --git a/internal/tarutil/tarutil_test.go b/internal/tarutil/tarutil_test.go index aefd9640fd..a7ed7f6c0b 100644 --- a/internal/tarutil/tarutil_test.go +++ b/internal/tarutil/tarutil_test.go @@ -452,7 +452,7 @@ func TestRoundTripDeviceNode(t *testing.T) { // records a replaced lower-layer directory as a trusted.overlay.opaque xattr // on the upper directory, and dropping it would merge the old lower contents // back in after a restore. trusted.* needs CAP_SYS_ADMIN on both sides, as -// ateom has. +// worker has. func TestRoundTripTrustedOverlayXattrs(t *testing.T) { roottest.Require(t, "trusted.* xattrs require root") diff --git a/internal/volume/mock.go b/internal/volume/mock.go index 320781432a..b745e819c8 100644 --- a/internal/volume/mock.go +++ b/internal/volume/mock.go @@ -24,7 +24,7 @@ import ( "github.com/agent-substrate/substrate/internal/nodepath" ) -// Use a directory that is shared between atelet and ateom but not cleaned up by atelet +// Use a directory that is shared between atelet and worker but not cleaned up by atelet var mockVolumeDirectories string = filepath.Join(nodepath.BasePath, "mockvolumes") var ( diff --git a/internal/wakeupprobe/wakeupprobe.go b/internal/wakeupprobe/wakeupprobe.go index 1f4d233ecc..6995cc284a 100644 --- a/internal/wakeupprobe/wakeupprobe.go +++ b/internal/wakeupprobe/wakeupprobe.go @@ -13,7 +13,7 @@ // limitations under the License. // Package wakeupprobe polls a container's HTTP wakeup endpoint from inside an -// ateom. The intent is to detect the moment a container's HTTP server +// worker. The intent is to detect the moment a container's HTTP server // starts accepting connections with single-millisecond latency: while the // server is still booting the kernel returns RST in microseconds, so a // sub-millisecond poll loop spends almost no time blocked, and once the @@ -31,7 +31,7 @@ import ( "strings" "time" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "golang.org/x/sync/errgroup" ) @@ -70,7 +70,7 @@ func newClient(dial DialFunc) *http.Client { // WaitAll blocks until every container with a wakeup probe set reports 200 through dial, // or returns the first error. Containers without a probe are skipped (their // absence means "no wakeup gate"). -func WaitAll(ctx context.Context, containers []*ateompb.Container, actorIP string, dial DialFunc) error { +func WaitAll(ctx context.Context, containers []*ateworkerpb.Container, actorIP string, dial DialFunc) error { g, gctx := errgroup.WithContext(ctx) for _, ac := range containers { if ac.GetWakeupProbe() == nil { @@ -86,7 +86,7 @@ func WaitAll(ctx context.Context, containers []*ateompb.Container, actorIP strin // Wait polls the configured HTTP endpoint through dial until it returns 200, // the context is cancelled, or the overall deadline is exceeded. -func Wait(ctx context.Context, containerName string, probe *ateompb.WakeupProbe, actorIP string, dial DialFunc) error { +func Wait(ctx context.Context, containerName string, probe *ateworkerpb.WakeupProbe, actorIP string, dial DialFunc) error { url, err := URL(probe, actorIP) if err != nil { return fmt.Errorf("invalid wakeup probe config for %q: %w", containerName, err) @@ -141,7 +141,7 @@ func Wait(ctx context.Context, containerName string, probe *ateompb.WakeupProbe, } } -func pollTimeout(probe *ateompb.WakeupProbe) (time.Duration, error) { +func pollTimeout(probe *ateworkerpb.WakeupProbe) (time.Duration, error) { s := probe.GetTimeoutSeconds() if s <= 0 { return 0, fmt.Errorf("timeout_seconds must be positive, got %d", s) @@ -169,7 +169,7 @@ func tryOnce(ctx context.Context, client *http.Client, url string) (bool, error) // URL builds the probe endpoint URL. Exported so callers and tests can // validate a probe spec before kicking off a Wait. -func URL(probe *ateompb.WakeupProbe, actorIP string) (string, error) { +func URL(probe *ateworkerpb.WakeupProbe, actorIP string) (string, error) { hg := probe.GetHttpGet() if hg == nil { return "", fmt.Errorf("httpGet is required") diff --git a/internal/wakeupprobe/wakeupprobe_test.go b/internal/wakeupprobe/wakeupprobe_test.go index 481a05d9f5..fd43825824 100644 --- a/internal/wakeupprobe/wakeupprobe_test.go +++ b/internal/wakeupprobe/wakeupprobe_test.go @@ -26,50 +26,50 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) func TestURL(t *testing.T) { tests := []struct { name string - probe *ateompb.WakeupProbe + probe *ateworkerpb.WakeupProbe actorIP string want string wantErr bool }{ { name: "missing path", - probe: &ateompb.WakeupProbe{HttpGet: &ateompb.HttpGetAction{Port: 8080}}, + probe: &ateworkerpb.WakeupProbe{HttpGet: &ateworkerpb.HttpGetAction{Port: 8080}}, actorIP: "169.254.17.2", wantErr: true, }, { name: "explicit path", - probe: &ateompb.WakeupProbe{HttpGet: &ateompb.HttpGetAction{Path: "/health", Port: 9000}}, + probe: &ateworkerpb.WakeupProbe{HttpGet: &ateworkerpb.HttpGetAction{Path: "/health", Port: 9000}}, actorIP: "169.254.17.2", want: "http://169.254.17.2:9000/health", }, { name: "path without leading slash is normalized", - probe: &ateompb.WakeupProbe{HttpGet: &ateompb.HttpGetAction{Path: "ready", Port: 80}}, + probe: &ateworkerpb.WakeupProbe{HttpGet: &ateworkerpb.HttpGetAction{Path: "ready", Port: 80}}, actorIP: "10.0.0.1", want: "http://10.0.0.1:80/ready", }, { name: "missing httpGet", - probe: &ateompb.WakeupProbe{}, + probe: &ateworkerpb.WakeupProbe{}, actorIP: "1.2.3.4", wantErr: true, }, { name: "port zero", - probe: &ateompb.WakeupProbe{HttpGet: &ateompb.HttpGetAction{Port: 0}}, + probe: &ateworkerpb.WakeupProbe{HttpGet: &ateworkerpb.HttpGetAction{Port: 0}}, actorIP: "1.2.3.4", wantErr: true, }, { name: "port too large", - probe: &ateompb.WakeupProbe{HttpGet: &ateompb.HttpGetAction{Port: 70000}}, + probe: &ateworkerpb.WakeupProbe{HttpGet: &ateworkerpb.HttpGetAction{Port: 70000}}, actorIP: "1.2.3.4", wantErr: true, }, @@ -98,7 +98,7 @@ func TestWait_ReturnsOnFirst200(t *testing.T) { defer srv.Close() ip, port := splitHostPort(t, srv.URL) - probe := &ateompb.WakeupProbe{HttpGet: &ateompb.HttpGetAction{Path: "/readyz", Port: int32(port)}, TimeoutSeconds: 1} + probe := &ateworkerpb.WakeupProbe{HttpGet: &ateworkerpb.HttpGetAction{Path: "/readyz", Port: int32(port)}, TimeoutSeconds: 1} ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) defer cancel() @@ -119,7 +119,7 @@ func TestWait_WaitsForServerToBecomeReady(t *testing.T) { defer srv.Close() ip, port := splitHostPort(t, srv.URL) - probe := &ateompb.WakeupProbe{HttpGet: &ateompb.HttpGetAction{Path: "/readyz", Port: int32(port)}, TimeoutSeconds: 1} + probe := &ateworkerpb.WakeupProbe{HttpGet: &ateworkerpb.HttpGetAction{Path: "/readyz", Port: int32(port)}, TimeoutSeconds: 1} flipAt := time.Now().Add(50 * time.Millisecond) go func() { @@ -146,7 +146,7 @@ func TestWait_ContextCancellation(t *testing.T) { // Bind a port and immediately close to ensure connect-refused, so the // poll loop is exercised but no server ever returns 200. port := pickFreePort(t) - probe := &ateompb.WakeupProbe{HttpGet: &ateompb.HttpGetAction{Path: "/readyz", Port: int32(port)}, TimeoutSeconds: 1} + probe := &ateworkerpb.WakeupProbe{HttpGet: &ateworkerpb.HttpGetAction{Path: "/readyz", Port: int32(port)}, TimeoutSeconds: 1} ctx, cancel := context.WithCancel(context.Background()) go func() { @@ -163,23 +163,23 @@ func TestWait_ContextCancellation(t *testing.T) { func TestPollTimeout(t *testing.T) { tests := []struct { name string - probe *ateompb.WakeupProbe + probe *ateworkerpb.WakeupProbe want time.Duration wantErr bool }{ { name: "unset is rejected", - probe: &ateompb.WakeupProbe{}, + probe: &ateworkerpb.WakeupProbe{}, wantErr: true, }, { name: "explicit value is honored", - probe: &ateompb.WakeupProbe{TimeoutSeconds: 300}, + probe: &ateworkerpb.WakeupProbe{TimeoutSeconds: 300}, want: 300 * time.Second, }, { name: "negative is rejected", - probe: &ateompb.WakeupProbe{TimeoutSeconds: -1}, + probe: &ateworkerpb.WakeupProbe{TimeoutSeconds: -1}, wantErr: true, }, } @@ -200,8 +200,8 @@ func TestWait_GivesUpAtProbeTimeout(t *testing.T) { // Nothing ever binds this port, so the poll loop runs until the // probe's own deadline rather than the package default. port := pickFreePort(t) - probe := &ateompb.WakeupProbe{ - HttpGet: &ateompb.HttpGetAction{Path: "/readyz", Port: int32(port)}, + probe := &ateworkerpb.WakeupProbe{ + HttpGet: &ateworkerpb.HttpGetAction{Path: "/readyz", Port: int32(port)}, TimeoutSeconds: 1, } @@ -226,7 +226,7 @@ func TestWait_GivesUpAtProbeTimeout(t *testing.T) { func TestWaitAll_SkipsContainersWithoutProbe(t *testing.T) { // No server bound, but no probes => should return nil immediately. - containers := []*ateompb.Container{ + containers := []*ateworkerpb.Container{ {Name: "a"}, {Name: "b"}, } diff --git a/internal/workercapacity/workercapacity.go b/internal/workercapacity/workercapacity.go new file mode 100644 index 0000000000..8ebab0d1bd --- /dev/null +++ b/internal/workercapacity/workercapacity.go @@ -0,0 +1,23 @@ +// Copyright 2026 Google LLC +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Package workercapacity names the files the atecontroller projects into the +// worker container from the downward API, in milli-cores and bytes. +package workercapacity + +const ( + MountPath = "/run/worker-capacity" + CPULimitFile = "cpu_milli" + MemoryLimitFile = "memory_bytes" +) diff --git a/internal/ateomstats/activation.go b/internal/workerstats/activation.go similarity index 87% rename from internal/ateomstats/activation.go rename to internal/workerstats/activation.go index 5662d82de1..f91e4a552f 100644 --- a/internal/ateomstats/activation.go +++ b/internal/workerstats/activation.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomstats +package workerstats import ( "context" @@ -21,11 +21,11 @@ import ( "sync" "time" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // Activation is the usage state of one activation of an actor: one Run or -// Restore. An ateom creates one each time it hosts an actor, so a re-hosted +// Restore. A worker creates one each time it hosts an actor, so a re-hosted // actor starts a new one. // // It is safe for concurrent use: the stats reads, the sampler, and the @@ -49,8 +49,8 @@ type Activation struct { last map[string]uint64 // latest is what the discovery read serves, pending included; measured // is the newest sample with numbers, for the final record. - latest *ateompb.WorkloadStatsSample - measured *ateompb.WorkloadStatsSample + latest *ateworkerpb.WorkloadStatsSample + measured *ateworkerpb.WorkloadStatsSample // initialDone is set once the initial reading is taken or has failed, and // ended by the final record. Periodic records are written only between the // two, so the initial record comes first and the final one last. @@ -70,7 +70,7 @@ func NewActivation(now time.Time, resumesCounters bool) *Activation { func (a *Activation) Epoch() int64 { return a.epoch } // WithEpoch sets the epoch on s, a sample with no CPU reading, and returns it. -func (a *Activation) WithEpoch(s *ateompb.WorkloadStatsSample) *ateompb.WorkloadStatsSample { +func (a *Activation) WithEpoch(s *ateworkerpb.WorkloadStatsSample) *ateworkerpb.WorkloadStatsSample { s.EpochUnixNano = a.epoch return s } @@ -88,7 +88,7 @@ var errNoSample = errors.New("usage reading returned no sample") // not seen before, in the first reading or a later one, counts in full only // when the activation's counters start at zero. Waiting for another reading to // finish ends with ctx. -func (a *Activation) Measure(ctx context.Context, read func() (*ateompb.WorkloadStatsSample, map[string]uint64, error)) (*ateompb.WorkloadStatsSample, error) { +func (a *Activation) Measure(ctx context.Context, read func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error)) (*ateworkerpb.WorkloadStatsSample, error) { select { case a.readLock <- struct{}{}: case <-ctx.Done(): @@ -144,19 +144,19 @@ func addSat(a, b uint64) uint64 { // Store records s as the latest sample, and as the newest measured one when it // has numbers, each unless a newer one is already there. -func (a *Activation) Store(s *ateompb.WorkloadStatsSample) { +func (a *Activation) Store(s *ateworkerpb.WorkloadStatsSample) { a.mu.Lock() defer a.mu.Unlock() a.store(s) } -func (a *Activation) store(s *ateompb.WorkloadStatsSample) { +func (a *Activation) store(s *ateworkerpb.WorkloadStatsSample) { if a.latest == nil || s.GetObservedAtUnixNano() >= a.latest.GetObservedAtUnixNano() { a.latest = s } // Compared on its own: a pending sample stored first must not discard a // measured one read earlier. - if s.GetSource() != ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED && + if s.GetSource() != ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED && (a.measured == nil || s.GetObservedAtUnixNano() >= a.measured.GetObservedAtUnixNano()) { a.measured = s } @@ -167,7 +167,7 @@ func (a *Activation) store(s *ateompb.WorkloadStatsSample) { // was written first; nil s means the reading failed, and periodic records start // without an initial one. write runs under the activation's lock and must not // block or call into a. -func (a *Activation) Initial(s *ateompb.WorkloadStatsSample, write func()) { +func (a *Activation) Initial(s *ateworkerpb.WorkloadStatsSample, write func()) { a.mu.Lock() defer a.mu.Unlock() if a.initialDone { @@ -193,7 +193,7 @@ func (a *Activation) Sampling() bool { // initial reading and its final record, which a sweep that read s may have // raced. write runs under the activation's lock and must not block or call // into a. -func (a *Activation) Periodic(s *ateompb.WorkloadStatsSample, write func()) { +func (a *Activation) Periodic(s *ateworkerpb.WorkloadStatsSample, write func()) { a.mu.Lock() defer a.mu.Unlock() if !a.initialDone || a.ended { @@ -206,7 +206,7 @@ func (a *Activation) Periodic(s *ateompb.WorkloadStatsSample, write func()) { // Final marks the activation ended and runs write with the newest measured // sample, or nil when there is none, once: a later call does nothing. write // runs under the activation's lock and must not block or call into a. -func (a *Activation) Final(write func(measured *ateompb.WorkloadStatsSample)) { +func (a *Activation) Final(write func(measured *ateworkerpb.WorkloadStatsSample)) { a.mu.Lock() defer a.mu.Unlock() if a.ended { @@ -218,7 +218,7 @@ func (a *Activation) Final(write func(measured *ateompb.WorkloadStatsSample)) { // Latest is the last stored sample, or nil before the first. Callers must not // modify it. -func (a *Activation) Latest() *ateompb.WorkloadStatsSample { +func (a *Activation) Latest() *ateworkerpb.WorkloadStatsSample { a.mu.Lock() defer a.mu.Unlock() return a.latest diff --git a/internal/ateomstats/activation_test.go b/internal/workerstats/activation_test.go similarity index 76% rename from internal/ateomstats/activation_test.go rename to internal/workerstats/activation_test.go index 195d57b1a6..3a9366cb48 100644 --- a/internal/ateomstats/activation_test.go +++ b/internal/workerstats/activation_test.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomstats +package workerstats import ( "context" @@ -24,7 +24,7 @@ import ( "testing" "time" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) // reading is one read of counters by name; a single counter reads as "". @@ -33,8 +33,8 @@ type reading map[string]uint64 // measure folds r into a as one reading and returns the activation's CPU. func measure(t *testing.T, a *Activation, r reading) uint64 { t.Helper() - s, err := a.Measure(context.Background(), func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { - s := &ateompb.WorkloadStatsSample{Source: ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT} + s, err := a.Measure(context.Background(), func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { + s := &ateworkerpb.WorkloadStatsSample{Source: ateworkerpb.StatsSource_STATS_SOURCE_GUEST_AGENT} if len(r) == 1 { if v, ok := r[""]; ok { s.CpuUsageUsec = v @@ -53,11 +53,11 @@ func TestActivationEpoch(t *testing.T) { t.Parallel() start := time.Unix(1700, 5) a := NewActivation(start, false) - if got := a.WithEpoch(&ateompb.WorkloadStatsSample{}).GetEpochUnixNano(); got != start.UnixNano() { + if got := a.WithEpoch(&ateworkerpb.WorkloadStatsSample{}).GetEpochUnixNano(); got != start.UnixNano() { t.Errorf("pending epoch = %d, want %d", got, start.UnixNano()) } - s, _ := a.Measure(context.Background(), func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { - return &ateompb.WorkloadStatsSample{Source: ateompb.StatsSource_STATS_SOURCE_CGROUP}, nil, nil + s, _ := a.Measure(context.Background(), func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { + return &ateworkerpb.WorkloadStatsSample{Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP}, nil, nil }) if s.GetEpochUnixNano() != start.UnixNano() { t.Errorf("measured epoch = %d, want %d", s.GetEpochUnixNano(), start.UnixNano()) @@ -110,7 +110,7 @@ func TestActivationMeasureError(t *testing.T) { t.Parallel() a := NewActivation(time.Now(), true) want := errors.New("guest gone") - if _, err := a.Measure(context.Background(), func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { return nil, nil, want }); !errors.Is(err, want) { + if _, err := a.Measure(context.Background(), func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { return nil, nil, want }); !errors.Is(err, want) { t.Fatalf("Measure() error = %v, want %v", err, want) } if got := measure(t, a, reading{"": 700}); got != 0 { @@ -124,8 +124,8 @@ func TestActivationStore(t *testing.T) { if a.Latest() != nil { t.Fatal("Latest() before any Store is not nil") } - measured := func(at int64) *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, ObservedAtUnixNano: at} + measured := func(at int64) *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, ObservedAtUnixNano: at} } newer, older := measured(20), measured(10) a.Store(newer) @@ -133,12 +133,12 @@ func TestActivationStore(t *testing.T) { if a.Latest() != newer { t.Error("an older sample replaced a newer one") } - p := &ateompb.WorkloadStatsSample{ObservedAtUnixNano: 30} + p := &ateworkerpb.WorkloadStatsSample{ObservedAtUnixNano: 30} a.Store(p) if a.Latest() != p { t.Error("Latest() is not the newest sample") } - a.Final(func(m *ateompb.WorkloadStatsSample) { + a.Final(func(m *ateworkerpb.WorkloadStatsSample) { if m != newer { t.Errorf("final sample = %v, want the newest measured one", m) } @@ -148,13 +148,13 @@ func TestActivationStore(t *testing.T) { func TestActivationInitialAfterFinal(t *testing.T) { t.Parallel() a := NewActivation(time.Now(), false) - a.Final(func(m *ateompb.WorkloadStatsSample) { + a.Final(func(m *ateworkerpb.WorkloadStatsSample) { if m != nil { t.Errorf("final sample with nothing measured = %v, want nil", m) } }) wrote := false - a.Initial(&ateompb.WorkloadStatsSample{Source: ateompb.StatsSource_STATS_SOURCE_CGROUP}, func() { wrote = true }) + a.Initial(&ateworkerpb.WorkloadStatsSample{Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP}, func() { wrote = true }) if wrote || a.Latest() != nil { t.Error("initial record written after the final one") } @@ -174,13 +174,13 @@ func TestActivationConcurrentMeasure(t *testing.T) { go func() { defer wg.Done() <-start - _, err := a.Measure(context.Background(), func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { + _, err := a.Measure(context.Background(), func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { if inRead.Add(1) > 1 { t.Error("readings overlapped") } defer inRead.Add(-1) time.Sleep(time.Millisecond) - return &ateompb.WorkloadStatsSample{Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, CpuUsageUsec: raw.Add(10)}, nil, nil + return &ateworkerpb.WorkloadStatsSample{Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, CpuUsageUsec: raw.Add(10)}, nil, nil }) if err != nil { t.Error(err) @@ -200,16 +200,16 @@ func TestActivationMeasureGivesUp(t *testing.T) { t.Parallel() a := NewActivation(time.Now(), false) held, release := make(chan struct{}), make(chan struct{}) - go a.Measure(context.Background(), func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { + go a.Measure(context.Background(), func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { close(held) <-release - return &ateompb.WorkloadStatsSample{}, nil, nil + return &ateworkerpb.WorkloadStatsSample{}, nil, nil }) <-held defer close(release) ctx, cancel := context.WithCancel(context.Background()) cancel() - if _, err := a.Measure(ctx, func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { + if _, err := a.Measure(ctx, func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { t.Error("read ran while another held the lock") return nil, nil, nil }); !errors.Is(err, context.Canceled) { @@ -221,8 +221,8 @@ func TestActivationMeasureGivesUp(t *testing.T) { // only between the initial reading and the final record. func TestActivationPeriodicWindow(t *testing.T) { t.Parallel() - sample := func(at int64) *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, ObservedAtUnixNano: at} + sample := func(at int64) *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, ObservedAtUnixNano: at} } a := NewActivation(time.Now(), false) var wrote []string @@ -238,7 +238,7 @@ func TestActivationPeriodicWindow(t *testing.T) { } a.Periodic(sample(3), write("periodic")) a.Initial(sample(4), write("initial-again")) - a.Final(func(*ateompb.WorkloadStatsSample) { wrote = append(wrote, "final") }) + a.Final(func(*ateworkerpb.WorkloadStatsSample) { wrote = append(wrote, "final") }) if a.Sampling() { t.Error("Sampling() after the final record = true") } @@ -259,7 +259,7 @@ func TestActivationFailedInitialOpensWindow(t *testing.T) { t.Fatal("Sampling() after a failed initial reading = false") } wrote := false - a.Periodic(&ateompb.WorkloadStatsSample{ObservedAtUnixNano: 1}, func() { wrote = true }) + a.Periodic(&ateworkerpb.WorkloadStatsSample{ObservedAtUnixNano: 1}, func() { wrote = true }) if !wrote { t.Error("no periodic record after a failed initial reading") } @@ -270,8 +270,8 @@ func TestActivationFinalOnce(t *testing.T) { t.Parallel() a := NewActivation(time.Now(), false) writes := 0 - a.Final(func(*ateompb.WorkloadStatsSample) { writes++ }) - a.Final(func(*ateompb.WorkloadStatsSample) { writes++ }) + a.Final(func(*ateworkerpb.WorkloadStatsSample) { writes++ }) + a.Final(func(*ateworkerpb.WorkloadStatsSample) { writes++ }) if writes != 1 { t.Errorf("Final wrote %d records, want 1", writes) } @@ -283,15 +283,15 @@ func TestActivationFinalOnce(t *testing.T) { func TestActivationStoreMeasuredAfterNewerPending(t *testing.T) { t.Parallel() a := NewActivation(time.Now(), false) - older := &ateompb.WorkloadStatsSample{Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, ObservedAtUnixNano: 10} - a.Store(&ateompb.WorkloadStatsSample{Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, ObservedAtUnixNano: 5}) - pending := &ateompb.WorkloadStatsSample{ObservedAtUnixNano: 20} + older := &ateworkerpb.WorkloadStatsSample{Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, ObservedAtUnixNano: 10} + a.Store(&ateworkerpb.WorkloadStatsSample{Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, ObservedAtUnixNano: 5}) + pending := &ateworkerpb.WorkloadStatsSample{ObservedAtUnixNano: 20} a.Store(pending) a.Store(older) if a.Latest() != pending { t.Errorf("Latest() = %v, want the newer pending sample", a.Latest()) } - a.Final(func(m *ateompb.WorkloadStatsSample) { + a.Final(func(m *ateworkerpb.WorkloadStatsSample) { if m != older { t.Errorf("final sample = %v, want the measured sample read at 10", m) } @@ -303,7 +303,7 @@ func TestActivationStoreMeasuredAfterNewerPending(t *testing.T) { func TestActivationMeasureNilSample(t *testing.T) { t.Parallel() a := NewActivation(time.Now(), true) - if _, err := a.Measure(context.Background(), func() (*ateompb.WorkloadStatsSample, map[string]uint64, error) { + if _, err := a.Measure(context.Background(), func() (*ateworkerpb.WorkloadStatsSample, map[string]uint64, error) { return nil, nil, nil }); !errors.Is(err, errNoSample) { t.Fatalf("Measure() error = %v, want errNoSample", err) diff --git a/internal/ateomstats/actorattribution.go b/internal/workerstats/actorattribution.go similarity index 72% rename from internal/ateomstats/actorattribution.go rename to internal/workerstats/actorattribution.go index db52f98de5..332ec026c6 100644 --- a/internal/ateomstats/actorattribution.go +++ b/internal/workerstats/actorattribution.go @@ -12,20 +12,20 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateomstats holds the usage telemetry both ateom runtimes share: the -// attribution an ateom retains for each actor, the per-activation state behind +// Package workerstats holds the usage telemetry both worker runtimes share: the +// attribution a worker retains for each actor, the per-activation state behind // every sample, the sampler, and the ate.actor.usage_sampled records. The // per-runtime measurement reads live with their runtimes (the cgroup read is // only meaningful inside the gVisor worker's cgroup namespace, the guest-agent // read only over the micro-VM's vsock). -package ateomstats +package workerstats import ( - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" ) -// attributionSource is the attribution-bearing subset of the ateom requests +// attributionSource is the attribution-bearing subset of the worker requests // that name the actor they act on. type attributionSource interface { GetAtespace() string @@ -36,13 +36,13 @@ type attributionSource interface { } var ( - _ attributionSource = (*ateompb.RunWorkloadRequest)(nil) - _ attributionSource = (*ateompb.RestoreWorkloadRequest)(nil) - _ attributionSource = (*ateompb.CheckpointWorkloadRequest)(nil) - _ attributionSource = (*ateompb.TerminateWorkloadRequest)(nil) + _ attributionSource = (*ateworkerpb.RunWorkloadRequest)(nil) + _ attributionSource = (*ateworkerpb.RestoreWorkloadRequest)(nil) + _ attributionSource = (*ateworkerpb.CheckpointWorkloadRequest)(nil) + _ attributionSource = (*ateworkerpb.TerminateWorkloadRequest)(nil) ) -// ActorAttributionFromRequest extracts the attribution an ateom should retain +// ActorAttributionFromRequest extracts the attribution a worker should retain // for the workload req starts: nothing later in the run, checkpoint, or restore // paths carries it, and GetWorkloadStats needs it. func ActorAttributionFromRequest(req attributionSource) resources.ActorAttribution { diff --git a/internal/ateomstats/actorattribution_test.go b/internal/workerstats/actorattribution_test.go similarity index 86% rename from internal/ateomstats/actorattribution_test.go rename to internal/workerstats/actorattribution_test.go index a0e9971e18..51163f9bf4 100644 --- a/internal/ateomstats/actorattribution_test.go +++ b/internal/workerstats/actorattribution_test.go @@ -12,14 +12,14 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomstats +package workerstats import ( "testing" "github.com/google/go-cmp/cmp" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" ) @@ -42,7 +42,7 @@ func TestActorAttributionFromRequest(t *testing.T) { }{ { name: "run request", - req: &ateompb.RunWorkloadRequest{ + req: &ateworkerpb.RunWorkloadRequest{ Atespace: "atespace-a", ActorName: "actor-b", ActorUid: "uid-c", @@ -56,7 +56,7 @@ func TestActorAttributionFromRequest(t *testing.T) { // a sample taken after a restore must be attributable to the actor // it was taken before the checkpoint. name: "restore request", - req: &ateompb.RestoreWorkloadRequest{ + req: &ateworkerpb.RestoreWorkloadRequest{ Atespace: "atespace-a", ActorName: "actor-b", ActorUid: "uid-c", @@ -67,7 +67,7 @@ func TestActorAttributionFromRequest(t *testing.T) { }, { name: "checkpoint request", - req: &ateompb.CheckpointWorkloadRequest{ + req: &ateworkerpb.CheckpointWorkloadRequest{ Atespace: "atespace-a", ActorName: "actor-b", ActorUid: "uid-c", @@ -78,7 +78,7 @@ func TestActorAttributionFromRequest(t *testing.T) { }, { name: "terminate request", - req: &ateompb.TerminateWorkloadRequest{ + req: &ateworkerpb.TerminateWorkloadRequest{ Atespace: "atespace-a", ActorName: "actor-b", ActorUid: "uid-c", @@ -89,12 +89,12 @@ func TestActorAttributionFromRequest(t *testing.T) { }, { name: "empty run request", - req: &ateompb.RunWorkloadRequest{}, + req: &ateworkerpb.RunWorkloadRequest{}, want: resources.ActorAttribution{}, }, { name: "empty terminate request", - req: &ateompb.TerminateWorkloadRequest{}, + req: &ateworkerpb.TerminateWorkloadRequest{}, want: resources.ActorAttribution{}, }, { @@ -103,22 +103,22 @@ func TestActorAttributionFromRequest(t *testing.T) { // is the honest answer. Pinned so a hand-written getter or a switch // to direct field access does not turn this into a panic. name: "nil run request", - req: (*ateompb.RunWorkloadRequest)(nil), + req: (*ateworkerpb.RunWorkloadRequest)(nil), want: resources.ActorAttribution{}, }, { name: "nil restore request", - req: (*ateompb.RestoreWorkloadRequest)(nil), + req: (*ateworkerpb.RestoreWorkloadRequest)(nil), want: resources.ActorAttribution{}, }, { name: "nil checkpoint request", - req: (*ateompb.CheckpointWorkloadRequest)(nil), + req: (*ateworkerpb.CheckpointWorkloadRequest)(nil), want: resources.ActorAttribution{}, }, { name: "nil terminate request", - req: (*ateompb.TerminateWorkloadRequest)(nil), + req: (*ateworkerpb.TerminateWorkloadRequest)(nil), want: resources.ActorAttribution{}, }, } diff --git a/internal/ateomstats/usage.go b/internal/workerstats/usage.go similarity index 88% rename from internal/ateomstats/usage.go rename to internal/workerstats/usage.go index 20646f0243..7f20843aff 100644 --- a/internal/ateomstats/usage.go +++ b/internal/workerstats/usage.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomstats +package workerstats import ( "context" @@ -27,11 +27,11 @@ import ( "github.com/agent-substrate/substrate/internal/actorevent" "github.com/agent-substrate/substrate/internal/ateattr" "github.com/agent-substrate/substrate/internal/contextlogging" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" "github.com/agent-substrate/substrate/internal/resources" ) -// Pool is the WorkerPool of the ateom's pod. +// Pool is the WorkerPool of the worker's pod. type Pool struct { Namespace string Name string @@ -55,7 +55,7 @@ func NewUsageEmitter(lp *sdklog.LoggerProvider, stdout slog.Handler, pool Pool) // Emit writes one record for s, dated when s was read. kind is one of the // ateattr.StatsKind values. -func (e *UsageEmitter) Emit(ctx context.Context, kind string, s *ateompb.WorkloadStatsSample) { +func (e *UsageEmitter) Emit(ctx context.Context, kind string, s *ateworkerpb.WorkloadStatsSample) { if e == nil { return } @@ -66,7 +66,7 @@ func (e *UsageEmitter) Emit(ctx context.Context, kind string, s *ateompb.Workloa // UsageAttrs is the attribute set of ate.actor.usage_sampled for s. The // measurements are absent while s has no source, and the peak also when the // source reports none. -func UsageAttrs(pool Pool, kind string, s *ateompb.WorkloadStatsSample) []slog.Attr { +func UsageAttrs(pool Pool, kind string, s *ateworkerpb.WorkloadStatsSample) []slog.Attr { attrs := append(ateattr.ActorLogAttrs(resources.ActorAttribution{ Ref: resources.ActorRef{Atespace: s.GetAtespace(), Name: s.GetActorName()}, UID: s.GetActorUid(), @@ -80,7 +80,7 @@ func UsageAttrs(pool Pool, kind string, s *ateompb.WorkloadStatsSample) []slog.A slog.String(string(ateattr.StatsKindKey), kind), slog.Int64(string(ateattr.ActorEpochKey), s.GetEpochUnixNano()), ) - if s.GetSource() == ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED { + if s.GetSource() == ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED { return attrs } attrs = append(attrs, @@ -95,11 +95,11 @@ func UsageAttrs(pool Pool, kind string, s *ateompb.WorkloadStatsSample) []slog.A } // StatsSourceLabel maps the wire enum to the ate.stats.source values. -func StatsSourceLabel(s ateompb.StatsSource) string { +func StatsSourceLabel(s ateworkerpb.StatsSource) string { switch s { - case ateompb.StatsSource_STATS_SOURCE_CGROUP: + case ateworkerpb.StatsSource_STATS_SOURCE_CGROUP: return ateattr.StatsSourceCgroup - case ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT: + case ateworkerpb.StatsSource_STATS_SOURCE_GUEST_AGENT: return ateattr.StatsSourceGuestAgent default: return ateattr.StatsSourceUnspecified @@ -111,7 +111,7 @@ func StatsSourceLabel(s ateompb.StatsSource) string { const stdoutQueue = 4096 // NewStdoutHandler writes the stdout form of the usage records to w in the -// ateom's JSON format, at a fixed level so --log-level does not silence it, and +// worker's JSON format, at a fixed level so --log-level does not silence it, and // off the caller's goroutine. func NewStdoutHandler(w io.Writer) *actorevent.AsyncHandler { json := slog.NewJSONHandler(w, &slog.HandlerOptions{Level: slog.LevelInfo}) @@ -145,7 +145,7 @@ func StartSampler(ctx context.Context, interval time.Duration, sweep func(contex } // sweepOnce runs one sweep. It recovers from panics: the sampler is a -// background job, and a bug in it must not take the ateom down and every actor +// background job, and a bug in it must not take the worker down and every actor // on the worker with it. func sweepOnce(ctx context.Context, sweep func(context.Context)) { defer func() { diff --git a/internal/ateomstats/usage_test.go b/internal/workerstats/usage_test.go similarity index 91% rename from internal/ateomstats/usage_test.go rename to internal/workerstats/usage_test.go index 397e63476d..41aad1f8a3 100644 --- a/internal/ateomstats/usage_test.go +++ b/internal/workerstats/usage_test.go @@ -12,7 +12,7 @@ // See the License for the specific language governing permissions and // limitations under the License. -package ateomstats +package workerstats import ( "context" @@ -25,16 +25,16 @@ import ( "github.com/agent-substrate/substrate/internal/actorevent" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/proto/ateompb" + "github.com/agent-substrate/substrate/internal/proto/ateworkerpb" ) var testPool = Pool{Namespace: "team-a", Name: "default"} -func measuredSample() *ateompb.WorkloadStatsSample { - return &ateompb.WorkloadStatsSample{ +func measuredSample() *ateworkerpb.WorkloadStatsSample { + return &ateworkerpb.WorkloadStatsSample{ Atespace: "ns", ActorName: "a", ActorUid: "uid-1", ActorTemplateAtespace: "ns", ActorTemplateName: "t", SandboxClass: "gvisor", - Source: ateompb.StatsSource_STATS_SOURCE_CGROUP, + Source: ateworkerpb.StatsSource_STATS_SOURCE_CGROUP, MemoryCurrentBytes: 40 << 20, MemoryPeakBytes: 48 << 20, MemoryWorkingSetBytes: 32 << 20, @@ -101,7 +101,7 @@ func TestUsageAttrsMeasured(t *testing.T) { func TestUsageAttrsPendingHasNoMeasurements(t *testing.T) { t.Parallel() s := measuredSample() - s.Source = ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED + s.Source = ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED got := attrMap(UsageAttrs(testPool, ateattr.StatsKindPeriodic, s)) checkShape(t, got) for _, k := range actorevent.UsageSampled.Conditional { @@ -225,10 +225,10 @@ func TestStartSamplerSurvivesAPanic(t *testing.T) { func TestStatsSourceLabel(t *testing.T) { t.Parallel() - for src, want := range map[ateompb.StatsSource]string{ - ateompb.StatsSource_STATS_SOURCE_CGROUP: ateattr.StatsSourceCgroup, - ateompb.StatsSource_STATS_SOURCE_GUEST_AGENT: ateattr.StatsSourceGuestAgent, - ateompb.StatsSource_STATS_SOURCE_UNSPECIFIED: ateattr.StatsSourceUnspecified, + for src, want := range map[ateworkerpb.StatsSource]string{ + ateworkerpb.StatsSource_STATS_SOURCE_CGROUP: ateattr.StatsSourceCgroup, + ateworkerpb.StatsSource_STATS_SOURCE_GUEST_AGENT: ateattr.StatsSourceGuestAgent, + ateworkerpb.StatsSource_STATS_SOURCE_UNSPECIFIED: ateattr.StatsSourceUnspecified, } { if got := StatsSourceLabel(src); got != want { t.Errorf("StatsSourceLabel(%v) = %q, want %q", src, got, want) diff --git a/internal/ateomstats/ateomstatstest/ateomstatstest.go b/internal/workerstats/workerstatstest/workerstatstest.go similarity index 86% rename from internal/ateomstats/ateomstatstest/ateomstatstest.go rename to internal/workerstats/workerstatstest/workerstatstest.go index 5877b1095f..55d4d3940d 100644 --- a/internal/ateomstats/ateomstatstest/ateomstatstest.go +++ b/internal/workerstats/workerstatstest/workerstatstest.go @@ -12,8 +12,8 @@ // See the License for the specific language governing permissions and // limitations under the License. -// Package ateomstatstest records the usage records an ateom writes, for tests. -package ateomstatstest +// Package workerstatstest records the usage records a worker writes, for tests. +package workerstatstest import ( "context" @@ -21,7 +21,7 @@ import ( "sync" "github.com/agent-substrate/substrate/internal/ateattr" - "github.com/agent-substrate/substrate/internal/ateomstats" + "github.com/agent-substrate/substrate/internal/workerstats" ) // Recorder is a stdout handler that keeps the kind and source of each record. @@ -32,9 +32,9 @@ type Recorder struct { } // NewEmitter returns an emitter that writes every record to a new Recorder. -func NewEmitter() (*ateomstats.UsageEmitter, *Recorder) { +func NewEmitter() (*workerstats.UsageEmitter, *Recorder) { rec := &Recorder{} - return ateomstats.NewUsageEmitter(nil, rec, ateomstats.Pool{}), rec + return workerstats.NewUsageEmitter(nil, rec, workerstats.Pool{}), rec } // Kinds is the ate.stats.kind of each record, in order. diff --git a/manifests/ate-install/ate-api-server.yaml b/manifests/ate-install/ate-api-server.yaml index 260b5a3363..bc2beac704 100644 --- a/manifests/ate-install/ate-api-server.yaml +++ b/manifests/ate-install/ate-api-server.yaml @@ -142,7 +142,7 @@ spec: - --pod-identity-ca-certs=/run/podidentity.podcert.ate.dev/trust-bundle.pem # TODO(lior):Remove when egress API pr is merged. This turns on pluggable actor # egress cluster-wide. ateapi stamps this address onto every atelet - # Run/Restore, ateom hands it to atunnel, and actor TCP egress is + # Run/Restore, ateworker hands it to atunnel, and actor TCP egress is # transparently redirected (nftables) into atunnel, which wraps it in # mTLS + HTTP CONNECT to the egress gateway. - --default-egress-gateway-address=atenet-egress.ate-system.svc:443 diff --git a/manifests/ate-install/ate-controller.yaml b/manifests/ate-install/ate-controller.yaml index 0f0f1df6a2..7dfd673f6d 100644 --- a/manifests/ate-install/ate-controller.yaml +++ b/manifests/ate-install/ate-controller.yaml @@ -123,7 +123,7 @@ spec: value: k8s.namespace.name=$(POD_NAMESPACE),k8s.pod.name=$(POD_NAME),k8s.pod.uid=$(POD_UID),service.instance.id=$(POD_UID) # OTLP exporter settings are shared by every control plane component; # see manifests/ate-install/ate-otel-config.yaml. The controller also - # propagates these to the ateom worker pods it creates. + # propagates these to the worker pods it creates. envFrom: - configMapRef: name: ate-otel-config diff --git a/manifests/ate-install/ate-otel-config.yaml b/manifests/ate-install/ate-otel-config.yaml index 9668f45601..0c65ca7ea2 100644 --- a/manifests/ate-install/ate-otel-config.yaml +++ b/manifests/ate-install/ate-otel-config.yaml @@ -15,7 +15,7 @@ # Single source of truth for the OTLP exporter settings shared by every control # plane component (ate-api-server, ate-controller, atelet, atenet-router, # atenet-egress), which consume it via `envFrom`. ate-controller additionally propagates these values -# to the ateom worker pods it creates, which have no other way to receive them, +# to the worker pods it creates, which have no other way to receive them, # and atenet-router's --otlp-collector-address defaults to the endpoint below, # so Envoy's own spans follow it too. # diff --git a/manifests/ate-install/ate-system-networkpolicies.yaml b/manifests/ate-install/ate-system-networkpolicies.yaml index eb10e2480c..69e88d6028 100644 --- a/manifests/ate-install/ate-system-networkpolicies.yaml +++ b/manifests/ate-install/ate-system-networkpolicies.yaml @@ -19,7 +19,7 @@ # so a port is reachable exactly when some policy here (or another one that # selects the pod, such as k8s-credential-provider's) admits it. Ports that no # policy admits -- the router's /statusz port (4040), the bundled postgres from -# anything but ate-api-server, atelet's AteomHerder from anything but +# anything but ate-api-server, atelet's Atelet from anything but # ate-api-server -- are reachable only through `kubectl port-forward`, which # NetworkPolicy does not filter. # diff --git a/manifests/ate-install/atelet.yaml b/manifests/ate-install/atelet.yaml index d1359f4e6b..4db08c5760 100644 --- a/manifests/ate-install/atelet.yaml +++ b/manifests/ate-install/atelet.yaml @@ -196,7 +196,7 @@ spec: - configMapRef: name: ate-otel-config volumeMounts: - - name: run-ateom + - name: ate-base mountPath: /var/lib/ate - name: objectstore-plugin mountPath: /run/objectstore-plugin @@ -224,7 +224,7 @@ spec: - --drain-timeout=5m - --objectstore-plugin-socket=/run/objectstore-plugin/node.sock # atelet does no mounts, netlink, device, or namespace operations (those - # live in the ateom worker pod) — it only reads/writes the + # live in the worker pod) — it only reads/writes the # /var/lib/ate hostPath as root. CAP_DAC_OVERRIDE is for one thing: # emptying a directory a non-root actor created in its 0777 durable dir. # It is owned by the actor's uid, typically 0755, so root has no write @@ -295,7 +295,7 @@ spec: initialDelaySeconds: 10 periodSeconds: 10 volumeMounts: - - name: run-ateom + - name: ate-base mountPath: /var/lib/ate - name: objectstore-plugin mountPath: /run/objectstore-plugin @@ -326,7 +326,7 @@ spec: - name: asset-staging emptyDir: sizeLimit: 9Gi - - name: run-ateom + - name: ate-base hostPath: path: /var/lib/ate type: DirectoryOrCreate diff --git a/manifests/ate-install/atenet-egress.yaml b/manifests/ate-install/atenet-egress.yaml index 1537a30bfb..f2672cfeef 100644 --- a/manifests/ate-install/atenet-egress.yaml +++ b/manifests/ate-install/atenet-egress.yaml @@ -234,6 +234,12 @@ data: untyped: - dev.ate.egress - dev.ate.policy.egress + - name: envoy.filters.http.dynamic_modules + typed_config: + "@type": type.googleapis.com/envoy.extensions.filters.http.dynamic_modules.v3.DynamicModuleFilter + dynamic_module_config: + name: envoy_substrate_egress_policy + filter_name: envoy_substrate_egress_policy_store # The CONNECT leg's answers, copied into filter state shared with # the inner listener. Must stay after ext_proc. - name: envoy.filters.http.set_filter_state @@ -251,15 +257,15 @@ data: text_format_source: inline_string: "%DYNAMIC_METADATA(dev.ate.egress:dialed_port)%" omit_empty_values: true - # SNI rules for the egress-policy module. If absent, all TLS - # is denied. - - object_key: dev.ate.policy.egress + # Workaround for dynamic module inability to store filter state that is shared with the upstream. + # TODO(yanavlasov): Remove after Envoy v1.40 is released. + - object_key: dev.ate.policy.egress.hash factory_key: envoy.hashable_string skip_if_empty: true shared_with_upstream: ONCE format_string: text_format_source: - inline_string: "%DYNAMIC_METADATA(dev.ate.policy.egress)%" + inline_string: "%FILTER_STATE(dev.ate.policy.egress.hash.not_shared:PLAIN)%" omit_empty_values: true - name: envoy.filters.http.router typed_config: @@ -866,7 +872,7 @@ data: # Create one pool per unique tuple, containing the following filter state objects: # actor ID from "dev.ate.actor.identity" # CONNECT authority from "dev.ate.connect.authority" - # Egress policy from "dev.ate.policy.egress" + # Egress policy hash from "dev.ate.policy.egress.hash" # TODO(yanavlasov): use just the port from CONNECT athority to minimize number of pools. # Pools are create using hashes of filter state, which includes filter state objects that use # envoy.hashable_string factory. @@ -889,7 +895,7 @@ data: idle_timeout: 30s # Copies every filter state object shared with the upstream onto the # internal listener's downstream connection: dev.ate.actor.identity, - # dev.ate.connect.authority, dev.ate.policy.egress, and + # dev.ate.connect.authority, dev.ate.policy.egress.hash, and # envoy.upstream.dynamic_port, all set on the CONNECT leg above. # # Without this none of them reaches mitm_listener, and nothing reports it: diff --git a/manifests/ate-install/components/agentgateway/kustomization.yaml b/manifests/ate-install/components/agentgateway/kustomization.yaml index c641c1040d..646d21958e 100644 --- a/manifests/ate-install/components/agentgateway/kustomization.yaml +++ b/manifests/ate-install/components/agentgateway/kustomization.yaml @@ -42,7 +42,7 @@ patches: path: /spec/template/spec/containers/0 value: name: agentgateway - image: ghcr.io/kagent-dev/substrate/agentgateway:9557280b@sha256:59602f697f2ab16346c18e0a5a57044e5ff37e56bbed1b7f9de72445e29aee0c + image: ghcr.io/kagent-dev/substrate/agentgateway:3554db38a0b2@sha256:309f6811560635471958db67b0f68dd083d84048dffc681967599d159ebfc83d args: - -f - /etc/agentgateway/config.yaml @@ -118,7 +118,7 @@ patches: path: /spec/template/spec/containers/0 value: name: agentgateway - image: ghcr.io/kagent-dev/substrate/agentgateway:9557280b@sha256:59602f697f2ab16346c18e0a5a57044e5ff37e56bbed1b7f9de72445e29aee0c + image: ghcr.io/kagent-dev/substrate/agentgateway:3554db38a0b2@sha256:309f6811560635471958db67b0f68dd083d84048dffc681967599d159ebfc83d args: - -f - /etc/agentgateway/config.yaml diff --git a/manifests/ate-install/components/router-static-mtls/kustomization.yaml b/manifests/ate-install/components/router-static-mtls/kustomization.yaml new file mode 100644 index 0000000000..ea57dcff67 --- /dev/null +++ b/manifests/ate-install/components/router-static-mtls/kustomization.yaml @@ -0,0 +1,51 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Switches the ingress router to --ingress-auth-mode=static-mtls: every client +# must present a podidentity.podcert.ate.dev certificate whose SPIFFE ID is on +# the allowlist. Selected by cmd/ate-setup under --ingress-auth-mode=static-mtls +# (or ATE_INGRESS_AUTH_MODE=static-mtls), which wraps every control plane apply +# in a kustomization that includes this component. +# +# The flags are appended to the router container's args; the router parses +# its flags with pflag, where a later occurrence wins, so --port-http=0 and +# --port-connect=0 override the plaintext ports the base manifest sets. Those +# two listeners cannot authenticate a client, and the router refuses to start +# in static-mtls mode while either is enabled. +# +# The client CA bundle and the allowed SPIFFE IDs come from the base manifest +# (atenet-router.yaml), where the TLS listeners already check any certificate a +# client presents. + +# TODO(ahmedtd): All of this code (and atenet ingress's support for plaintext +# listeners) should be removed when the deprecated-insecure mode is removed. + +apiVersion: kustomize.config.k8s.io/v1alpha1 +kind: Component + +patches: + - target: + kind: Deployment + name: atenet-router + # containers/0 is the atenet-router container; the envoy sidecar follows it. + patch: |- + - op: add + path: /spec/template/spec/containers/0/args/- + value: --ingress-auth-mode=static-mtls + - op: add + path: /spec/template/spec/containers/0/args/- + value: --port-http=0 + - op: add + path: /spec/template/spec/containers/0/args/- + value: --port-connect=0 diff --git a/manifests/ate-install/generated/ate.dev_workerpools.yaml b/manifests/ate-install/generated/ate.dev_workerpools.yaml index 428862e728..7d07f9c432 100644 --- a/manifests/ate-install/generated/ate.dev_workerpools.yaml +++ b/manifests/ate-install/generated/ate.dev_workerpools.yaml @@ -483,7 +483,7 @@ spec: x-kubernetes-list-type: atomic type: object workerImage: - description: WorkerImage is the ateom container image to deploy as + description: WorkerImage is the worker container image to deploy as workers. minLength: 1 type: string diff --git a/manifests/ate-install/kind/ate-api-server/kustomization.yaml b/manifests/ate-install/kind/ate-api-server/kustomization.yaml new file mode 100644 index 0000000000..c362bb6b00 --- /dev/null +++ b/manifests/ate-install/kind/ate-api-server/kustomization.yaml @@ -0,0 +1,61 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - ../../ate-api-server.yaml + +# OTEL_TRACES_SAMPLER pins ate-api-server to parentbased_always_on so every API +# call it roots is traced; the other components keep their binary defaults and +# follow its sampled flag via their ParentBased samplers. It stays an inline +# patch rather than a key in ate-otel-config.yaml because that ConfigMap is +# shared by every component via envFrom -- putting always_on there would pin the +# router and the control plane to 100% too, undoing their per-component ratios. +# +# The storage settings point ate-api-server's object-store plugin at the same +# rustfs the ../atelet overlay points atelet's at: both ends of an external +# snapshot's life have to agree on where it lives. +patches: + - patch: |- + apiVersion: apps/v1 + kind: Deployment + metadata: + name: ate-api-server + namespace: ate-system + spec: + template: + spec: + containers: + - name: ate-api-server + env: + - name: OTEL_TRACES_SAMPLER + value: parentbased_always_on + initContainers: + - name: objectstore-plugin + env: + - name: ATE_STORAGE_BACKEND + value: s3 + - name: AWS_REGION + value: us-east-1 + - name: AWS_ENDPOINT_URL + value: http://rustfs.ate-system.svc:9000 + - name: AWS_S3_USE_PATH_STYLE + value: "true" + # TODO: use a secret / identity management + - name: AWS_ACCESS_KEY_ID + value: rustfsadmin + - name: AWS_SECRET_ACCESS_KEY + value: rustfsadmin diff --git a/manifests/ate-install/kind/ate-otel-config.yaml b/manifests/ate-install/kind/ate-otel-config.yaml index ed2c8b9334..ce58ab76b6 100644 --- a/manifests/ate-install/kind/ate-otel-config.yaml +++ b/manifests/ate-install/kind/ate-otel-config.yaml @@ -27,11 +27,11 @@ # the shorter interval. Matching it to the interval keeps every tick an attempt. # # On ate-controller both do double duty: the controller also propagates them to -# the ateom worker pods it creates, which have no other way to receive them. +# the worker pods it creates, which have no other way to receive them. # Kind only; production keeps the SDK defaults. # # OTEL_LOGS_EXPORTER=otlp sends the actor events, ateapi's lifecycle records -# and the ateoms' usage samples, over OTLP instead of stdout; kind adds console +# and the ateworkers' usage samples, over OTLP instead of stdout; kind adds console # to keep them in kubectl logs too. serverboot defaults it to none, so the base # ConfigMap leaves it unset and every other environment keeps them on stdout. apiVersion: v1 diff --git a/manifests/ate-install/kind/kustomization.yaml b/manifests/ate-install/kind/kustomization.yaml index 6562870924..ff65fefdd9 100644 --- a/manifests/ate-install/kind/kustomization.yaml +++ b/manifests/ate-install/kind/kustomization.yaml @@ -17,17 +17,17 @@ kind: Kustomization # ate-otel-config.yaml replaces the base ../ate-otel-config.yaml (same ConfigMap # name) so every component's `envFrom` resolves to the in-cluster collector -# instead of the GKE one. It is listed here rather than in ./atelet because that -# overlay is also built standalone; listing it in both would be a duplicate -# resource. hack/install-ate.sh applies the ConfigMap directly for the targeted -# single-component redeploys. +# instead of the GKE one. It is listed here rather than in ./atelet or +# ./ate-api-server because those overlays are also built standalone; listing it +# in the bundle and an overlay would be a duplicate resource. ate-setup applies +# the ConfigMap directly for the single-component redeploys. # # pod-certificate-controller.yaml is deliberately absent, as in ../base: the # installers (hack/install-ate.sh and cmd/ate-setup) apply and delete it on # their own, and re-applying the base file from this bundle would revert the # WORKERS_PER_SIGNER override. resources: - - ../ate-api-server.yaml + - ./ate-api-server - ../ate-controller.yaml - ./atelet - ../atenet-router.yaml @@ -36,46 +36,3 @@ resources: - rustfs.yaml - ./otel-collector.yaml - ./prometheus.yaml - -# OTEL_TRACES_SAMPLER pins ate-api-server to parentbased_always_on so every API -# call it roots is traced; the other components keep their binary defaults and -# follow its sampled flag via their ParentBased samplers. It stays an inline -# patch rather than a key in ate-otel-config.yaml because that ConfigMap is -# shared by every component via envFrom -- putting always_on there would pin the -# router and the control plane to 100% too, undoing their per-component ratios. -# -# The storage settings point ate-api-server's object-store plugin at the same -# rustfs the ./atelet overlay points atelet's at: both ends of an external -# snapshot's life have to agree on where it lives. -patches: - - patch: |- - apiVersion: apps/v1 - kind: Deployment - metadata: - name: ate-api-server - namespace: ate-system - spec: - template: - spec: - containers: - - name: ate-api-server - env: - - name: OTEL_TRACES_SAMPLER - value: parentbased_always_on - initContainers: - - name: objectstore-plugin - env: - - name: ATE_STORAGE_BACKEND - value: s3 - - name: AWS_REGION - value: us-east-1 - - name: AWS_ENDPOINT_URL - value: http://rustfs.ate-system.svc:9000 - - name: AWS_S3_USE_PATH_STYLE - value: "true" - # TODO: use a secret / identity management - - name: AWS_ACCESS_KEY_ID - value: rustfsadmin - - name: AWS_SECRET_ACCESS_KEY - value: rustfsadmin - diff --git a/manifests/ate-install/sandboxconfig-gvisor.yaml b/manifests/ate-install/sandboxconfig-gvisor.yaml index 800bfc608c..a4f6b843df 100644 --- a/manifests/ate-install/sandboxconfig-gvisor.yaml +++ b/manifests/ate-install/sandboxconfig-gvisor.yaml @@ -31,7 +31,7 @@ spec: versions: # gVisor nightly 2026-10-10, for the split filesystem checkpoint # (runsc fscheckpoint, -fs-checkpoint-paths and -fs-restore-image-path) that - # ateom-gvisor uses to save and restore durable-dir volumes. + # ateworker-gvisor uses to save and restore durable-dir volumes. - name: v3 state: Enabled # The root sandbox container's image. On GCP, prefer the in-project mirror diff --git a/manifests/microvm/sandboxconfig-microvm.yaml.tmpl b/manifests/microvm/sandboxconfig-microvm.yaml.tmpl index 81702215cb..d801d8aaef 100644 --- a/manifests/microvm/sandboxconfig-microvm.yaml.tmpl +++ b/manifests/microvm/sandboxconfig-microvm.yaml.tmpl @@ -26,7 +26,7 @@ # object-store plugin - NOT baked into the worker image. ${BUCKET_SCHEME} is # that store's URL scheme (s3 on kind, gs on GKE): the plugin serves only its # own backend's scheme. The assets must stay outside any snapshot location, -# which the plugin refuses to serve as assets. ateom boots cloud-hypervisor +# which the plugin refuses to serve as assets. ateworker boots cloud-hypervisor # itself and gives the actor an overlay rootfs (virtio-fs RO lower + # guest-tmpfs upper), so it fetches virtiofsd too. # diff --git a/pkg/api/v1alpha1/sandboxconfig_types.go b/pkg/api/v1alpha1/sandboxconfig_types.go index 5659e4c0f8..25c8dd863c 100644 --- a/pkg/api/v1alpha1/sandboxconfig_types.go +++ b/pkg/api/v1alpha1/sandboxconfig_types.go @@ -25,9 +25,9 @@ import ( type SandboxClass string const ( - // SandboxClassGvisor is the gVisor/runsc runtime (cmd/ateom-gvisor). Default. + // SandboxClassGvisor is the gVisor/runsc runtime (cmd/ateworker-gvisor). Default. SandboxClassGvisor SandboxClass = sandboxclass.GVisor - // SandboxClassMicroVM is the micro-VM runtime (cmd/ateom-microvm); needs + // SandboxClassMicroVM is the micro-VM runtime (cmd/ateworker-microvm); needs // /dev/kvm and vhost devices. SandboxClassMicroVM SandboxClass = sandboxclass.MicroVM ) diff --git a/pkg/api/v1alpha1/workerpool_types.go b/pkg/api/v1alpha1/workerpool_types.go index d31080749f..cb2e450510 100644 --- a/pkg/api/v1alpha1/workerpool_types.go +++ b/pkg/api/v1alpha1/workerpool_types.go @@ -91,7 +91,7 @@ type WorkerPoolSpec struct { // +kubebuilder:validation:Minimum=0 Replicas int32 `json:"replicas"` - // WorkerImage is the ateom container image to deploy as workers. + // WorkerImage is the worker container image to deploy as workers. // +kubebuilder:validation:MinLength=1 // +required WorkerImage string `json:"workerImage"` diff --git a/pkg/api/v1alpha1/workerpool_validation_test.go b/pkg/api/v1alpha1/workerpool_validation_test.go index e5700c4c62..dbaa294231 100644 --- a/pkg/api/v1alpha1/workerpool_validation_test.go +++ b/pkg/api/v1alpha1/workerpool_validation_test.go @@ -43,7 +43,7 @@ func TestWorkerPoolValidation(t *testing.T) { }, Spec: WorkerPoolSpec{ Replicas: 1, - WorkerImage: "ateom:latest", + WorkerImage: "worker:latest", SandboxClasses: []WorkerPoolSandboxClass{{Name: SandboxClassGvisor}}, }, } @@ -302,7 +302,7 @@ func TestWorkerPoolReservedMetadataUpdate(t *testing.T) { }, Spec: WorkerPoolSpec{ Replicas: 1, - WorkerImage: "example.com/ateom:latest", + WorkerImage: "example.com/worker:latest", SandboxClasses: []WorkerPoolSandboxClass{{Name: SandboxClassGvisor}}, }, } @@ -353,7 +353,7 @@ func TestWorkerPoolSandboxClassesUpdate(t *testing.T) { }, Spec: WorkerPoolSpec{ Replicas: 1, - WorkerImage: "example.com/ateom:latest", + WorkerImage: "example.com/worker:latest", SandboxClasses: []WorkerPoolSandboxClass{{Name: SandboxClassGvisor}}, }, } diff --git a/pkg/proto/ateapipb/ateapi.pb.go b/pkg/proto/ateapipb/ateapi.pb.go index 74b2b36834..ad88e8685f 100644 --- a/pkg/proto/ateapipb/ateapi.pb.go +++ b/pkg/proto/ateapipb/ateapi.pb.go @@ -431,7 +431,7 @@ const ( // Pod terminating. Not schedulable. Final: a draining Worker never becomes // active or unavailable again. WorkerState_WORKER_STATE_DRAINING WorkerState = 2 - // Registered but not Ready, or its ateom restarting. Not schedulable. Every + // Registered but not Ready, or its the worker restarting. Not schedulable. Every // Worker starts here until MarkWorkerAvailable. WorkerState_WORKER_STATE_UNAVAILABLE WorkerState = 3 // Keep this in sync with WorkerStatus.state's maximum. ) @@ -1936,6 +1936,7 @@ type ActorStatus struct { // They are deleted when the actor is deleted. Each template volume // appears at most once. // + // +ate:preview=ExternalVolumes // +k8s:optional // +k8s:maxItems=32 # matches the template's volumes bound // +k8s:listType=map @@ -2577,9 +2578,12 @@ type ActorTemplate struct { // // +k8s:required SandboxConfig *SandboxConfig `protobuf:"bytes,6,opt,name=sandbox_config,json=sandboxConfig,proto3" json:"sandbox_config,omitempty"` - // Resource usage configuration. + // resources is the actor's compute size. Both cpu and memory limits are + // required: the scheduler books them against worker capacity, and a + // micro-VM is booted at that size. // - // +k8s:optional + // +k8s:required + // +k8s:customValidation # cpu and memory limits are both set Resources *Resources `protobuf:"bytes,7,opt,name=resources,proto3" json:"resources,omitempty"` // +k8s:optional Status *ActorTemplateStatus `protobuf:"bytes,8,opt,name=status,proto3" json:"status,omitempty"` @@ -3589,6 +3593,8 @@ type Volume struct { // +k8s:optional // +k8s:unionMember DurableDir *DurableDirVolumeSource `protobuf:"bytes,2,opt,name=durable_dir,json=durableDir,proto3" json:"durable_dir,omitempty"` + // +ate:preview=ExternalVolumes + // +k8s:ifDisabled(ExternalVolumes)=+k8s:forbidden // +k8s:optional // +k8s:unionMember ExternalVolumeTemplate *ExternalVolumeTemplate `protobuf:"bytes,3,opt,name=external_volume_template,json=externalVolumeTemplate,proto3" json:"external_volume_template,omitempty"` @@ -5714,7 +5720,7 @@ type MintActorJwtResponse struct { // // The subject and claim name say the token speaks for the actor itself. // A token minted for a Substrate component acting on the actor's behalf - // (WorkerService.MintAteomActorJwt) uses `ateom-for-actor` in both places + // (WorkerService.MintWorkerActorJwt) uses `worker-for-actor` in both places // instead, so a relying party can grant the two different access. ActorJwt string `protobuf:"bytes,1,opt,name=actor_jwt,json=actorJwt,proto3" json:"actor_jwt,omitempty"` // expire_time is when actor_jwt expires. Equal to its exp claim. @@ -7202,10 +7208,10 @@ type Worker struct { // +k8s:eachKey=+k8s:format=k8s-label-key // +k8s:eachVal=+k8s:format=k8s-label-value Labels map[string]string `protobuf:"bytes,9,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` - // epoch counts the runs of the worker pod's ateom container: 1 for its first + // epoch counts the runs of the worker pod's worker container: 1 for its first // run and one more for each restart. 0 means it has not been reported. // - // A restarted ateom has lost the sandboxes of the Actors it was hosting, so + // A restarted worker has lost the sandboxes of the Actors it was hosting, so // once epoch rises the control plane crashes every Actor placed during an // earlier epoch; status.observed_epoch reports when that is done. It may only // increase. @@ -7358,7 +7364,7 @@ type WorkerStatus struct { Allocated *WorkerResources `protobuf:"bytes,3,opt,name=allocated,proto3" json:"allocated,omitempty"` // observed_epoch is the latest epoch whose earlier Actors the control plane // has crashed and released. While it is below epoch, Actors placed before - // the ateom's last restart may still be reported as running. + // the worker's last restart may still be reported as running. // // +k8s:optional // +k8s:minimum=0 @@ -7950,8 +7956,8 @@ func (x *RegisterWorkerResponse) GetWorker() *Worker { return nil } -// Request message for MintAteomActorCertificate. -type MintAteomActorCertificateRequest struct { +// Request message for MintWorkerActorCertificate. +type MintWorkerActorCertificateRequest struct { state protoimpl.MessageState `protogen:"open.v1"` // The actor for which the certificate should be issued. // @@ -7977,20 +7983,20 @@ type MintAteomActorCertificateRequest struct { sizeCache protoimpl.SizeCache } -func (x *MintAteomActorCertificateRequest) Reset() { - *x = MintAteomActorCertificateRequest{} +func (x *MintWorkerActorCertificateRequest) Reset() { + *x = MintWorkerActorCertificateRequest{} mi := &file_ateapi_proto_msgTypes[111] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *MintAteomActorCertificateRequest) String() string { +func (x *MintWorkerActorCertificateRequest) String() string { return protoimpl.X.MessageStringOf(x) } -func (*MintAteomActorCertificateRequest) ProtoMessage() {} +func (*MintWorkerActorCertificateRequest) ProtoMessage() {} -func (x *MintAteomActorCertificateRequest) ProtoReflect() protoreflect.Message { +func (x *MintWorkerActorCertificateRequest) ProtoReflect() protoreflect.Message { mi := &file_ateapi_proto_msgTypes[111] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -8002,34 +8008,34 @@ func (x *MintAteomActorCertificateRequest) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use MintAteomActorCertificateRequest.ProtoReflect.Descriptor instead. -func (*MintAteomActorCertificateRequest) Descriptor() ([]byte, []int) { +// Deprecated: Use MintWorkerActorCertificateRequest.ProtoReflect.Descriptor instead. +func (*MintWorkerActorCertificateRequest) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{111} } -func (x *MintAteomActorCertificateRequest) GetActor() *ObjectRef { +func (x *MintWorkerActorCertificateRequest) GetActor() *ObjectRef { if x != nil { return x.Actor } return nil } -func (x *MintAteomActorCertificateRequest) GetActorUid() string { +func (x *MintWorkerActorCertificateRequest) GetActorUid() string { if x != nil { return x.ActorUid } return "" } -func (x *MintAteomActorCertificateRequest) GetCertificateSigningRequest() []byte { +func (x *MintWorkerActorCertificateRequest) GetCertificateSigningRequest() []byte { if x != nil { return x.CertificateSigningRequest } return nil } -// Response message for MintAteomActorCertificate. -type MintAteomActorCertificateResponse struct { +// Response message for MintWorkerActorCertificate. +type MintWorkerActorCertificateResponse struct { state protoimpl.MessageState `protogen:"open.v1"` // Response contains a list of DER encoded certificates. The first entry is the // leaf certificate, and any remaining entries are intermediates in @@ -8039,20 +8045,20 @@ type MintAteomActorCertificateResponse struct { sizeCache protoimpl.SizeCache } -func (x *MintAteomActorCertificateResponse) Reset() { - *x = MintAteomActorCertificateResponse{} +func (x *MintWorkerActorCertificateResponse) Reset() { + *x = MintWorkerActorCertificateResponse{} mi := &file_ateapi_proto_msgTypes[112] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *MintAteomActorCertificateResponse) String() string { +func (x *MintWorkerActorCertificateResponse) String() string { return protoimpl.X.MessageStringOf(x) } -func (*MintAteomActorCertificateResponse) ProtoMessage() {} +func (*MintWorkerActorCertificateResponse) ProtoMessage() {} -func (x *MintAteomActorCertificateResponse) ProtoReflect() protoreflect.Message { +func (x *MintWorkerActorCertificateResponse) ProtoReflect() protoreflect.Message { mi := &file_ateapi_proto_msgTypes[112] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -8064,20 +8070,20 @@ func (x *MintAteomActorCertificateResponse) ProtoReflect() protoreflect.Message return mi.MessageOf(x) } -// Deprecated: Use MintAteomActorCertificateResponse.ProtoReflect.Descriptor instead. -func (*MintAteomActorCertificateResponse) Descriptor() ([]byte, []int) { +// Deprecated: Use MintWorkerActorCertificateResponse.ProtoReflect.Descriptor instead. +func (*MintWorkerActorCertificateResponse) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{112} } -func (x *MintAteomActorCertificateResponse) GetActorCertificates() [][]byte { +func (x *MintWorkerActorCertificateResponse) GetActorCertificates() [][]byte { if x != nil { return x.ActorCertificates } return nil } -// Request message for MintAteomActorJwt. -type MintAteomActorJwtRequest struct { +// Request message for MintWorkerActorJwt. +type MintWorkerActorJwtRequest struct { state protoimpl.MessageState `protogen:"open.v1"` // The actor for which the JWT should be issued. // @@ -8105,20 +8111,20 @@ type MintAteomActorJwtRequest struct { sizeCache protoimpl.SizeCache } -func (x *MintAteomActorJwtRequest) Reset() { - *x = MintAteomActorJwtRequest{} +func (x *MintWorkerActorJwtRequest) Reset() { + *x = MintWorkerActorJwtRequest{} mi := &file_ateapi_proto_msgTypes[113] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *MintAteomActorJwtRequest) String() string { +func (x *MintWorkerActorJwtRequest) String() string { return protoimpl.X.MessageStringOf(x) } -func (*MintAteomActorJwtRequest) ProtoMessage() {} +func (*MintWorkerActorJwtRequest) ProtoMessage() {} -func (x *MintAteomActorJwtRequest) ProtoReflect() protoreflect.Message { +func (x *MintWorkerActorJwtRequest) ProtoReflect() protoreflect.Message { mi := &file_ateapi_proto_msgTypes[113] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -8130,41 +8136,41 @@ func (x *MintAteomActorJwtRequest) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use MintAteomActorJwtRequest.ProtoReflect.Descriptor instead. -func (*MintAteomActorJwtRequest) Descriptor() ([]byte, []int) { +// Deprecated: Use MintWorkerActorJwtRequest.ProtoReflect.Descriptor instead. +func (*MintWorkerActorJwtRequest) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{113} } -func (x *MintAteomActorJwtRequest) GetActor() *ObjectRef { +func (x *MintWorkerActorJwtRequest) GetActor() *ObjectRef { if x != nil { return x.Actor } return nil } -func (x *MintAteomActorJwtRequest) GetActorUid() string { +func (x *MintWorkerActorJwtRequest) GetActorUid() string { if x != nil { return x.ActorUid } return "" } -func (x *MintAteomActorJwtRequest) GetAudiences() []string { +func (x *MintWorkerActorJwtRequest) GetAudiences() []string { if x != nil { return x.Audiences } return nil } -// Response message for MintAteomActorJwt. -type MintAteomActorJwtResponse struct { +// Response message for MintWorkerActorJwt. +type MintWorkerActorJwtResponse struct { state protoimpl.MessageState `protogen:"open.v1"` // The signed JWT. Its claims are those described on - // MintActorJwtResponse.actor_jwt, except that it speaks for an ateom + // MintActorJwtResponse.actor_jwt, except that it speaks for a worker // acting on behalf of the actor rather than for the actor itself: // - // * sub: `ateom-for-actor//`. - // * `ateom-for-actor.ate.dev`: A JSON object with the actor's `atespace`, + // * sub: `worker-for-actor//`. + // * `worker-for-actor.ate.dev`: A JSON object with the actor's `atespace`, // `actorName`, and `actorUID`. // // A relying party that trusts actor JWTs must not accept this token as the @@ -8176,20 +8182,20 @@ type MintAteomActorJwtResponse struct { sizeCache protoimpl.SizeCache } -func (x *MintAteomActorJwtResponse) Reset() { - *x = MintAteomActorJwtResponse{} +func (x *MintWorkerActorJwtResponse) Reset() { + *x = MintWorkerActorJwtResponse{} mi := &file_ateapi_proto_msgTypes[114] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } -func (x *MintAteomActorJwtResponse) String() string { +func (x *MintWorkerActorJwtResponse) String() string { return protoimpl.X.MessageStringOf(x) } -func (*MintAteomActorJwtResponse) ProtoMessage() {} +func (*MintWorkerActorJwtResponse) ProtoMessage() {} -func (x *MintAteomActorJwtResponse) ProtoReflect() protoreflect.Message { +func (x *MintWorkerActorJwtResponse) ProtoReflect() protoreflect.Message { mi := &file_ateapi_proto_msgTypes[114] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) @@ -8201,19 +8207,19 @@ func (x *MintAteomActorJwtResponse) ProtoReflect() protoreflect.Message { return mi.MessageOf(x) } -// Deprecated: Use MintAteomActorJwtResponse.ProtoReflect.Descriptor instead. -func (*MintAteomActorJwtResponse) Descriptor() ([]byte, []int) { +// Deprecated: Use MintWorkerActorJwtResponse.ProtoReflect.Descriptor instead. +func (*MintWorkerActorJwtResponse) Descriptor() ([]byte, []int) { return file_ateapi_proto_rawDescGZIP(), []int{114} } -func (x *MintAteomActorJwtResponse) GetActorJwt() string { +func (x *MintWorkerActorJwtResponse) GetActorJwt() string { if x != nil { return x.ActorJwt } return "" } -func (x *MintAteomActorJwtResponse) GetExpireTime() *timestamppb.Timestamp { +func (x *MintWorkerActorJwtResponse) GetExpireTime() *timestamppb.Timestamp { if x != nil { return x.ExpireTime } @@ -9745,18 +9751,18 @@ const file_ateapi_proto_rawDesc = "" + "\x0fdefault_runtime\x18\x03 \x01(\v2\x16.ateapi.SandboxRuntimeR\x0edefaultRuntime\x12G\n" + "\x13restorable_runtimes\x18\x04 \x03(\v2\x16.ateapi.SandboxRuntimeR\x12restorableRuntimes\"@\n" + "\x16RegisterWorkerResponse\x12&\n" + - "\x06worker\x18\x01 \x01(\v2\x0e.ateapi.WorkerR\x06worker\"\xa8\x01\n" + - " MintAteomActorCertificateRequest\x12'\n" + + "\x06worker\x18\x01 \x01(\v2\x0e.ateapi.WorkerR\x06worker\"\xa9\x01\n" + + "!MintWorkerActorCertificateRequest\x12'\n" + "\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" + "\tactor_uid\x18\x02 \x01(\tR\bactorUid\x12>\n" + - "\x1bcertificate_signing_request\x18\x03 \x01(\fR\x19certificateSigningRequest\"R\n" + - "!MintAteomActorCertificateResponse\x12-\n" + - "\x12actor_certificates\x18\x01 \x03(\fR\x11actorCertificates\"~\n" + - "\x18MintAteomActorJwtRequest\x12'\n" + + "\x1bcertificate_signing_request\x18\x03 \x01(\fR\x19certificateSigningRequest\"S\n" + + "\"MintWorkerActorCertificateResponse\x12-\n" + + "\x12actor_certificates\x18\x01 \x03(\fR\x11actorCertificates\"\x7f\n" + + "\x19MintWorkerActorJwtRequest\x12'\n" + "\x05actor\x18\x01 \x01(\v2\x11.ateapi.ObjectRefR\x05actor\x12\x1b\n" + "\tactor_uid\x18\x02 \x01(\tR\bactorUid\x12\x1c\n" + - "\taudiences\x18\x03 \x03(\tR\taudiences\"z\n" + - "\x19MintAteomActorJwtResponse\x12 \n" + + "\taudiences\x18\x03 \x03(\tR\taudiences\"{\n" + + "\x1aMintWorkerActorJwtResponse\x12 \n" + "\tactor_jwt\x18\x01 \x01(\tB\x03\x80\x01\x01R\bactorJwt\x12;\n" + "\vexpire_time\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampR\n" + "expireTime\"\x8d\x01\n" + @@ -9958,11 +9964,11 @@ const file_ateapi_proto_rawDesc = "" + "\x1aUpdateAtespaceAccessPolicy\x12).ateapi.UpdateAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + "\fAccessPolicy\x12q\n" + "\x1aDeleteAtespaceAccessPolicy\x12).ateapi.DeleteAtespaceAccessPolicyRequest\x1a\x14.ateapi.AccessPolicy\"\x12\x82\xb5\x18\x0e\n" + - "\fAccessPolicy2\xfb\x03\n" + + "\fAccessPolicy2\x81\x04\n" + "\rWorkerService\x12O\n" + - "\x0eRegisterWorker\x12\x1d.ateapi.RegisterWorkerRequest\x1a\x1e.ateapi.RegisterWorkerResponse\x12p\n" + - "\x19MintAteomActorCertificate\x12(.ateapi.MintAteomActorCertificateRequest\x1a).ateapi.MintAteomActorCertificateResponse\x12X\n" + - "\x11MintAteomActorJwt\x12 .ateapi.MintAteomActorJwtRequest\x1a!.ateapi.MintAteomActorJwtResponse\x12^\n" + + "\x0eRegisterWorker\x12\x1d.ateapi.RegisterWorkerRequest\x1a\x1e.ateapi.RegisterWorkerResponse\x12s\n" + + "\x1aMintWorkerActorCertificate\x12).ateapi.MintWorkerActorCertificateRequest\x1a*.ateapi.MintWorkerActorCertificateResponse\x12[\n" + + "\x12MintWorkerActorJwt\x12!.ateapi.MintWorkerActorJwtRequest\x1a\".ateapi.MintWorkerActorJwtResponse\x12^\n" + "\x13RequestActorSuspend\x12\".ateapi.RequestActorSuspendRequest\x1a#.ateapi.RequestActorSuspendResponse\x12m\n" + "\x18ReportActorSnapshotState\x12'.ateapi.ReportActorSnapshotStateRequest\x1a(.ateapi.ReportActorSnapshotStateResponse:Y\n" + "\bresource\x12\x1f.google.protobuf.MessageOptions\x18І\x03 \x01(\v2\x1a.ateapi.ResourceAnnotationR\bresource:R\n" + @@ -10103,10 +10109,10 @@ var file_ateapi_proto_goTypes = []any{ (*ActorAssignment)(nil), // 117: ateapi.ActorAssignment (*RegisterWorkerRequest)(nil), // 118: ateapi.RegisterWorkerRequest (*RegisterWorkerResponse)(nil), // 119: ateapi.RegisterWorkerResponse - (*MintAteomActorCertificateRequest)(nil), // 120: ateapi.MintAteomActorCertificateRequest - (*MintAteomActorCertificateResponse)(nil), // 121: ateapi.MintAteomActorCertificateResponse - (*MintAteomActorJwtRequest)(nil), // 122: ateapi.MintAteomActorJwtRequest - (*MintAteomActorJwtResponse)(nil), // 123: ateapi.MintAteomActorJwtResponse + (*MintWorkerActorCertificateRequest)(nil), // 120: ateapi.MintWorkerActorCertificateRequest + (*MintWorkerActorCertificateResponse)(nil), // 121: ateapi.MintWorkerActorCertificateResponse + (*MintWorkerActorJwtRequest)(nil), // 122: ateapi.MintWorkerActorJwtRequest + (*MintWorkerActorJwtResponse)(nil), // 123: ateapi.MintWorkerActorJwtResponse (*RequestActorSuspendRequest)(nil), // 124: ateapi.RequestActorSuspendRequest (*RequestActorSuspendResponse)(nil), // 125: ateapi.RequestActorSuspendResponse (*ReportActorSnapshotStateRequest)(nil), // 126: ateapi.ReportActorSnapshotStateRequest @@ -10284,9 +10290,9 @@ var file_ateapi_proto_depIdxs = []int32{ 113, // 149: ateapi.RegisterWorkerRequest.default_runtime:type_name -> ateapi.SandboxRuntime 113, // 150: ateapi.RegisterWorkerRequest.restorable_runtimes:type_name -> ateapi.SandboxRuntime 111, // 151: ateapi.RegisterWorkerResponse.worker:type_name -> ateapi.Worker - 32, // 152: ateapi.MintAteomActorCertificateRequest.actor:type_name -> ateapi.ObjectRef - 32, // 153: ateapi.MintAteomActorJwtRequest.actor:type_name -> ateapi.ObjectRef - 144, // 154: ateapi.MintAteomActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp + 32, // 152: ateapi.MintWorkerActorCertificateRequest.actor:type_name -> ateapi.ObjectRef + 32, // 153: ateapi.MintWorkerActorJwtRequest.actor:type_name -> ateapi.ObjectRef + 144, // 154: ateapi.MintWorkerActorJwtResponse.expire_time:type_name -> google.protobuf.Timestamp 32, // 155: ateapi.RequestActorSuspendRequest.worker:type_name -> ateapi.ObjectRef 32, // 156: ateapi.RequestActorSuspendRequest.actor:type_name -> ateapi.ObjectRef 15, // 157: ateapi.RequestActorSuspendResponse.actor:type_name -> ateapi.Actor @@ -10361,8 +10367,8 @@ var file_ateapi_proto_depIdxs = []int32{ 137, // 226: ateapi.Control.UpdateAtespaceAccessPolicy:input_type -> ateapi.UpdateAtespaceAccessPolicyRequest 138, // 227: ateapi.Control.DeleteAtespaceAccessPolicy:input_type -> ateapi.DeleteAtespaceAccessPolicyRequest 118, // 228: ateapi.WorkerService.RegisterWorker:input_type -> ateapi.RegisterWorkerRequest - 120, // 229: ateapi.WorkerService.MintAteomActorCertificate:input_type -> ateapi.MintAteomActorCertificateRequest - 122, // 230: ateapi.WorkerService.MintAteomActorJwt:input_type -> ateapi.MintAteomActorJwtRequest + 120, // 229: ateapi.WorkerService.MintWorkerActorCertificate:input_type -> ateapi.MintWorkerActorCertificateRequest + 122, // 230: ateapi.WorkerService.MintWorkerActorJwt:input_type -> ateapi.MintWorkerActorJwtRequest 124, // 231: ateapi.WorkerService.RequestActorSuspend:input_type -> ateapi.RequestActorSuspendRequest 126, // 232: ateapi.WorkerService.ReportActorSnapshotState:input_type -> ateapi.ReportActorSnapshotStateRequest 15, // 233: ateapi.Control.GetActor:output_type -> ateapi.Actor @@ -10410,8 +10416,8 @@ var file_ateapi_proto_depIdxs = []int32{ 130, // 275: ateapi.Control.UpdateAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy 130, // 276: ateapi.Control.DeleteAtespaceAccessPolicy:output_type -> ateapi.AccessPolicy 119, // 277: ateapi.WorkerService.RegisterWorker:output_type -> ateapi.RegisterWorkerResponse - 121, // 278: ateapi.WorkerService.MintAteomActorCertificate:output_type -> ateapi.MintAteomActorCertificateResponse - 123, // 279: ateapi.WorkerService.MintAteomActorJwt:output_type -> ateapi.MintAteomActorJwtResponse + 121, // 278: ateapi.WorkerService.MintWorkerActorCertificate:output_type -> ateapi.MintWorkerActorCertificateResponse + 123, // 279: ateapi.WorkerService.MintWorkerActorJwt:output_type -> ateapi.MintWorkerActorJwtResponse 125, // 280: ateapi.WorkerService.RequestActorSuspend:output_type -> ateapi.RequestActorSuspendResponse 127, // 281: ateapi.WorkerService.ReportActorSnapshotState:output_type -> ateapi.ReportActorSnapshotStateResponse 233, // [233:282] is the sub-list for method output_type diff --git a/pkg/proto/ateapipb/ateapi.proto b/pkg/proto/ateapipb/ateapi.proto index 7d5ecde1e9..233b3fb63e 100644 --- a/pkg/proto/ateapipb/ateapi.proto +++ b/pkg/proto/ateapipb/ateapi.proto @@ -174,7 +174,7 @@ service Control { // if another write lands on the Worker first; retry. // // Every Actor placed after this call is stamped with the Worker's epoch as - // it stands then, so a caller raises epoch for a restarted ateom first. + // it stands then, so a caller raises epoch for a restarted the worker first. rpc MarkWorkerAvailable(MarkWorkerAvailableRequest) returns (MarkWorkerAvailableResponse) { option (method).resource = "Worker"; } @@ -881,6 +881,7 @@ message ActorStatus { // They are deleted when the actor is deleted. Each template volume // appears at most once. // + // +ate:preview=ExternalVolumes // +k8s:optional // +k8s:maxItems=32 # matches the template's volumes bound // +k8s:listType=map @@ -1141,9 +1142,12 @@ message ActorTemplate { // +k8s:required SandboxConfig sandbox_config = 6; - // Resource usage configuration. + // resources is the actor's compute size. Both cpu and memory limits are + // required: the scheduler books them against worker capacity, and a + // micro-VM is booted at that size. // - // +k8s:optional + // +k8s:required + // +k8s:customValidation # cpu and memory limits are both set Resources resources = 7; // +k8s:optional @@ -1446,6 +1450,8 @@ message Volume { // +k8s:unionMember DurableDirVolumeSource durable_dir = 2; + // +ate:preview=ExternalVolumes + // +k8s:ifDisabled(ExternalVolumes)=+k8s:forbidden // +k8s:optional // +k8s:unionMember ExternalVolumeTemplate external_volume_template = 3; @@ -1923,7 +1929,7 @@ message MintActorJwtResponse { // // The subject and claim name say the token speaks for the actor itself. // A token minted for a Substrate component acting on the actor's behalf - // (WorkerService.MintAteomActorJwt) uses `ateom-for-actor` in both places + // (WorkerService.MintWorkerActorJwt) uses `worker-for-actor` in both places // instead, so a relying party can grant the two different access. string actor_jwt = 1 [ debug_redact = true ]; @@ -2295,10 +2301,10 @@ message Worker { // +k8s:eachVal=+k8s:format=k8s-label-value map labels = 9; - // epoch counts the runs of the worker pod's ateom container: 1 for its first + // epoch counts the runs of the worker pod's worker container: 1 for its first // run and one more for each restart. 0 means it has not been reported. // - // A restarted ateom has lost the sandboxes of the Actors it was hosting, so + // A restarted worker has lost the sandboxes of the Actors it was hosting, so // once epoch rises the control plane crashes every Actor placed during an // earlier epoch; status.observed_epoch reports when that is done. It may only // increase. @@ -2325,7 +2331,7 @@ enum WorkerState { // Pod terminating. Not schedulable. Final: a draining Worker never becomes // active or unavailable again. WORKER_STATE_DRAINING = 2; - // Registered but not Ready, or its ateom restarting. Not schedulable. Every + // Registered but not Ready, or its the worker restarting. Not schedulable. Every // Worker starts here until MarkWorkerAvailable. WORKER_STATE_UNAVAILABLE = 3; // Keep this in sync with WorkerStatus.state's maximum. @@ -2358,7 +2364,7 @@ message WorkerStatus { // observed_epoch is the latest epoch whose earlier Actors the control plane // has crashed and released. While it is below epoch, Actors placed before - // the ateom's last restart may still be reported as running. + // the worker's last restart may still be reported as running. // // +k8s:optional // +k8s:minimum=0 @@ -2521,23 +2527,23 @@ service WorkerService { // write. rpc RegisterWorker(RegisterWorkerRequest) returns (RegisterWorkerResponse); - // Create a Substrate-issued SPIFFE certificate that asserts an ateom acting + // Create a Substrate-issued SPIFFE certificate that asserts a worker acting // on behalf of a particular actor. // - // SPIFFE URI: spiffe://${trustdomain}/ateom-for-actor/${atespace}/${actor} - rpc MintAteomActorCertificate(MintAteomActorCertificateRequest) returns (MintAteomActorCertificateResponse); + // SPIFFE URI: spiffe://${trustdomain}/worker-for-actor/${atespace}/${actor} + rpc MintWorkerActorCertificate(MintWorkerActorCertificateRequest) returns (MintWorkerActorCertificateResponse); - // MintAteomActorJwt creates a Substrate-issued JWT that asserts an ateom + // MintWorkerActorJwt creates a Substrate-issued JWT that asserts a worker // acting on behalf of a particular actor: the JWT counterpart of - // MintAteomActorCertificate. atelet calls it with its own client + // MintWorkerActorCertificate. atelet calls it with its own client // certificate and hands the JWT to node-side plugins, which exchange it for // credentials scoped to that actor. // - // The token speaks for the ateom, not the actor: its subject is - // `ateom-for-actor//` and its claims sit under - // `ateom-for-actor.ate.dev`, see MintAteomActorJwtResponse. The lifetime is + // The token speaks for the worker, not the actor: its subject is + // `worker-for-actor//` and its claims sit under + // `worker-for-actor.ate.dev`, see MintWorkerActorJwtResponse. The lifetime is // fixed at one hour. - rpc MintAteomActorJwt(MintAteomActorJwtRequest) returns (MintAteomActorJwtResponse); + rpc MintWorkerActorJwt(MintWorkerActorJwtRequest) returns (MintWorkerActorJwtResponse); // RequestActorSuspend asks the control plane to suspend an Actor that the // calling Worker hosts. Only the Worker can observe what makes an Actor @@ -2611,8 +2617,8 @@ message RegisterWorkerResponse { Worker worker = 1; } -// Request message for MintAteomActorCertificate. -message MintAteomActorCertificateRequest { +// Request message for MintWorkerActorCertificate. +message MintWorkerActorCertificateRequest { // The actor for which the certificate should be issued. // // Must be a valid actor that currently exists according to the actor store. @@ -2637,16 +2643,16 @@ message MintAteomActorCertificateRequest { bytes certificate_signing_request = 3; } -// Response message for MintAteomActorCertificate. -message MintAteomActorCertificateResponse { +// Response message for MintWorkerActorCertificate. +message MintWorkerActorCertificateResponse { // Response contains a list of DER encoded certificates. The first entry is the // leaf certificate, and any remaining entries are intermediates in // leaf-to-root order. repeated bytes actor_certificates = 1; } -// Request message for MintAteomActorJwt. -message MintAteomActorJwtRequest { +// Request message for MintWorkerActorJwt. +message MintWorkerActorJwtRequest { // The actor for which the JWT should be issued. // // Must be a valid actor that currently exists according to the actor store. @@ -2673,14 +2679,14 @@ message MintAteomActorJwtRequest { repeated string audiences = 3; } -// Response message for MintAteomActorJwt. -message MintAteomActorJwtResponse { +// Response message for MintWorkerActorJwt. +message MintWorkerActorJwtResponse { // The signed JWT. Its claims are those described on - // MintActorJwtResponse.actor_jwt, except that it speaks for an ateom + // MintActorJwtResponse.actor_jwt, except that it speaks for a worker // acting on behalf of the actor rather than for the actor itself: // - // * sub: `ateom-for-actor//`. - // * `ateom-for-actor.ate.dev`: A JSON object with the actor's `atespace`, + // * sub: `worker-for-actor//`. + // * `worker-for-actor.ate.dev`: A JSON object with the actor's `atespace`, // `actorName`, and `actorUID`. // // A relying party that trusts actor JWTs must not accept this token as the diff --git a/pkg/proto/ateapipb/ateapi_grpc.pb.go b/pkg/proto/ateapipb/ateapi_grpc.pb.go index c8e48710b7..4043f05a5a 100644 --- a/pkg/proto/ateapipb/ateapi_grpc.pb.go +++ b/pkg/proto/ateapipb/ateapi_grpc.pb.go @@ -161,7 +161,7 @@ type ControlClient interface { // if another write lands on the Worker first; retry. // // Every Actor placed after this call is stamped with the Worker's epoch as - // it stands then, so a caller raises epoch for a restarted ateom first. + // it stands then, so a caller raises epoch for a restarted the worker first. MarkWorkerAvailable(ctx context.Context, in *MarkWorkerAvailableRequest, opts ...grpc.CallOption) (*MarkWorkerAvailableResponse, error) // Mark an ACTIVE Worker UNAVAILABLE so the scheduler stops placing Actors on // it, as while its Pod is not Ready. Idempotent. Leaves any bound Actor @@ -738,7 +738,7 @@ type ControlServer interface { // if another write lands on the Worker first; retry. // // Every Actor placed after this call is stamped with the Worker's epoch as - // it stands then, so a caller raises epoch for a restarted ateom first. + // it stands then, so a caller raises epoch for a restarted the worker first. MarkWorkerAvailable(context.Context, *MarkWorkerAvailableRequest) (*MarkWorkerAvailableResponse, error) // Mark an ACTIVE Worker UNAVAILABLE so the scheduler stops placing Actors on // it, as while its Pod is not Ready. Idempotent. Leaves any bound Actor @@ -1927,11 +1927,11 @@ var Control_ServiceDesc = grpc.ServiceDesc{ } const ( - WorkerService_RegisterWorker_FullMethodName = "/ateapi.WorkerService/RegisterWorker" - WorkerService_MintAteomActorCertificate_FullMethodName = "/ateapi.WorkerService/MintAteomActorCertificate" - WorkerService_MintAteomActorJwt_FullMethodName = "/ateapi.WorkerService/MintAteomActorJwt" - WorkerService_RequestActorSuspend_FullMethodName = "/ateapi.WorkerService/RequestActorSuspend" - WorkerService_ReportActorSnapshotState_FullMethodName = "/ateapi.WorkerService/ReportActorSnapshotState" + WorkerService_RegisterWorker_FullMethodName = "/ateapi.WorkerService/RegisterWorker" + WorkerService_MintWorkerActorCertificate_FullMethodName = "/ateapi.WorkerService/MintWorkerActorCertificate" + WorkerService_MintWorkerActorJwt_FullMethodName = "/ateapi.WorkerService/MintWorkerActorJwt" + WorkerService_RequestActorSuspend_FullMethodName = "/ateapi.WorkerService/RequestActorSuspend" + WorkerService_ReportActorSnapshotState_FullMethodName = "/ateapi.WorkerService/ReportActorSnapshotState" ) // WorkerServiceClient is the client API for WorkerService service. @@ -1954,22 +1954,22 @@ type WorkerServiceClient interface { // MintCert. Idempotent: re-sending the same capacity and runtimes is not a // write. RegisterWorker(ctx context.Context, in *RegisterWorkerRequest, opts ...grpc.CallOption) (*RegisterWorkerResponse, error) - // Create a Substrate-issued SPIFFE certificate that asserts an ateom acting + // Create a Substrate-issued SPIFFE certificate that asserts a worker acting // on behalf of a particular actor. // - // SPIFFE URI: spiffe://${trustdomain}/ateom-for-actor/${atespace}/${actor} - MintAteomActorCertificate(ctx context.Context, in *MintAteomActorCertificateRequest, opts ...grpc.CallOption) (*MintAteomActorCertificateResponse, error) - // MintAteomActorJwt creates a Substrate-issued JWT that asserts an ateom + // SPIFFE URI: spiffe://${trustdomain}/worker-for-actor/${atespace}/${actor} + MintWorkerActorCertificate(ctx context.Context, in *MintWorkerActorCertificateRequest, opts ...grpc.CallOption) (*MintWorkerActorCertificateResponse, error) + // MintWorkerActorJwt creates a Substrate-issued JWT that asserts a worker // acting on behalf of a particular actor: the JWT counterpart of - // MintAteomActorCertificate. atelet calls it with its own client + // MintWorkerActorCertificate. atelet calls it with its own client // certificate and hands the JWT to node-side plugins, which exchange it for // credentials scoped to that actor. // - // The token speaks for the ateom, not the actor: its subject is - // `ateom-for-actor//` and its claims sit under - // `ateom-for-actor.ate.dev`, see MintAteomActorJwtResponse. The lifetime is + // The token speaks for the worker, not the actor: its subject is + // `worker-for-actor//` and its claims sit under + // `worker-for-actor.ate.dev`, see MintWorkerActorJwtResponse. The lifetime is // fixed at one hour. - MintAteomActorJwt(ctx context.Context, in *MintAteomActorJwtRequest, opts ...grpc.CallOption) (*MintAteomActorJwtResponse, error) + MintWorkerActorJwt(ctx context.Context, in *MintWorkerActorJwtRequest, opts ...grpc.CallOption) (*MintWorkerActorJwtResponse, error) // RequestActorSuspend asks the control plane to suspend an Actor that the // calling Worker hosts. Only the Worker can observe what makes an Actor // worth reclaiming -- whether its workload still has anything to do -- and @@ -2023,20 +2023,20 @@ func (c *workerServiceClient) RegisterWorker(ctx context.Context, in *RegisterWo return out, nil } -func (c *workerServiceClient) MintAteomActorCertificate(ctx context.Context, in *MintAteomActorCertificateRequest, opts ...grpc.CallOption) (*MintAteomActorCertificateResponse, error) { +func (c *workerServiceClient) MintWorkerActorCertificate(ctx context.Context, in *MintWorkerActorCertificateRequest, opts ...grpc.CallOption) (*MintWorkerActorCertificateResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) - out := new(MintAteomActorCertificateResponse) - err := c.cc.Invoke(ctx, WorkerService_MintAteomActorCertificate_FullMethodName, in, out, cOpts...) + out := new(MintWorkerActorCertificateResponse) + err := c.cc.Invoke(ctx, WorkerService_MintWorkerActorCertificate_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } return out, nil } -func (c *workerServiceClient) MintAteomActorJwt(ctx context.Context, in *MintAteomActorJwtRequest, opts ...grpc.CallOption) (*MintAteomActorJwtResponse, error) { +func (c *workerServiceClient) MintWorkerActorJwt(ctx context.Context, in *MintWorkerActorJwtRequest, opts ...grpc.CallOption) (*MintWorkerActorJwtResponse, error) { cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...) - out := new(MintAteomActorJwtResponse) - err := c.cc.Invoke(ctx, WorkerService_MintAteomActorJwt_FullMethodName, in, out, cOpts...) + out := new(MintWorkerActorJwtResponse) + err := c.cc.Invoke(ctx, WorkerService_MintWorkerActorJwt_FullMethodName, in, out, cOpts...) if err != nil { return nil, err } @@ -2083,22 +2083,22 @@ type WorkerServiceServer interface { // MintCert. Idempotent: re-sending the same capacity and runtimes is not a // write. RegisterWorker(context.Context, *RegisterWorkerRequest) (*RegisterWorkerResponse, error) - // Create a Substrate-issued SPIFFE certificate that asserts an ateom acting + // Create a Substrate-issued SPIFFE certificate that asserts a worker acting // on behalf of a particular actor. // - // SPIFFE URI: spiffe://${trustdomain}/ateom-for-actor/${atespace}/${actor} - MintAteomActorCertificate(context.Context, *MintAteomActorCertificateRequest) (*MintAteomActorCertificateResponse, error) - // MintAteomActorJwt creates a Substrate-issued JWT that asserts an ateom + // SPIFFE URI: spiffe://${trustdomain}/worker-for-actor/${atespace}/${actor} + MintWorkerActorCertificate(context.Context, *MintWorkerActorCertificateRequest) (*MintWorkerActorCertificateResponse, error) + // MintWorkerActorJwt creates a Substrate-issued JWT that asserts a worker // acting on behalf of a particular actor: the JWT counterpart of - // MintAteomActorCertificate. atelet calls it with its own client + // MintWorkerActorCertificate. atelet calls it with its own client // certificate and hands the JWT to node-side plugins, which exchange it for // credentials scoped to that actor. // - // The token speaks for the ateom, not the actor: its subject is - // `ateom-for-actor//` and its claims sit under - // `ateom-for-actor.ate.dev`, see MintAteomActorJwtResponse. The lifetime is + // The token speaks for the worker, not the actor: its subject is + // `worker-for-actor//` and its claims sit under + // `worker-for-actor.ate.dev`, see MintWorkerActorJwtResponse. The lifetime is // fixed at one hour. - MintAteomActorJwt(context.Context, *MintAteomActorJwtRequest) (*MintAteomActorJwtResponse, error) + MintWorkerActorJwt(context.Context, *MintWorkerActorJwtRequest) (*MintWorkerActorJwtResponse, error) // RequestActorSuspend asks the control plane to suspend an Actor that the // calling Worker hosts. Only the Worker can observe what makes an Actor // worth reclaiming -- whether its workload still has anything to do -- and @@ -2145,11 +2145,11 @@ type UnimplementedWorkerServiceServer struct{} func (UnimplementedWorkerServiceServer) RegisterWorker(context.Context, *RegisterWorkerRequest) (*RegisterWorkerResponse, error) { return nil, status.Error(codes.Unimplemented, "method RegisterWorker not implemented") } -func (UnimplementedWorkerServiceServer) MintAteomActorCertificate(context.Context, *MintAteomActorCertificateRequest) (*MintAteomActorCertificateResponse, error) { - return nil, status.Error(codes.Unimplemented, "method MintAteomActorCertificate not implemented") +func (UnimplementedWorkerServiceServer) MintWorkerActorCertificate(context.Context, *MintWorkerActorCertificateRequest) (*MintWorkerActorCertificateResponse, error) { + return nil, status.Error(codes.Unimplemented, "method MintWorkerActorCertificate not implemented") } -func (UnimplementedWorkerServiceServer) MintAteomActorJwt(context.Context, *MintAteomActorJwtRequest) (*MintAteomActorJwtResponse, error) { - return nil, status.Error(codes.Unimplemented, "method MintAteomActorJwt not implemented") +func (UnimplementedWorkerServiceServer) MintWorkerActorJwt(context.Context, *MintWorkerActorJwtRequest) (*MintWorkerActorJwtResponse, error) { + return nil, status.Error(codes.Unimplemented, "method MintWorkerActorJwt not implemented") } func (UnimplementedWorkerServiceServer) RequestActorSuspend(context.Context, *RequestActorSuspendRequest) (*RequestActorSuspendResponse, error) { return nil, status.Error(codes.Unimplemented, "method RequestActorSuspend not implemented") @@ -2196,38 +2196,38 @@ func _WorkerService_RegisterWorker_Handler(srv interface{}, ctx context.Context, return interceptor(ctx, in, info, handler) } -func _WorkerService_MintAteomActorCertificate_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(MintAteomActorCertificateRequest) +func _WorkerService_MintWorkerActorCertificate_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(MintWorkerActorCertificateRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(WorkerServiceServer).MintAteomActorCertificate(ctx, in) + return srv.(WorkerServiceServer).MintWorkerActorCertificate(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: WorkerService_MintAteomActorCertificate_FullMethodName, + FullMethod: WorkerService_MintWorkerActorCertificate_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(WorkerServiceServer).MintAteomActorCertificate(ctx, req.(*MintAteomActorCertificateRequest)) + return srv.(WorkerServiceServer).MintWorkerActorCertificate(ctx, req.(*MintWorkerActorCertificateRequest)) } return interceptor(ctx, in, info, handler) } -func _WorkerService_MintAteomActorJwt_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(MintAteomActorJwtRequest) +func _WorkerService_MintWorkerActorJwt_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(MintWorkerActorJwtRequest) if err := dec(in); err != nil { return nil, err } if interceptor == nil { - return srv.(WorkerServiceServer).MintAteomActorJwt(ctx, in) + return srv.(WorkerServiceServer).MintWorkerActorJwt(ctx, in) } info := &grpc.UnaryServerInfo{ Server: srv, - FullMethod: WorkerService_MintAteomActorJwt_FullMethodName, + FullMethod: WorkerService_MintWorkerActorJwt_FullMethodName, } handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(WorkerServiceServer).MintAteomActorJwt(ctx, req.(*MintAteomActorJwtRequest)) + return srv.(WorkerServiceServer).MintWorkerActorJwt(ctx, req.(*MintWorkerActorJwtRequest)) } return interceptor(ctx, in, info, handler) } @@ -2280,12 +2280,12 @@ var WorkerService_ServiceDesc = grpc.ServiceDesc{ Handler: _WorkerService_RegisterWorker_Handler, }, { - MethodName: "MintAteomActorCertificate", - Handler: _WorkerService_MintAteomActorCertificate_Handler, + MethodName: "MintWorkerActorCertificate", + Handler: _WorkerService_MintWorkerActorCertificate_Handler, }, { - MethodName: "MintAteomActorJwt", - Handler: _WorkerService_MintAteomActorJwt_Handler, + MethodName: "MintWorkerActorJwt", + Handler: _WorkerService_MintWorkerActorJwt_Handler, }, { MethodName: "RequestActorSuspend", diff --git a/pkg/sandboxclass/sandboxclass.go b/pkg/sandboxclass/sandboxclass.go index 533c05a3af..6be60e944e 100644 --- a/pkg/sandboxclass/sandboxclass.go +++ b/pkg/sandboxclass/sandboxclass.go @@ -21,9 +21,9 @@ package sandboxclass import "slices" const ( - // GVisor is the gVisor/runsc runtime (cmd/ateom-gvisor). Default. + // GVisor is the gVisor/runsc runtime (cmd/ateworker-gvisor). Default. GVisor = "gvisor" - // MicroVM is the micro-VM runtime (cmd/ateom-microvm). It needs /dev/kvm + // MicroVM is the micro-VM runtime (cmd/ateworker-microvm). It needs /dev/kvm // and vhost devices. MicroVM = "microvm" ) diff --git a/tools/apitool/internal/lint/documented.go b/tools/apitool/internal/lint/documented.go index 8ed3298c51..3d25d30199 100644 --- a/tools/apitool/internal/lint/documented.go +++ b/tools/apitool/internal/lint/documented.go @@ -62,7 +62,7 @@ func checkMethodDocumented(rpc model.Method) []Finding { func docText(comment string) string { var kept []string for _, line := range strings.Split(comment, "\n") { - if isValidationGenTag(line) { + if isValidationGenTag(line) || isAteCommentTag(line) { continue } kept = append(kept, line) @@ -75,3 +75,9 @@ func docText(comment string) string { func isValidationGenTag(line string) bool { return strings.HasPrefix(strings.TrimSpace(line), "+k8s:") } + +// isAteCommentTag reports whether line is one of our own metadata tags, +// e.g. +ate:preview. +func isAteCommentTag(line string) bool { + return strings.HasPrefix(strings.TrimSpace(line), "+ate:") +} diff --git a/tools/setup-gcp/dashboards/ate-e2e-latency-dashboard.json b/tools/setup-gcp/dashboards/ate-e2e-latency-dashboard.json index 212135af63..f964ce1ebc 100644 --- a/tools/setup-gcp/dashboards/ate-e2e-latency-dashboard.json +++ b/tools/setup-gcp/dashboards/ate-e2e-latency-dashboard.json @@ -76,7 +76,7 @@ }, { "timeSeriesQuery": { - "prometheusQuery": "histogram_quantile(0.99, sum by (le) (rate({\"rpc.server.call.duration_bucket\", top_level_controller_name=\"atelet\", \"rpc.method\"=\"atelet.AteomHerder/Restore\"}[5m])))", + "prometheusQuery": "histogram_quantile(0.99, sum by (le) (rate({\"rpc.server.call.duration_bucket\", top_level_controller_name=\"atelet\", \"rpc.method\"=\"atelet.Atelet/Restore\"}[5m])))", "unitOverride": "s" }, "plotType": "LINE", diff --git a/vendor/k8s.io/utils/clock/testing/fake_clock.go b/vendor/k8s.io/utils/clock/testing/fake_clock.go new file mode 100644 index 0000000000..7274299ead --- /dev/null +++ b/vendor/k8s.io/utils/clock/testing/fake_clock.go @@ -0,0 +1,374 @@ +/* +Copyright 2014 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package testing + +import ( + "sync" + "time" + + "k8s.io/utils/clock" +) + +var ( + _ = clock.PassiveClock(&FakePassiveClock{}) + _ = clock.WithTicker(&FakeClock{}) + _ = clock.Clock(&IntervalClock{}) +) + +// FakePassiveClock implements PassiveClock, but returns an arbitrary time. +type FakePassiveClock struct { + lock sync.RWMutex + time time.Time +} + +// FakeClock implements clock.Clock, but returns an arbitrary time. +type FakeClock struct { + FakePassiveClock + + // waiters are waiting for the fake time to pass their specified time + waiters []*fakeClockWaiter +} + +type fakeClockWaiter struct { + targetTime time.Time + stepInterval time.Duration + skipIfBlocked bool + destChan chan time.Time + afterFunc func() +} + +// NewFakePassiveClock returns a new FakePassiveClock. +func NewFakePassiveClock(t time.Time) *FakePassiveClock { + return &FakePassiveClock{ + time: t, + } +} + +// NewFakeClock constructs a fake clock set to the provided time. +func NewFakeClock(t time.Time) *FakeClock { + return &FakeClock{ + FakePassiveClock: *NewFakePassiveClock(t), + } +} + +// Now returns f's time. +func (f *FakePassiveClock) Now() time.Time { + f.lock.RLock() + defer f.lock.RUnlock() + return f.time +} + +// Since returns time since the time in f. +func (f *FakePassiveClock) Since(ts time.Time) time.Duration { + f.lock.RLock() + defer f.lock.RUnlock() + return f.time.Sub(ts) +} + +// SetTime sets the time on the FakePassiveClock. +func (f *FakePassiveClock) SetTime(t time.Time) { + f.lock.Lock() + defer f.lock.Unlock() + f.time = t +} + +// After is the fake version of time.After(d). +func (f *FakeClock) After(d time.Duration) <-chan time.Time { + f.lock.Lock() + defer f.lock.Unlock() + stopTime := f.time.Add(d) + ch := make(chan time.Time, 1) // Don't block! + f.waiters = append(f.waiters, &fakeClockWaiter{ + targetTime: stopTime, + destChan: ch, + }) + return ch +} + +// NewTimer constructs a fake timer, akin to time.NewTimer(d). +func (f *FakeClock) NewTimer(d time.Duration) clock.Timer { + f.lock.Lock() + defer f.lock.Unlock() + stopTime := f.time.Add(d) + ch := make(chan time.Time, 1) // Don't block! + timer := &fakeTimer{ + fakeClock: f, + waiter: fakeClockWaiter{ + targetTime: stopTime, + destChan: ch, + }, + } + f.waiters = append(f.waiters, &timer.waiter) + return timer +} + +// AfterFunc is the Fake version of time.AfterFunc(d, cb). +func (f *FakeClock) AfterFunc(d time.Duration, cb func()) clock.Timer { + f.lock.Lock() + defer f.lock.Unlock() + stopTime := f.time.Add(d) + ch := make(chan time.Time, 1) // Don't block! + + timer := &fakeTimer{ + fakeClock: f, + waiter: fakeClockWaiter{ + targetTime: stopTime, + destChan: ch, + afterFunc: cb, + }, + } + f.waiters = append(f.waiters, &timer.waiter) + return timer +} + +// Tick constructs a fake ticker, akin to time.Tick +func (f *FakeClock) Tick(d time.Duration) <-chan time.Time { + if d <= 0 { + return nil + } + f.lock.Lock() + defer f.lock.Unlock() + tickTime := f.time.Add(d) + ch := make(chan time.Time, 1) // hold one tick + f.waiters = append(f.waiters, &fakeClockWaiter{ + targetTime: tickTime, + stepInterval: d, + skipIfBlocked: true, + destChan: ch, + }) + + return ch +} + +// NewTicker returns a new Ticker. +func (f *FakeClock) NewTicker(d time.Duration) clock.Ticker { + f.lock.Lock() + defer f.lock.Unlock() + tickTime := f.time.Add(d) + ch := make(chan time.Time, 1) // hold one tick + f.waiters = append(f.waiters, &fakeClockWaiter{ + targetTime: tickTime, + stepInterval: d, + skipIfBlocked: true, + destChan: ch, + }) + + return &fakeTicker{ + c: ch, + } +} + +// Step moves the clock by Duration and notifies anyone that's called After, +// Tick, or NewTimer. +func (f *FakeClock) Step(d time.Duration) { + f.lock.Lock() + defer f.lock.Unlock() + f.setTimeLocked(f.time.Add(d)) +} + +// SetTime sets the time. +func (f *FakeClock) SetTime(t time.Time) { + f.lock.Lock() + defer f.lock.Unlock() + f.setTimeLocked(t) +} + +// Actually changes the time and checks any waiters. f must be write-locked. +func (f *FakeClock) setTimeLocked(t time.Time) { + f.time = t + newWaiters := make([]*fakeClockWaiter, 0, len(f.waiters)) + for i := range f.waiters { + w := f.waiters[i] + if !w.targetTime.After(t) { + if w.skipIfBlocked { + select { + case w.destChan <- t: + default: + } + } else { + w.destChan <- t + } + + if w.afterFunc != nil { + w.afterFunc() + } + + if w.stepInterval > 0 { + for !w.targetTime.After(t) { + w.targetTime = w.targetTime.Add(w.stepInterval) + } + newWaiters = append(newWaiters, w) + } + + } else { + newWaiters = append(newWaiters, f.waiters[i]) + } + } + f.waiters = newWaiters +} + +// HasWaiters returns true if Waiters() returns non-0 (so you can write race-free tests). +func (f *FakeClock) HasWaiters() bool { + f.lock.RLock() + defer f.lock.RUnlock() + return len(f.waiters) > 0 +} + +// Waiters returns the number of "waiters" on the clock (so you can write race-free +// tests). A waiter exists for: +// - every call to After that has not yet signaled its channel. +// - every call to AfterFunc that has not yet called its callback. +// - every timer created with NewTimer which is currently ticking. +// - every ticker created with NewTicker which is currently ticking. +// - every ticker created with Tick. +func (f *FakeClock) Waiters() int { + f.lock.RLock() + defer f.lock.RUnlock() + return len(f.waiters) +} + +// Sleep is akin to time.Sleep +func (f *FakeClock) Sleep(d time.Duration) { + f.Step(d) +} + +// IntervalClock implements clock.PassiveClock, but each invocation of Now steps the clock forward the specified duration. +// IntervalClock technically implements the other methods of clock.Clock, but each implementation is just a panic. +// +// Deprecated: See SimpleIntervalClock for an alternative that only has the methods of PassiveClock. +type IntervalClock struct { + Time time.Time + Duration time.Duration +} + +// Now returns i's time. +func (i *IntervalClock) Now() time.Time { + i.Time = i.Time.Add(i.Duration) + return i.Time +} + +// Since returns time since the time in i. +func (i *IntervalClock) Since(ts time.Time) time.Duration { + return i.Time.Sub(ts) +} + +// After is unimplemented, will panic. +// TODO: make interval clock use FakeClock so this can be implemented. +func (*IntervalClock) After(_ time.Duration) <-chan time.Time { + panic("IntervalClock doesn't implement After") +} + +// NewTimer is unimplemented, will panic. +// TODO: make interval clock use FakeClock so this can be implemented. +func (*IntervalClock) NewTimer(_ time.Duration) clock.Timer { + panic("IntervalClock doesn't implement NewTimer") +} + +// AfterFunc is unimplemented, will panic. +// TODO: make interval clock use FakeClock so this can be implemented. +func (*IntervalClock) AfterFunc(_ time.Duration, _ func()) clock.Timer { + panic("IntervalClock doesn't implement AfterFunc") +} + +// Tick is unimplemented, will panic. +// TODO: make interval clock use FakeClock so this can be implemented. +func (*IntervalClock) Tick(_ time.Duration) <-chan time.Time { + panic("IntervalClock doesn't implement Tick") +} + +// NewTicker has no implementation yet and is omitted. +// TODO: make interval clock use FakeClock so this can be implemented. +func (*IntervalClock) NewTicker(_ time.Duration) clock.Ticker { + panic("IntervalClock doesn't implement NewTicker") +} + +// Sleep is unimplemented, will panic. +func (*IntervalClock) Sleep(_ time.Duration) { + panic("IntervalClock doesn't implement Sleep") +} + +var _ = clock.Timer(&fakeTimer{}) + +// fakeTimer implements clock.Timer based on a FakeClock. +type fakeTimer struct { + fakeClock *FakeClock + waiter fakeClockWaiter +} + +// C returns the channel that notifies when this timer has fired. +func (f *fakeTimer) C() <-chan time.Time { + return f.waiter.destChan +} + +// Stop prevents the Timer from firing. It returns true if the call stops the +// timer, false if the timer has already expired or been stopped. +func (f *fakeTimer) Stop() bool { + f.fakeClock.lock.Lock() + defer f.fakeClock.lock.Unlock() + + active := false + newWaiters := make([]*fakeClockWaiter, 0, len(f.fakeClock.waiters)) + for i := range f.fakeClock.waiters { + w := f.fakeClock.waiters[i] + if w != &f.waiter { + newWaiters = append(newWaiters, w) + continue + } + // If timer is found, it has not been fired yet. + active = true + } + + f.fakeClock.waiters = newWaiters + + return active +} + +// Reset changes the timer to expire after duration d. It returns true if the +// timer had been active, false if the timer had expired or been stopped. +func (f *fakeTimer) Reset(d time.Duration) bool { + f.fakeClock.lock.Lock() + defer f.fakeClock.lock.Unlock() + + active := false + + f.waiter.targetTime = f.fakeClock.time.Add(d) + + for i := range f.fakeClock.waiters { + w := f.fakeClock.waiters[i] + if w == &f.waiter { + // If timer is found, it has not been fired yet. + active = true + break + } + } + if !active { + f.fakeClock.waiters = append(f.fakeClock.waiters, &f.waiter) + } + + return active +} + +type fakeTicker struct { + c <-chan time.Time +} + +func (t *fakeTicker) C() <-chan time.Time { + return t.c +} + +func (t *fakeTicker) Stop() { +} diff --git a/vendor/k8s.io/utils/clock/testing/simple_interval_clock.go b/vendor/k8s.io/utils/clock/testing/simple_interval_clock.go new file mode 100644 index 0000000000..951ca4d170 --- /dev/null +++ b/vendor/k8s.io/utils/clock/testing/simple_interval_clock.go @@ -0,0 +1,44 @@ +/* +Copyright 2021 The Kubernetes Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package testing + +import ( + "time" + + "k8s.io/utils/clock" +) + +var ( + _ = clock.PassiveClock(&SimpleIntervalClock{}) +) + +// SimpleIntervalClock implements clock.PassiveClock, but each invocation of Now steps the clock forward the specified duration +type SimpleIntervalClock struct { + Time time.Time + Duration time.Duration +} + +// Now returns i's time. +func (i *SimpleIntervalClock) Now() time.Time { + i.Time = i.Time.Add(i.Duration) + return i.Time +} + +// Since returns time since the time in i. +func (i *SimpleIntervalClock) Since(ts time.Time) time.Duration { + return i.Time.Sub(ts) +} diff --git a/vendor/modules.txt b/vendor/modules.txt index c74daaab72..67eb30ed51 100644 --- a/vendor/modules.txt +++ b/vendor/modules.txt @@ -2198,6 +2198,7 @@ k8s.io/streaming/pkg/runtime ## explicit; go 1.23 k8s.io/utils/buffer k8s.io/utils/clock +k8s.io/utils/clock/testing k8s.io/utils/dump k8s.io/utils/internal/third_party/forked/golang/golang-lru k8s.io/utils/internal/third_party/forked/golang/net